`pathfinder`/`integrate_1d` exception

I am using pathfinder to sample a model built with brms with the cmdstanr backend. The model has some custom Stan functions and it involves some integration (see How to implement custom family with integrate_1d - Interfaces / brms - The Stan Forums).

The problem is that exceptions are thrown during sampling:

Path [1] :Initial log joint density = -5933.517643 
Exception: Exception: integrate: error estimate of integral 1.44459e-06 exceeds the given relative tolerance times norm of integral (in '/tmp/Rtmpy0CRpy/model-4fd3bdf1884.stan', line 13, column 6 to line 17, column 53) (in '/tmp/Rtmpy0CRpy/model-4fd3bdf1884.stan', line 68, column 6 to column 62) 
Exception: Exception: integrate: error estimate of integral 3.54559e-121 exceeds the given relative tolerance times norm of integral (in '/tmp/Rtmpy0CRpy/model-4fd3bdf1884.stan', line 13, column 6 to line 17, column 53) (in '/tmp/Rtmpy0CRpy/model-4fd3bdf1884.stan', line 68, column 6 to column 62) 
Path [1] : Iter      log prob        ||dx||      ||grad||     alpha      alpha0      # evals       ELBO    Best ELBO        Notes  
             12      -5.416e+02      5.152e-02   7.691e-01    2.125e-12  1.000e-03       276        nan -5.457e+02LS failed, Hessian reset 
Optimization terminated with error: Line search failed to achieve a sufficient decrease, no more progress can be made Stan will still attempt pathfinder but may fail or produce incorrect results.

According to Computing One Dimensional Integrals, I could change the relative_tolerance in integrate_1d:

The relative_tolerance parameter can be optionally specified as the last argument to integrate_1d . By default, integrate_1d follows the Boost library recommendation of setting relative_tolerance to the square root of the machine epsilon of double precision floating point numbers (about 1e-8 ).

but no matter the value (e.g., 1e-2), I still get that expection. Also, this happens with Pathfinder but not with other VI algos. So it is unclear if the problem lies with Pathfinder or Integrate_1d.

Would you be able to help?

Thanks for reporting.

Probably not without a reproducible example. The errorā€™s happening within the line search in the integrator, so itā€™s unlikely to be a bug, per se, in Pathfinder. What may be happening is that Pathfinder is finding a part of the density where itā€™s hard to evaluate this integral.

Is that from where you set the relative tolerance to 1e-2?

@Bob_Carpenter hereā€™s the example pathfinder-test.R

It is not purely Stan codeā€”I used the brms::custom_family to inject some Stan code to sample a custom PDF. Hope you can run it. Perhaps your expert eyes can suggest some optimization of that Stan code that could resolve the Pathfinder error.

I donā€™t know brms and havenā€™t really used R for the past several years. I tried running what you sent, but Iā€™m missing a bunch of dependencies and gave up after a few minutes of trying to update. If you want to send me some Stan code, Iā€™m happy to take a look.

Iā€™m going to ping @charlesm93 and @mitzimorris as I think Charles wrote the 1d integrator or at least knows a bit about it and Mitzi knows about brms custom families.

I copied the stan code generated by brms here model.stan. And I have attached the data data.rdata (1.2 KB). Would this work with you? I could also provide a Dockerfile, but I am not sure that would make your life much easier.

The other VI algos donā€™t have line search. Having a large threshold value makes the solver result to have bigger jumps given small changes in parameter values making the target ā€œjumpyā€ which can make the line search to fail.

The following figure by Ross H. Johnstone (Uncertainty characterisation in action potential modelling for cardiac drug safety, PhD thesis) is illustrating the log density with different thresholds (not in Stan, but illustrates the phenomenon well).

So you may actually want to make the threshold much smaller.

The messages ā€œException: Exception: integrate: error estimate of integral 1.44459e-06 exceeds the given relative tolerance times norm of integralā€ are annoying, but they are not stopping the Pathfinder.

You may also get ā€œOptimization terminated with error: Line search failed to achieve a sufficient decrease, no more progress can be madeā€ if you start at the mode or very near the mode of log density (in unconstrained space).

2 Likes

Thank you! I was uncertain whether Pathfinder was working fine. I can recover the model parameters quite well despite those many warningsā€”much better than the other VI algos.

What threshold should I tune specifically? The pathfinder function has a few different ones, plus I have the one for the integration. Would you be able to recommend some values?

This tolerance you were talking about in your first message.

Got it thanks! If I set it low (e.g., 1e-36), then I get a long list of exceptions and Pathfinders fails

Exception: Exception: integrate: error estimate of integral 2.42861e-17 exceeds the given relative tolerance times norm of integral (in ā€˜/tmp/RtmpInnpXD/model-4b0453cc859.stanā€™, line 13, column 6 to line 17, column 54) (in ā€˜/tmp/RtmpInnpXD/model-4b0453cc859.stanā€™, line 68, column 6 to column 62)
Optimization terminated with error: Line search failed to achieve a sufficient decrease, no more progress can be made Stan will still attempt pathfinder but may fail or produce incorrect results.
No pathfinders ran successfully

if I set it a bit lower than the default (e.g., 1e-12), then I get fewer exceptions

Start sampling
Path [1] :Initial log joint density = -2061.841367
Exception: Exception: integrate: error estimate of integral 8.35554e-13 exceeds the given relative tolerance times norm of integral (in ā€˜/tmp/RtmpInnpXD/model-4b060f45f2b.stanā€™, line 13, column 6 to line 17, column 54) (in ā€˜/tmp/RtmpInnpXD/model-4b060f45f2b.stanā€™, line 68, column 6 to column 62)
Path [1] : Iter log prob ||dx|| ||grad|| alpha alpha0 # evals ELBO Best ELBO Notes
20 -2.193e+02 8.660e-04 9.894e-04 1.000e+00 1.000e+00 501 -2.214e+02 -2.214e+02
Path [1] :Best Iter: [15] ELBO (-221.281270) evaluations: (501)
Path [2] :Initial log joint density = -375.366631
Path [2] : Iter log prob ||dx|| ||grad|| alpha alpha0 # evals ELBO Best ELBO Notes
14 -2.193e+02 3.920e-04 2.599e-03 1.000e+00 1.000e+00 351 -2.216e+02 -2.216e+02
Path [2] :Best Iter: [13] ELBO (-221.432315) evaluations: (351)
Path [3] :Initial log joint density = -429.186812
Path [3] : Iter log prob ||dx|| ||grad|| alpha alpha0 # evals ELBO Best ELBO Notes
21 -2.193e+02 1.848e-04 1.984e-03 3.805e-01 1.000e+00 526 -2.214e+02 -2.214e+02
Path [3] :Best Iter: [21] ELBO (-221.364472) evaluations: (526)
Path [4] :Initial log joint density = -545.834089
Exception: Exception: integrate: error estimate of integral 3.54238e-11 exceeds the given relative tolerance times norm of integral (in ā€˜/tmp/RtmpInnpXD/model-4b060f45f2b.stanā€™, line 13, column 6 to line 17, column 54) (in ā€˜/tmp/RtmpInnpXD/model-4b060f45f2b.stanā€™, line 68, column 6 to column 62)
Exception: Exception: integrate: error estimate of integral 2.7013e-14 exceeds the given relative tolerance times norm of integral (in ā€˜/tmp/RtmpInnpXD/model-4b060f45f2b.stanā€™, line 13, column 6 to line 17, column 54) (in ā€˜/tmp/RtmpInnpXD/model-4b060f45f2b.stanā€™, line 68, column 6 to column 62)
Exception: Exception: integrate: error estimate of integral 1.93731e-05 exceeds the given relative tolerance times norm of integral (in ā€˜/tmp/RtmpInnpXD/model-4b060f45f2b.stanā€™, line 13, column 6 to line 17, column 54) (in ā€˜/tmp/RtmpInnpXD/model-4b060f45f2b.stanā€™, line 68, column 6 to column 62)
Exception: Exception: integrate: error estimate of integral 4.14718e-12 exceeds the given relative tolerance times norm of integral (in ā€˜/tmp/RtmpInnpXD/model-4b060f45f2b.stanā€™, line 13, column 6 to line 17, column 54) (in ā€˜/tmp/RtmpInnpXD/model-4b060f45f2b.stanā€™, line 68, column 6 to column 62)
Path [4] : Iter log prob ||dx|| ||grad|| alpha alpha0 # evals ELBO Best ELBO Notes
20 -2.193e+02 2.361e-04 7.845e-04 1.000e+00 1.000e+00 501 -2.212e+02 -2.212e+02
Path [4] :Best Iter: [20] ELBO (-221.170131) evaluations: (501)
Total log probability function evaluations:5779
Finished in 13.3 seconds.

I guess there is a sweet spot there, but I feel going much smaller does not work as you suggested (assuming I understood you correctly)

As the double floating presentation accuracy near 1 is 1e-16, there is no sense having the threshold smaller, as there will be errors just from the floating point computation. Something like 1e-4 to 1e-8 should be enough accuracy for most integrals. 1e-2 can be too coarse, causing too big discontinuities in the target.

It can be sometimes annoying to find a good threshold, and it would be nice to get fewer exceptions, but how itā€™s currently implemented there is no easy way to suppress these.

Great if you could get the Pathfinder to finish successfully

1 Like