The problem is that exceptions are thrown during sampling:
Path [1] :Initial log joint density = -5933.517643
Exception: Exception: integrate: error estimate of integral 1.44459e-06 exceeds the given relative tolerance times norm of integral (in '/tmp/Rtmpy0CRpy/model-4fd3bdf1884.stan', line 13, column 6 to line 17, column 53) (in '/tmp/Rtmpy0CRpy/model-4fd3bdf1884.stan', line 68, column 6 to column 62)
Exception: Exception: integrate: error estimate of integral 3.54559e-121 exceeds the given relative tolerance times norm of integral (in '/tmp/Rtmpy0CRpy/model-4fd3bdf1884.stan', line 13, column 6 to line 17, column 53) (in '/tmp/Rtmpy0CRpy/model-4fd3bdf1884.stan', line 68, column 6 to column 62)
Path [1] : Iter log prob ||dx|| ||grad|| alpha alpha0 # evals ELBO Best ELBO Notes
12 -5.416e+02 5.152e-02 7.691e-01 2.125e-12 1.000e-03 276 nan -5.457e+02LS failed, Hessian reset
Optimization terminated with error: Line search failed to achieve a sufficient decrease, no more progress can be made Stan will still attempt pathfinder but may fail or produce incorrect results.
The relative_tolerance parameter can be optionally specified as the last argument to integrate_1d . By default, integrate_1d follows the Boost library recommendation of setting relative_tolerance to the square root of the machine epsilon of double precision floating point numbers (about 1e-8 ).
but no matter the value (e.g., 1e-2), I still get that expection. Also, this happens with Pathfinder but not with other VI algos. So it is unclear if the problem lies with Pathfinder or Integrate_1d.
Probably not without a reproducible example. The errorās happening within the line search in the integrator, so itās unlikely to be a bug, per se, in Pathfinder. What may be happening is that Pathfinder is finding a part of the density where itās hard to evaluate this integral.
Is that from where you set the relative tolerance to 1e-2?
It is not purely Stan codeāI used the brms::custom_family to inject some Stan code to sample a custom PDF. Hope you can run it. Perhaps your expert eyes can suggest some optimization of that Stan code that could resolve the Pathfinder error.
I donāt know brms and havenāt really used R for the past several years. I tried running what you sent, but Iām missing a bunch of dependencies and gave up after a few minutes of trying to update. If you want to send me some Stan code, Iām happy to take a look.
Iām going to ping @charlesm93 and @mitzimorris as I think Charles wrote the 1d integrator or at least knows a bit about it and Mitzi knows about brms custom families.
I copied the stan code generated by brms here model.stan. And I have attached the data data.rdata (1.2 KB). Would this work with you? I could also provide a Dockerfile, but I am not sure that would make your life much easier.
The other VI algos donāt have line search. Having a large threshold value makes the solver result to have bigger jumps given small changes in parameter values making the target ājumpyā which can make the line search to fail.
The following figure by Ross H. Johnstone (Uncertainty characterisation in action potential modelling for cardiac drug safety, PhD thesis) is illustrating the log density with different thresholds (not in Stan, but illustrates the phenomenon well).
So you may actually want to make the threshold much smaller.
The messages āException: Exception: integrate: error estimate of integral 1.44459e-06 exceeds the given relative tolerance times norm of integralā are annoying, but they are not stopping the Pathfinder.
You may also get āOptimization terminated with error: Line search failed to achieve a sufficient decrease, no more progress can be madeā if you start at the mode or very near the mode of log density (in unconstrained space).
Thank you! I was uncertain whether Pathfinder was working fine. I can recover the model parameters quite well despite those many warningsāmuch better than the other VI algos.
What threshold should I tune specifically? The pathfinder function has a few different ones, plus I have the one for the integration. Would you be able to recommend some values?
Got it thanks! If I set it low (e.g., 1e-36), then I get a long list of exceptions and Pathfinders fails
Exception: Exception: integrate: error estimate of integral 2.42861e-17 exceeds the given relative tolerance times norm of integral (in ā/tmp/RtmpInnpXD/model-4b0453cc859.stanā, line 13, column 6 to line 17, column 54) (in ā/tmp/RtmpInnpXD/model-4b0453cc859.stanā, line 68, column 6 to column 62)
Optimization terminated with error: Line search failed to achieve a sufficient decrease, no more progress can be made Stan will still attempt pathfinder but may fail or produce incorrect results.
No pathfinders ran successfully
if I set it a bit lower than the default (e.g., 1e-12), then I get fewer exceptions
Start sampling
Path [1] :Initial log joint density = -2061.841367
Exception: Exception: integrate: error estimate of integral 8.35554e-13 exceeds the given relative tolerance times norm of integral (in ā/tmp/RtmpInnpXD/model-4b060f45f2b.stanā, line 13, column 6 to line 17, column 54) (in ā/tmp/RtmpInnpXD/model-4b060f45f2b.stanā, line 68, column 6 to column 62)
Path [1] : Iter log prob ||dx|| ||grad|| alpha alpha0 # evals ELBO Best ELBO Notes
20 -2.193e+02 8.660e-04 9.894e-04 1.000e+00 1.000e+00 501 -2.214e+02 -2.214e+02
Path [1] :Best Iter: [15] ELBO (-221.281270) evaluations: (501)
Path [2] :Initial log joint density = -375.366631
Path [2] : Iter log prob ||dx|| ||grad|| alpha alpha0 # evals ELBO Best ELBO Notes
14 -2.193e+02 3.920e-04 2.599e-03 1.000e+00 1.000e+00 351 -2.216e+02 -2.216e+02
Path [2] :Best Iter: [13] ELBO (-221.432315) evaluations: (351)
Path [3] :Initial log joint density = -429.186812
Path [3] : Iter log prob ||dx|| ||grad|| alpha alpha0 # evals ELBO Best ELBO Notes
21 -2.193e+02 1.848e-04 1.984e-03 3.805e-01 1.000e+00 526 -2.214e+02 -2.214e+02
Path [3] :Best Iter: [21] ELBO (-221.364472) evaluations: (526)
Path [4] :Initial log joint density = -545.834089
Exception: Exception: integrate: error estimate of integral 3.54238e-11 exceeds the given relative tolerance times norm of integral (in ā/tmp/RtmpInnpXD/model-4b060f45f2b.stanā, line 13, column 6 to line 17, column 54) (in ā/tmp/RtmpInnpXD/model-4b060f45f2b.stanā, line 68, column 6 to column 62)
Exception: Exception: integrate: error estimate of integral 2.7013e-14 exceeds the given relative tolerance times norm of integral (in ā/tmp/RtmpInnpXD/model-4b060f45f2b.stanā, line 13, column 6 to line 17, column 54) (in ā/tmp/RtmpInnpXD/model-4b060f45f2b.stanā, line 68, column 6 to column 62)
Exception: Exception: integrate: error estimate of integral 1.93731e-05 exceeds the given relative tolerance times norm of integral (in ā/tmp/RtmpInnpXD/model-4b060f45f2b.stanā, line 13, column 6 to line 17, column 54) (in ā/tmp/RtmpInnpXD/model-4b060f45f2b.stanā, line 68, column 6 to column 62)
Exception: Exception: integrate: error estimate of integral 4.14718e-12 exceeds the given relative tolerance times norm of integral (in ā/tmp/RtmpInnpXD/model-4b060f45f2b.stanā, line 13, column 6 to line 17, column 54) (in ā/tmp/RtmpInnpXD/model-4b060f45f2b.stanā, line 68, column 6 to column 62)
Path [4] : Iter log prob ||dx|| ||grad|| alpha alpha0 # evals ELBO Best ELBO Notes
20 -2.193e+02 2.361e-04 7.845e-04 1.000e+00 1.000e+00 501 -2.212e+02 -2.212e+02
Path [4] :Best Iter: [20] ELBO (-221.170131) evaluations: (501)
Total log probability function evaluations:5779
Finished in 13.3 seconds.
I guess there is a sweet spot there, but I feel going much smaller does not work as you suggested (assuming I understood you correctly)
As the double floating presentation accuracy near 1 is 1e-16, there is no sense having the threshold smaller, as there will be errors just from the floating point computation. Something like 1e-4 to 1e-8 should be enough accuracy for most integrals. 1e-2 can be too coarse, causing too big discontinuities in the target.
It can be sometimes annoying to find a good threshold, and it would be nice to get fewer exceptions, but how itās currently implemented there is no easy way to suppress these.
Great if you could get the Pathfinder to finish successfully
