I’m simulation testing a Stan model across several thousand draws from an operating model. 99% of the time the model works great and converges within about 2 minutes. However, sometimes the model has a really hard time with a specific simulation, and goes from a 2 minute runtime to a 3.5 hour runtime.
Obviously there’s something wrong with my model in those circumstances, but rather than tie up a core for 3.5 hours on that run, I’d rather allow the model to move on after a specified amount of time and just store as many iterations as it got through by that point, so that I can then take a look at those runs and figure out what went wrong after the fact (obviously not to actually use the estimates from that run). I could just cap the max treedepth, but I’m worried that that would mess up my successful runs that sometime need higher treedepths?
I don’t see any options for that in the calls for rstan or sampling, any thoughts?
This could be achieved by tweaking ezStan::watchStan to kill after a supplied time. I don’t have time to implement this for a couple weeks (defending my dissertation Aug 14!), but if you look at how I handle the “kill_on_divergence” argument it should be straightforward to emulate/tweak.
You can set streaming output, too, so that if you kill the process, it won’t kill your existing output. You can also store the random seed so that you can reproduce without storing anything.
Good to know, I’ll give that a shot, though would still need to find some way, maybe using Mike’s suggestion, to kill the process after some period of time right? Thanks!
A second congratulations. Hope you’re at institutions where the defense is pretty much a done deal. It was actually nerve-wracking in the UK getting outside examiners.