The following is a proposal to have Stan provide intermediate updates at fixed units of time rather than fixed units of iteration number.
Currently Stan spits out intermediate output every 10% of iterations, or the user can specify some other updating schedule based on #iterations. But doing it based on #iterations has two problems. First, if Stan runs fast, you’re filling up your console with tons of output (40 lines if you’re running 4 chains), which can make it harder to use the terminal or console. Second, if Stan runs slow, you can wait a long time even for that first output, so you’re sitting with the blank screen and no indication of estimated run time.
My proposal is to instead do this based on wall time. Here’s a proposed default: Updates at 15 seconds, 1 minute, 5 minutes, 20 minutes, 1 hour, 2 hours, 4 hours, 8 hours, 24 hours, and every 24 hours thereafter.
The user would have the option to specify a sequence of updating times (the way the user can now specify number of iterations for each update), for example the user could specify a vectors of time gaps in minutes, with the last gap repeating indefinitely, thus for example the vector (0.5, 1, 5) would correspond to updates at 30 seconds, 1.5 minutes, 6.5 minutes, 11.5 minutes, . . .)
There would also be the option to update after warmup is done.
And there should also be a silent, or update=0, option to suppress all these updates if so desired.
Each update goes as follows. The specified time is a lower bound. (For example, if an update is scheduled at 15 seconds, Stan won’t update exactly at 15 seconds; it will finish the current iteration before updating.)
At the update it would say something like:
Chain 3 update at 15 seconds: 47 iterations so far. Rough estimate 34.2 minutes until completion of chain 3.
Time is formatted in seconds if it’s less than a minute, minutes if it’s less than an hour, hours if it’s less than a day, and days after that. Everything rounded to nearest decimal place. Thus, it would say “1.3 minutes” (rather than “80 seconds or 1 minute 20 seconds”), and “2.1 days” rather than “2 days, 2 hours, 5 minutes, and 15 seconds” or “2:02:05:15” or whatever.
- Notes
I don’t have strong feelings about whether this should be done within Stan or within CmdStan. In future I expect to do most my of applied work using cmdstanr. I won’t try to specify syntax here because I think the programmers have a better sense of what makes sense here.
I’m assuming that this time updating would replace the current iteration-number updating, as I can’t really see why anyone would need iteration-number updating, once time updating is available.
With parallel chains, we’ll get multiple updates at each update time point, just as we currently do with updates by iteration number. That’s ok.
Finally, the estimated time to completion is just a rough estimate. That’s fine. That’s how it is with updates by iteration number also.
- Related issues
I don’t know how Stan and cmdstan deal with intermediate output, but I’m thinking it makes sense to dump output into a file after every iteration, or maybe every 10 iterations, at least as an option so that other programs can grab the output and make partial traceplots etc.
Another thing I’d like to do would be to run Stan for an (approximately) fixed amount of wall time, rather than pre-specifying the number of iterations per chain. So maybe we could talk about that too. But I wrote up the above proposal as a start as it should be simpler to do.