Hi,
I was trying to recreate a successful run I had made.
The new run was initialized with warm_up=0, adaptation=0, a random realization from the original successful run as init, the mass matrix and step-size from chain 1 was also given.
In my mind I should be able to begin sampling right away. But I end up with E-BFMI warnings, slightly correlated samples and generally bad result regarding the physical phenomena I am modelling.
Is there something I am forgetting ?
Thanks in advance!
I changed warm_up=100 while still having adaptation=0. This seems to have done the trick.
The only reason I can see for this to work is if the random realization used as initial guess was flawed.
what interface were you using? warm_up and adaptaton aren’t CmdStan argument names.
You are entirely correct. I have renamed the arguments in my project and had almost forgot they weren’t the real ones.
num_warmup=0
engaged=0
did you specify the random seed as well?
I don’t think that 100 warmup draws with no adaptation would improve things.
Did you change the random draw used to initialize the parameters? did you get the stepsize, mass matrix, and draw used for initial parameterization from the same chain? did you check that the stepsize and mass matrix from the run matched the specified stepsize and mass matrix?
what do you mean by “successful run”? did you run CmdStan’s bin/diagnose utility on the outputs from the good run?
No I am not manually specify the random seed. It automatically gives a seed to each chain.
No, I did not alter the random draw used to initialize, wouldn’t be against the point?
Yes, I got it from the same chain.
Yes, I have checked that they match
By successful run I mean that it passes all the tests provided in bin/diagnose including my own pertaining to the physical nature of the problem I am sampling.
you should specify the seed if you want to recreate a run -
https://betanalpha.github.io/assets/case_studies/rstan_workflow.html
We recommend explicitly specifying the seed of Stan’s random number generator, as we have done here, so that we can reproduce these exactly results in the future, at least when using the same machine, operating system, and interface. This is especially helpful for the more subtle pathologies that may not always be found, which results in seemingly stochastic behavior.
note that in RStan and PyStan, ditto examples in CmdStan Guide, that when you run multiple chains they are all run with the same seed.
To sample four chains using a Bash shell on Mac OS or Linux, execute1
> for i in {1..4} do ./my_model sample random seed=12345 \ id=$i data file=my_data \ output file=samples$i.csv & done
I will try that first thing tomorrow.
But I do have my doubts that specifying the seed will help.
If I was recreating the run with warm-up then the seed should be the same. But when I am skipping the warm-up and going straight to sampling (maybe to increase my sample size) using the earlier determined step-size, mass matrix and random realization as initialization then the seed wont matter, will it?
do you have output from a run with 0 warmup iterations and low E-BFMI warnings, bad results, etc?
if so, could you do another run using that seed and id and run with 100 warmup iterations.
my understanding is that running warmup with adaptation turned off isn’t necessary, but maybe it is - in which case, I hope that @betanalpha can explain why.
It depends on whether or not your initial state is in the typical set or not.
adapt engaged=0.adapt engaged=1 with care taken to set the window parameters appropriate to the small warmup period.The (short) initial phase of warmup is to find the typical set, but the bulk of warmup is to tune the adaptation.
CmdStan argument question - the recommendation for the warmup schedule is stated in terms of percentage of total warmup iterations - ideally 15/75/10 for init_buffer, window, and term_buffer respectively. Why not reflect this in the way these arguments are specified - as %'s that add up to 1?
window is not the length of the intermediate period, it’s the length of the metric adaptation period that is doubled until n_warmup - term_buffer iterations are reached. Percentages would make it annoying to figure out what the initial adaptation window length is. On top of that the initial and terminal buffers require a minimum length and the window length has to be dynamically tweaked to accommodate.
See https://github.com/stan-dev/stan/blob/develop/src/stan/mcmc/windowed_adaptation.hpp and page 44 of the CmdStan manual (https://github.com/stan-dev/cmdstan/releases/download/v2.19.1/cmdstan-guide-2.19.1.pdf). I have no idea why this isn’t link from the CmdStan readme or the main website.
thank you - this is very clear!
I found that the problem was due to an inconsistency in my syntax.
In my model the parameter being estimated is called m.
The initial values passed in the .R file was called M.
Apparently you are not prompted with a warning in cmdStan if it doesn’t understand your initialization file.
Now the sampler starts in the typical set and no warm-up or adaptation is needed.
what’s going on here is that the initialization file is a way to specify some parameters.
CmdStan parses the .R file into a varContext object - essentially a dict of parameter names and values.
When the sampler initializes the model parameters, if there is no value for parameter m in that varContext object, then the sampler will do the default initialization.
there are no checks that all entries in the varContext object correspond to parameter names - perhaps there should be.
Thanks for explaining. That makes sense!
This exact thing has bit us before! (Passing bad named value for initialization that was silently discarded)