I hope this is the right place to ask this question.
I was generating prior-predictive samples to assess the appropriateness of different prior specifications and decided to include a specification that has what one might consider diffuse/uninformative priors (e.g., a very wide normal prior). When I compared the prior-predictive samples across operating systems (Linux, Windows, MacOS), I noticed that they were different!
This difference does not occur when I generate prior-predictive samples based on more informative priors. The results are also the equivalent across OSs if I estimate a “regular” model with the data included to get posterior samples (even with the diffuse priors).
I made sure that the same version of R and rstan were used on each OS. I also used the same seed (using set.seed() and the seed argument in the sampling() function). I wonder if there is some inherent difference in how seeds are treated by different operating systems? Or if these differences might be related to how models are compiled across different OSs? I’m hoping that someone who knows more about these things can help me understand what’s happening? Any input would be much appreciated :)
Thank you for sharing that resource! I hadn’t thought to look at the general Stan reference guide, which seems like the obvious place to look in hindsight.
I just checked the mcmc error of the prior-predictive samples generated on a Mac and on a Linux. The estimates do not appear to be within mcmc error of each other. For example, for one generated variable, average estimate for the Mac is 4.643676 (mcmc se = 13.60324) but for the Linux, it’s -18.46626 (mcmc se = 12.80303).
The large se values make sense to me because of the diffuse priors (as I mentioned in the first post, these discrepancies disappear with more informative priors). But the estimates are really quite different. After reading the info on the page you shared, I know that there are many potential differences in hardware/libraries/etc. that I may not be able to control (the Linux machine is a cluster managed by my institution).
Hmm, true. Looking at the mcmc se values definitely helps explain some of the differences. I guess it’s just showing that the prior isn’t providing much direction (which is not strange) and that the resulting samples can come out very different depending on the specific computer/other underlying libraries etc. used. Thank you for helping me think through this, @jsocolar and @ahartikainen!