Different Outputs in RStan vs. PyStan

Interfaces
21

Hi @ahartikainen

I try to reproduce a rstan experiment using pystan, but the results are not identical.

Of course, I am sure about this:

  • All the variables have the same values.
  • I use the same OS and computer.
  • I set the same seed in the R and Python experiment (at the beginning of the scripts) and the same seed in sampling rstan/pystan functions.

The printed model summary should be identical? Or is it normal to get different results (beta, eta, pi, epsilon?

Clue: When I run big experiments (chain=4 and iter=1000), the results are pretty similar (but donā€™t match), and when I run toy experiments (chain=1 and iter=5) the results are different.

Best,
Dario

22

Ok, I have done some digging. With extra_compile_args=['-O3'] ( and RStan with -O3) and setting same seed, Iā€™m able to get the same random init.

I also needed to set chain_id = [1, 2, ā€¦] for each chain, so it would follow RStan.

Adaptation info is already different, so I will need to find out where the difference comes from.

23

I am not sure itā€™s worth following this up. Rstan has some slight patches to make things work with CRANā€¦

24

Itā€™s more of a hobby for me.

I think pystan3 / rstan3 should fix this.

25

Currently the difference is created in https://github.com/stan-dev/stan/blob/develop/src/stan/mcmc/hmc/nuts/base_nuts.hpp

Somewhere in transition, before this rng values are same. I had 2 chains where 1st chain had 187 extra calls for rng in RStan compared to PyStan. The seconds chain had 550 (donā€™t remember accurate diff right now) extra calls in PyStan compared to RStan.

26

Ok, there are some different flags for RStan and PyStan

Following settings for PyStan gives same results as in RStan (with the same compiler)

model = pystan.StanModel(..., extra_compile_args=['-O3', "-march=native", "-mtune=native"])

fit = model.sampling(..., seed=123, chain_id=[1,2, ...])

I would also use file= option for StanModel to make sure the model is the same.

See seed example here:


Edit. this of course depends what flags you have used

27 
extra_compile_args=['-O3', "-march=native", "-mtune=native"]

Are these not already set by default? They could give some easy performance benefits to Pystan users, since the flags enable the maximum level of compiler optimisation and SIMD vectorisation.

(Apologies if explaining what you already know!)

28

These were chosen a long time ago. I think these are not the defaults in RStan, but just recommended in the install guide.

cc @ariddell

29

Hereā€™s the relevant history: clang had problems compiling stanc with -O3, see https://github.com/stan-dev/pystan/pull/33 .