Error running Stan model with rstan 2.21 in Windows

@MauritsM this HMM seems to be work in serial or in parallel on @mike-lawrence’s Windows computer, so we need to figure out what the difference is. Can you do

and then tell me if you spot a material difference in the Compilation argument compared to:

Compilation argument:
C:/PROGRA~1/R/R-40~1.2/bin/x64/R CMD SHLIB file382c169e3b14.cpp 2> file382c169e3b14.cpp.err.txt
“C:/rtools40/mingw64/bin/“g++ -std=gnu++14 -I"C:/PROGRA~1/R/R-40~1.2/include” -DNDEBUG -I"C:/Users/Mike/Documents/R/win-library/4.0/Rcpp/include/” -I"C:/Users/Mike/Documents/R/win-library/4.0/RcppEigen/include/" -I"C:/Users/Mike/Documents/R/win-library/4.0/RcppEigen/include/unsupported" -I"C:/Users/Mike/Documents/R/win-library/4.0/BH/include" -I"C:/Users/Mike/Documents/R/win-library/4.0/StanHeaders/include/src/" -I"C:/Users/Mike/Documents/R/win-library/4.0/StanHeaders/include/" -I"C:/Users/Mike/Documents/R/win-library/4.0/RcppParallel/include/" -I"C:/Users/Mike/Documents/R/win-library/4.0/rstan/include" -DEIGEN_NO_DEBUG -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -DSTAN_THREADS -DBOOST_NO_AUTO_PTR -include “C:/Users/Mike/Documents/R/win-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp” -std=c++1y -march=core2 -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c file382c169e3b14.cpp -o file382c169e3b14.o

2 Likes

Hi Ben,

after installing StanHeaders and rstan from the thread you posted the model suddenly seems to sample fine. I’m not sure what’s different now, unfortunately.

I noticed that, when running with cores > 1 I can no longer see the progress of the chains in RStudio. When I click on the Viewer tab I get the following message:

The process cannot access the file because it is being used by another process

The second thing I noticed is that print(fit) is now very slow. It takes minutes to print the summary of the samples, where this used to be a couple of seconds.

I captured the compilation argument when compiling the model (see below). Hopefully this is still useful to you. This seems to be pretty much the same as Mike except for some small changes in Makevars.

Compilation argument:
 C:/PROGRA~1/R/R-40~1.2/bin/x64/R CMD SHLIB file2d786bde218d.cpp 2> file2d786bde218d.cpp.err.txt 
"C:/rtools40/mingw64/bin/"g++  -std=gnu++14 -I"C:/PROGRA~1/R/R-40~1.2/include" -DNDEBUG   -I"C:/Users/mista/Documents/R/win-library/4.0/Rcpp/include/"  -I"C:/Users/mista/Documents/R/win-library/4.0/RcppEigen/include/"  -I"C:/Users/mista/Documents/R/win-library/4.0/RcppEigen/include/unsupported"  -I"C:/Users/mista/Documents/R/win-library/4.0/BH/include" -I"C:/Users/mista/Documents/R/win-library/4.0/StanHeaders/include/src/"  -I"C:/Users/mista/Documents/R/win-library/4.0/StanHeaders/include/"  -I"C:/Users/mista/Documents/R/win-library/4.0/RcppParallel/include/"  -I"C:/Users/mista/Documents/R/win-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -DBOOST_DISABLE_ASSERTS  -DBOOST_PENDING_INTEGER_LOG2_HPP  -DSTAN_THREADS  -DBOOST_NO_AUTO_PTR  -include "C:/Users/mista/Documents/R/win-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp"  -std=c++1y    -march=core2     -O3 -mtune=native -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2  -c file2d786bde218d.cpp -o file2d786bde218d.o

I definitely could see the progress in the Viewer tab. All the chains write to the same file but I don’t know why it couldn’t be read. Is print(fit) slow on all models? I noticed it took a little while but figured that was due to having 10000 iterations.

1 Like

My computer is crashing with @MauritsM 's model.

It was working and then I added CXX14FLAGS=-O3 -mtune=native -march=native -Wno-unused-variable -Wno-unused-function to my ~/Documents/.R.

You want a login or want me to keep messing with this to figure out what’s going on?

Is it the case that -march=native is necessary and sufficient for the crash? This would suggest that it is processor-dependent in addition to Windows-dependent.

The March thing is not working ok on mingw. People should avoid using it on windows. The link I posted above is what can be used according to a colleague of mine.

I can confirm that removing -march=native and following your advice from above solves the issue for me, at least when using the winbuilder binaries Ben posted in Rstan on Windows

When I remove -march=native I have a -march=core2 in my build and the model runs.

I think you can get segfaults trying to execute instructions your cpu doesn’t support. I wonder if there’s a way to scan the object file to figure this out?

I think we just have to avoid -march=native on Windows for now. With g++-4.9 I assumed the processor tuning was due to having processors that were not in production when g++-4.9 had its final release, but perhaps there is something that messes up processor identification on Windows more generally.

2 Likes

Is that all the debugging we need to do now? Do we just watch the forums and make sure this combo of things solves the Windows issues?

This is probably enough to move forward but I would like to hear about any other problems people are encountering on Windows.

1 Like

I have encountered the -march=nativeissue before. It is a longstanding bug in MinGW.

If a CPU supports AVX instruction set -march=nativeallows use of these instuctions. Eigen than uses certain AVX instructions that require 32-bit aligment of memory locations they touch. Windows provides stack that is 16-bit aligned and compiler does not do any extra alignment, so executing these instructions crashes.

I also found a workaround that works for me, even if I have no idea why. If I skip the make and compile by invoking g++ manually, so that everything is compiled in a single invocation and no object files are generated things work for me. No idea how portable this workaround is. Other options are to use either clang or Linux.

5 Likes

The viewer also doesn’t seem to work for other models either. Do you have any idea what could cause that?

print(fit) was quick for the 8 schools model and some other models over here - it might be that for the HMM model I posted there is simply a lot more data to print (and as you mentioned, 10K iterations surely slows that down as well).

EDIT: after some more testing on different machines I get the feeling that this model has always been slow to print and I simply remembered incorrectly. It’s quicker on my Mac but my Mac is way more powerful than my Windows machine. On my Windows machine at work (which runs 2.19) the model is also slow to print, so please don’t spend any more time on this.

What version of RStudio are you using?

1.3.959

Oh, I see there’s a new version - I’ll try updating and will post back.

It still won’t show the progress with multiple cores. Something has changed in RStudio.

That’s strange. Updating to the newer version seems to have solved this problem - I can see progress in the viewer again. Not sure what went wrong but I’m happy it’s working again. Thanks for all the help!

A new version of RStudio or a new version of rstan changed the progress thing?

RStudio - I upgraded to 1.3.1056 and that made the problem go away (hopefully permanently).

1 Like