Error in unserialize, rstan 2.18, problems with native mtune and march

Can I get a

native <- system("g++ -dM -E - -march=native", intern = TRUE)
basic <- system("g++ -dM -E - ", intern = TRUE)
setdiff(native, basic)

?

No. g++ not in path, then when I run it in CMD from ming_64\bin it’s just hung.

In CMD, it would be something like

C:\Rtools\mingw_64\bin> echo "" | g++ -dM -E -march=native -

but I guess you would have to write it to a file and read it into R to figure out what are the differences with the -march=native option.

I probably should have just temp broken my path in the first place.

 [1] "#define __SSE4_1__ 1"             "#define __POPCNT__ 1"             "#define __corei7_avx 1"           "#define __sandybridge 1"         
 [5] "#define __XSAVEOPT__ 1"           "#define __BIGGEST_ALIGNMENT__ 32" "#define __SSE4_2__ 1"             "#define __tune_corei7_avx__ 1"   
 [9] "#define __AVX__ 1"                "#define __PCLMUL__ 1"             "#define __XSAVE__ 1"              "#define __sandybridge__ 1"       
[13] "#define __corei7_avx__ 1"         "#define __tune_sandybridge__ 1"   "#define __SSSE3__ 1"              "#define __AES__ 1"

If you do -march=core2 does it crash?

No crashes, that works fine.

Is there any difference betweeen

echo "" | g++ -dM -E -march=core2 -
echo "" | g++ -dM -E -

?

core2 <- system("g++ -dM -E - -march=core2 ", intern = TRUE)
basic <- system("g++ -dM -E - ", intern = TRUE)
setdiff(core2, basic)

[1] "#define __core2__ 1" "#define __SSSE3__ 1" "#define __core2 1"  



native <- system("g++ -dM -E - -march=native ", intern = TRUE)
core2 <- system("g++ -dM -E - -march=core2", intern = TRUE)
setdiff(native, core2)

 [1] "#define __SSE4_1__ 1"             "#define __POPCNT__ 1"             "#define __corei7_avx 1"           "#define __sandybridge 1"         
 [5] "#define __XSAVEOPT__ 1"           "#define __BIGGEST_ALIGNMENT__ 32" "#define __SSE4_2__ 1"             "#define __tune_corei7_avx__ 1"   
 [9] "#define __AVX__ 1"                "#define __PCLMUL__ 1"             "#define __XSAVE__ 1"              "#define __sandybridge__ 1"       
[13] "#define __corei7_avx__ 1"         "#define __tune_sandybridge__ 1"   "#define __AES__ 1"

OK. I guess we might get a bit of speed from enabling SSSE3 instructions by default with -march=core2 and users can add -march=native themselves if it does not crash on their computer.

1 Like

I am getting this error as well. I have two models, one that does an analytical solution for an ode system, and another that does a numerical solution. The analytical one compiles and samples just fine; the numerical one compiles but sometimes gives many warnings about closing unused connections after compiling. Trying sampling() gives the following errors:

Error in unserialize(socklist[[n]]) : error reading from connection
12.
unserialize(socklist[[n]])
11.
recvOneData.SOCKcluster(cl)
10.
recvOneData(cl)
9.
recvOneResult(cl)
8.
dynamicClusterApply(cl, fun, length(x), argfun)
7.
clusterApplyLB(cl = cl, x = splitList(X, nchunks), fun = lapply, FUN = fun, ...)
6.
do.call(c, clusterApplyLB(cl = cl, x = splitList(X, nchunks), fun = lapply, FUN = fun, ...), quote = TRUE)
5.
parallel::parLapplyLB(cl, X = 1:chains, fun = callFun)
4.
.local(object, ...)
3.
sampling(comp_model, pars = c("P_hat", "P_rep"), chains = 9, iter = 1, algorithm = "Fixed_param", init = params, refresh = 0)
2.
sampling(comp_model, pars = c("P_hat", "P_rep"), chains = 9, iter = 1, algorithm = "Fixed_param", init = params, refresh = 0) at functions.r#80
1.
prior_predict(model_s2, 2)
Error in serialize(data, node$con, xdr = FALSE) : error writing to connection
11.
serialize(data, node$con, xdr = FALSE)
10.
sendData.SOCK0node(con, list(type = type, data = value, tag = tag))
9.
sendData(con, list(type = type, data = value, tag = tag))
8.
postNode(n, "DONE")
7.
stopNode(n)
6.
stopCluster.default(cl)
5.
parallel::stopCluster(cl)
4.
.local(object, ...)
3.
sampling(comp_model, pars = c("P_hat", "P_rep"), chains = 9, iter = 1, algorithm = "Fixed_param", init = params, refresh = 0)
2.
sampling(comp_model, pars = c("P_hat", "P_rep"), chains = 9, iter = 1, algorithm = "Fixed_param", init = params, refresh = 0) at functions.r#80
1.
prior_predict(model_s2, 2)

I’ll attach both models, but I doubt the problem is in the Stan program. I’ve looked through a few threads on this error and I’ve tried multiple configuration of the Makevars file as well as reinstalling rstan but I haven’t found a working solution.

stage1.stan (4.2 KB)
stage2.stan (4.6 KB)

System Info:

sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17134)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252 
[2] LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] bayesplot_1.6.0    rstan_2.18.2       StanHeaders_2.18.0
[4] ggplot2_3.1.0
system("wmic cpu get caption, deviceid, name, numberofcores, maxclockspeed, status")
Caption                                 DeviceID  MaxClockSpeed  Name                                      NumberOfCores  Status  
Intel64 Family 6 Model 142 Stepping 10  CPU0      1800           Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz  4              OK      
M <- file.path(Sys.getenv("HOME"), ".R", ifelse(.Platform$OS.type == "windows", "Makevars.win", "Makevars"))
writeLines(readLines(M))
CXXFLAGS=-O3 -Wno-unused-variable -Wno-unused-function -Wno-unused-local-typedefs

CXXFLAGS=-O3 -Wno-unused-variable -Wno-unused-function

CXXFLAGS=-O3 -Wno-unused-variable -Wno-unused-function

CXX14FLAGS=-O3 -march=native -mtune=native
CXX11FLAGS=-O3 -march=native -mtune=native

Do the errors persist when you comment out any lines in ~/.R/Makevars that mention -march and / or -mtune?

1 Like

I’m not sure what syntax to use for comments in the Makevars file, so I just removed those lines and kept this:

CXXFLAGS=-O3 -Wno-unused-variable -Wno-unused-function -Wno-unused-local-typedefs

CXXFLAGS=-O3 -Wno-unused-variable -Wno-unused-function

CXXFLAGS=-O3 -Wno-unused-variable -Wno-unused-function

and still got the errors

Error in unserialize(socklist[[n]]) : error reading from connection
Error in serialize(data, node$con, xdr = FALSE) : error writing to connection

OK, it is probably not related to -mtune=native -march=native then. The next thing to verify is that there are no errors when you call sampling with cores = 1.

1 Like

Model compiles but trying sampling() now results in a fatal error, R session aborted. I tried cores = 1 with and without the -mtune=native -march=native in the Makevars file.

It may be worth noting that with my machine, parallel::detectCores(logical=TRUE) gives 8 while parallel::detectCores(logical=FALSE) gives 4

Edit: On a whim I decided to try switching the ODE solver in my model from integrate_ode_bdf() to integrare_ode_rk45. Now I don’t get a fatal error or an “Error in (un)serialize” but the sampling is not done and I get this error:

[1] "Error in sampler$call_sampler(args_list[[i]]) : "                                                   
[2] "  Exception: Max number of iterations exceeded (1000000).  (in 'model468f90242a_stage2' at line 88)"
[1] "error occurred during calling the sampler; sampling not done"

This looks like the program actually started to run but had some problem with the ode solver, which is a problem I can play around with on my own to try to fix. So maybe there’s some kind of issue with the parallelization in integrate_ode_bdf() ?

I don’t think it has much to do with parallelization but integrate_ode_bdf has been problematic on Windows.

1 Like

Hello @bgoodri @ScottAlder ,
I am also receiving the same error. A simpler version of my model runs ok but when it gets complicated the error appears. I tried to run with/without -march=native -mtune=native but the problem persists.
When I tried cores=1 the Rstudio crashed and the session aborted.

Any new suggestion how to fix this??

Since this is being brought up again, I do want to point out that I did encounter this error again, even while using the rk4 solver, when I tried to introduce a few more parameters to the ODE system. I was able to work without those extra parameters in the end though.

Just run the same piece of code on the uni’s linux server and works fine!. Still don’t know if it is a windows issue or an issue of my hardware.

1 Like

I think it is a weird Windows issue

3 Likes

Thanks a lot. Just installed a virtual machine with ubundu and works fine!

1 Like