My favorite Error in unserialize is back :-)

Interfaces RStan
1

Yes indeed - always a humbling experience.

Am I wrong or am I not the only one plagued by this?

Error in unserialize(socklist[[n]]) : error reading from connection
Error in serialize(data, node$con, xdr = FALSE) : error writing to connection
(RStudio crashes when switching to 1 core)

Shouldn’t we expect a less obscure message?

Still, I want to remain positive. The problem may well be a cockpit error (i.e., me). Though I have run a virtually identical code in the past with no problem.

I would be so grateful if some kind person could run the attached R code, using the attached Stan model and the attached Stan data.

A. Does it at least try to sample or do you also see this same error?

B. If it samples, I am curious what machine or system you are using.

C. If you also get the message, do you see any obvious problem with the code (I have spent 2 days looking at it and I find nothing obvious wrong).

D. Can you suggest any diagnostics to help identify the cause? I’ve used Stan for a number of years but I am not an IT professional.

I am running on an HP Zbook (x64, 16GRAM, Intel i7-5500U 4 core CPU @ 2.40GHz) under Windows 10 Pro. R version 3.6.3, rstan version 2.19.3, RStudio version 1.2.1335.

I sincerely appreciate any advice. Thank you!!

SODF1.csv (8.0 KB) SODF1.r (1.5 KB) SODF1d.stan (1.0 KB)

2

Let us know if the problem persists with rstan 2.21.2.

3

It samples for me. Mac OS X 10.15.5, R 4.02, RStan 2.21.2. It hasn’t finished yet, but it definitely started sampling (at about iteration 2000 right now).

Yeah, that message is pretty bad. That’s not coming from rstan itself (I guess probably from the parallel R package, but I’d need to double check), but maybe we could catch it and report a nicer error message.

I doubt it’s anything you’re doing wrong. We haven’t seen many reports of this recently but I definitely remember a bunch of them in the past.

There are newer versions of R, RStan, and RStudio. Since we haven’t seen this reported much recently but we did in the past, it might be the case that installing some combination of those (preferably all of them) will resolve this. Sorry I don’t have a better answer than that!

4

I meant to add that you also may have more luck using the new CmdStanR interface since it uses a different method for parallelizing multiple chains (it doesn’t use the parallel package at all).

5

`Omigosh, Ben and Johah… Thank you both so much. I will upgrade and get back. I guess I have been asleep at the wheel here!! :-)

6

OK. Upgrade … so here is what I did…

  • upgraded R from 3.6.3 to 4.0.2 using the installr package. Worked great
  • upgraded RStudio from 1.2.1335 to 1.3.1056 from the RStudio/Help/check for updates menu. This worked OK, but I lost all my packages even though I requested they transfer. grrr… that’s OK I can deal with that.
  • upgraded rstan from 2.19.3 to 2.21.2 simply by installing rstan (since package was not available anymore anyway).

Now I’m ready to roll, right? wrong… Rtools 3.5 needs to be upgraded. so…

  • perusing the web I find a deluge of paths for acquiring Rtools that will work for rstan. Most seem to lead to the Stan Forum and end with no resolution. That’s OK. I’m fearless… downloaded Rtools4.0 from https://cran.r-project.org/bin/windows/Rtools/ and “installed”.
    pkgbuild::has_build_tools(debug = TRUE)
    [1] TRUE

NOW I’m ready to roll, right? wrong… Warning message:
In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
‘C:/Rtools/mingw_/bin/g++’ not found Doesn’t sound good.

  • So I searched for deeper guidance. I found this… https://github.com/stan-dev/rstan/wiki/RStan-Getting-Started. It stated “You have to do such and such…” then (in effect) “But be sure not to do such and such…” and “use the following guidance,… but on second thought don’t use this guidance”.

Well that is OK because I can’t parse “such and such” or the deep guidance IT lingo… “makevars file?”, “checking the tool chain?”, . I know I need to give rstan the ability to compile, but the underlying mechanistic relationship is black magic to me.

At this point I surrender. I can’t connect the dots. Is there any hope for me? I’m pretty good with simple recipes. Where is the “RTools+rstan for Dummies”? … help please …

Thank you :-)

7

Some additional details which I do not understand but might help diagnose:

  • Message reported after attempt to compile:
    Warning message:
    In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
    ‘C:/Rtools/mingw_/bin/g++’ not found

  • Message reported after subsequent attempt to sample:
    Error: cannot allocate vector of size 19529.8 Gb

  • Sys.info()

  sysname        release        version       nodename        machine   "Windows"
  "10 x64"  "build 18363"   "DAVE-ZBOOK"       "x86-64" 
   login           user effective_user 
 "Dave"         "Dave"         "Dave"
  • sessionInfo()

R version 4.0.2 (2020-06-22)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 18363)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252
[2] LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] forcats_0.5.0 stringr_1.4.0 dplyr_1.0.0
[4] purrr_0.3.4 readr_1.3.1 tidyr_1.1.0
[7] tibble_3.0.3 tidyverse_1.3.0 rstan_2.21.2
[10] ggplot2_3.3.2 StanHeaders_2.21.0-5

loaded via a namespace (and not attached):
[1] tidyselect_1.1.0 haven_2.3.1 V8_3.2.0
[4] colorspace_1.4-1 vctrs_0.3.2 generics_0.0.2
[7] stats4_4.0.2 loo_2.3.1 blob_1.2.1
[10] rlang_0.4.7 pkgbuild_1.1.0 pillar_1.4.6
[13] glue_1.4.1 withr_2.2.0 DBI_1.1.0
[16] dbplyr_1.4.4 readxl_1.3.1 modelr_0.1.8
[19] matrixStats_0.56.0 lifecycle_0.2.0 cellranger_1.1.0
[22] munsell_0.5.0 gtable_0.3.0 rvest_0.3.6
[25] codetools_0.2-16 inline_0.3.15 callr_3.4.3
[28] ps_1.3.3 parallel_4.0.2 curl_4.3
[31] fansi_0.4.1 broom_0.7.0 Rcpp_1.0.5
[34] scales_1.1.1 backports_1.1.7 RcppParallel_5.0.2
[37] jsonlite_1.7.0 fs_1.4.2 gridExtra_2.3
[40] hms_0.5.3 stringi_1.4.6 processx_3.4.3
[43] grid_4.0.2 cli_2.0.2 tools_4.0.2
[46] magrittr_1.5 crayon_1.3.4 pkgconfig_2.0.3
[49] ellipsis_0.3.1 xml2_1.3.2 prettyunits_1.1.1
[52] reprex_0.3.0 lubridate_1.7.9 assertthat_0.2.1
[55] httr_1.4.2 rstudioapi_0.11 R6_2.4.1
[58] compiler_4.0.2

  • Sys.getenv(“PATH”)… Notice how long this is!!

[1] “C:\Program Files\R\R-4.0.2\bin\x64;C:\Rtools\bin;C:\Rtools\bin;C:\Rtools\mingw_64\bin;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\Intel\iCLS Client;C:\Program Files\Intel\iCLS Client;C:\Windows\System32;C:\Windows;C:\Windows\System32\wbem;C:\Windows\System32\WindowsPowerShell\v1.0;c:\Program Files (x86)\ATI Technologies\ATI.ACE\Core-Static;C:\Program Files\Intel\WiFi\bin;C:\Program Files\Common Files\Intel\WirelessCommon;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\Hewlett-Packard\HP Performance Advisor;C:\Windows\System32\OpenSSH;C:\Users\Dave\AppData\Local\Programs\MiKTeX 2.9\miktex\bin\x64;C:\Users\Dave\AppData\Local\Microsoft\WindowsApps;C:\Users\Dave\AppData\Local\Box\Box Edit”

Any suggestions please? Thank you so much.

8

This error should be safe to ignore.

Did you successfully get Rtools 4 and the new rstan installed? You can check your rstan version by doing library(rstan) and checking sessionInfo().

I’m not familiar with that out of memory error, but you can try running a simple model with stan_demo("eight_schools"). Give that a go and see if you still get the memory errors.

9

Well, I gues you did given you got that error :P.

Anyhow, I went searching for the cannot allocate error and found this thread: Cannot resolve the "cannot allocate vector of size … " error - #57 by bgoodri

It looks like the discussion in that thread is around cleaning out old Makevars and R environment stuff. Like, delete all your Makevars.win, .Renviron, etc (or save backups), and then follow the Windows 10 instructions here: R 4.0, rstan, and you

10

Thanks. Yes, rstan 2.21.2 is indicated as an attached package after “sessionInfo()”.

I suspect Rtools 4 is the issue. I installed Rtools 4 yesterday and I see there is now a new directory “c:\rtools40” on my computer. However, this directory is not indicated in the PATH (see Sys.getenv(“PATH”) output above). Instead the PATH seems to have only the “older” path for (I suppose) Rtools3.5.

When I installed Rtools 4, I don’t remember it asking me about the PATH. At this point I am not sure if I can modify the PATH, or if I can what I need to put in the PATH. Any experience with that?

11

I’m not 100% where the R PATH comes from. I certainly didn’t need to add anything to my system PATH, and I think I didn’t have any extra paths left over from a previous install.

See if you have the files .R/Makevars or .R/Makevars.win or .Renviron and either delete them or rename then and follow the directions here and see if this works: R 4.0, rstan, and you

I think it would be a good to try to remove any of the straggling rtools 3.5 stuff you see.

12

Oh and did you uninstall rtools 3.5 before installing rtools 4? I don’t know if rtools4 will let you install alongside rtools 3.5, but if they’re both there maybe something is getting confused.

13

Couple points:

  • Thanks for the “8 schools demo” It seemed to run OK. But the program that I have been trying to run still will not sample. I get the message ** Warning message:
    In system(paste(CXX, ARGS), ignore.stdout = TRUE, ignore.stderr = TRUE) :
    ‘C:/Rtools/mingw_/bin/g++’ not found    **. So something is not right and I also do not feel comfortable with the Warning message the compiler generates (above). So…

  • I don’t see any “.Renviron” files but I have 24 files named Makevars.* on my system. About half of these are WIN files. However, only one of these is located in “c:\Users\Dave\Documents.R”. Is this the one I should delete/rename?

  • No I did not deinstall Rtools 3.5 prior to installing Rtools 4. I will check the links you provided (THANKS!!) and report back.

14

That’s the one.

My computer gives me that warning if it makes you feel more comfortable.

Cool beans!

15

Hi. So I did the following:

writeLines(‘PATH=“{RTOOLS40_HOME}\\usr\\bin;{PATH}”’, con = “~/.Renviron”)

and verified the path with"Sys.which(“make”)
which yeilded “C:\rtools40\usr\bin\make.exe”

I also was able to load a package from source using

install.packages(“jsonlite”, type = “source”)

So according to the link above I am good to go… however, I still get the warning message (that you also get) when I compile my Stan code AND I still get the error message when I try to sample from the compiled.

Error: cannot allocate vector of size 19536.6 Gb

I would think there is some error in my code if Jonah Gabry had not already tried and succeeded in compiling and sampling from this exact code.

So I have traded one error for another and cannot sample. Gosh I really appreciate your guidance here. Is there anything else I can try?

Thanks

16

I’m pretty much out of ideas.

Could you try renaming your model to something else? Like if it is “fido.stan”, rename it to “spot.stan” and try to compile that.

rstan tries to avoid recompiling stuff it has already compiled. By renaming the file we force rstan to do a fresh compile.

Someone reported something like that here: Cannot resolve the "cannot allocate vector of size … " error - #59 by mauthier

17

Well your ideas worked pretty well @bbbales2!!

I made minor change to my Stan code which caused Stan to re-compile it and then it ran fine :-) :-)

So thanks to you and @jonah and @bgoodri I’m OK (at least right now). It is great to have some giants to stand on. Now I can get back to work.

Thank you thank you thank you!!

Dave

2 Likes