Error in unserialize(socklist[[n]]) : error reading from connection

Interfaces RStan
1

Hi!

I have a model that runs with 1/6th of my data. However, when I use all of the data I get the error code:
Error in unserialize(socklist[[n]]) : error reading from connection

When I run the model with only 1/6 of the data it only uses 17GB RAM to run. When I run the model with all of the data it uses 20GB memory before it then fails and returns the error. However, I have 200GB of RAM with no higher memory limits in RStudio, on a Linux machine (versions: RStan 2.18.2 and R 3.2.1). The full dataset is 3000 lines.

When I move the transformed parameters to the model block it runs fine with all of the data. However, I need to have access to these parameters so not storing them to memory is not an ideal solution.

Any ideas or insights would be greatly appreciated!

(I have read other posts about the error code and I have tried but they haven’t helped as far as I can tell:

  • deleting the Makevars and Makevars.win file
  • chains = 1 but R crashes
  • fewer iterations
  • I increased the RAM to 600GB).
2

I would save the draws to the disk by specifying the sample_file argument. You can not keep them in memory by specifying pars = "". Then you just have to get the draws off the disk without crashing R. The rstan_read_csv function may work or you may have to use data.table::fread or something.

3

I have the same problem with brms. rstan and rstanarm work, but brms with one core crashes R and with more cores Error in unserialize(socklist[[n]]) : error reading from connection. This shouldn’t be any memory problem as it fails also with ten observation logistic regression.

I have

  • rstan (Version 2.19.2, GitRev: 2e1f913d3ca3)
  • StanHeaders * 2.18.1-10
  • ‘brms’ package (version 2.9.0).

I have no idea how to debug this further.

4

Something coming in via the .RData file on startup?

5

I searched all .Rdata files in all directories, there was one which I removed and restarted R and the problem persists with both RStudio and R without studio.

6

If you use brms::make_stancode and brms::make_standata to create the inputs to stan does it work?

7 
library(rstan)
library(brms)
SEED <- 1655
set.seed(SEED)
n <- 1e4
x <- rnorm(n)
a <- 2
b <- 3
sigma <- 100
y <- a + b*x + sigma*rnorm(n)
fake <- data.frame(x, y)
sc<-make_stancode(y ~ x, data=fake)
sd<-make_standata(y ~ x, data=fake)
sm<-stan_model(model_code=sc)
fits<-sampling(sm, data=sd)

Works in fresh R session, but if I add

library(rstanarm)

then it crashes (I think I did try without rstanarm being loaded, previously with no success, but I may have made mistakes). I guess I need to reinstall rstanarm.

I’ll report tomorrow whether reinstalling rstanarm helps

8

It seems reinstalling rstanarm did help. It was strange that rstanarm did work, but loading it made brms to crash.

9

Thank you so much for your quick reply!

It is has started to sample without the error message - hopefully should know if it fully works in the next 5 hours or so! I have used pars=NA, include = FALSE.

Although I would like to understand why this is necessary because the transformed parameters x and P that are causing this memory issue are quite small:
x is an 11x3000 matrix and P is an 11x11x3000 array. If I am doing 1000 iterations, am I using:
8x11x11x3000x1000(iterations) = 3GB for P and 0.3GB for x? Not that much memory compared to what I have available… Is there are limit to the size of the parameters in STAN?

Thank you for you help :)