RStan crashing with large files on AWS

I’ve been running some stan models using Rstan on an AWS EC2 instance (using Louise Aslett’s AMI: http://www.louisaslett.com/RStudio_AMI/) to model wages. I’ve been running fairly complicated models and everything had been working fairly reasonably until I tried to substantially up the sample size above 500,000 or so. At that point I started getting an error that said “The previous R session was abnormally terminated due to an unexpected crash.”

Based on previous forum questions, I thought that this might have been due to insufficient RAM, so I moved to an AWS instance with almost a TB of ram, but this didn’t solve the problem (and based on memory usage while the code is running, it doesn’t seem that I’m even approaching using all the RAM even on instance’s with less RAM).

I cut most model features to generate something incredibly simple, and I still found this problem once the sample got much about 500,000.

I’ve put together a minimum reproducible example on github (with an EXTREMELY simplified version of the model that still generates the error):

Does anyone have any thoughts on this or advice for things to look into?

Thanks!
Alex

One possibility is that R, not the instance is running out of memory if you have a lot of parameters. In your case, I would suspect alphaJ1 to have more elements when the sample grows. If that is the case, there are a couple of things you could try.

  • If you don’t particularly care for alphaJ1, you can not save it by using the pars argument in the stan function.
  • I think you can stream the parameter draws instead of keeping them in R memory by using the sample_file argument.
  • You can switch over to cmdstan which does not suffer from R’s limitations and streams the parameter draws by default.

If this isn’t a RAM problem, rstan has some long-standing stability issues with the I/O that you’re not going to be able to work around yourself. CmdStan works fine in these situations so unfortunately that’s probably the way to go. We do know how to fix them technically but it would require changing how rstan uses R’s and Rcpp’s data structures and last I checked it wasn’t in the cards.

1 Like