Dear stan users: I am using Rstan to run a relatively complex model in a cluster. I initially set the wall times to be 8 hours but after 8 hours of waiting, my job got killed and I looked into the log file the job was only 40% complete and exceeded the set wall time. Gradient evaluation took 0.05 …

Note that you can set the sample_file parameter and your samples will be streamed to a file (e.g. even when the computation is interrupted, all/most samples will be stored). You can then use read_stan_csv to read this file in R.

As for the speed - very often problems in speed indicate problems with the model. However anything is hard to debug at this scale. Have you tested the model with small-scale data you created by simulating the exact model you have implemented? (e.g. draw the parameters from their priors and then gene…

thank you for your reply I will give it a try

for setting the sample_file parameter? I am wondering where do I set it? in the parameter block in stan model? or in the model object, ie here stan(file=‘full_state_space_true_classical.stan’, data=ll,chains=4, iter=500, control= list(adapt_delta = 0.9),verbose=FALSE,seed = 123456) sorry, I don’…

Hi Martin: Yes, I have tested a smaller model ( modelling only 1-day of data) it works fine, the model recovers the ‘true’ parameters values that I use to simulate the data and only takes moderate amount of time to run. Only when I extend to the full model, modelling 1-week of data then it gets ver…

[image] Jaslene_Lin: for setting the sample_file parameter? I am wondering where do I set it? Sorry, I was not clear the sample_file is a parameter to the stan and sampling functions in R (check the docs for RStan for more details). As for the speed: it is possible that your data is just too…

Where you’re doing: for(t in 1:TD){ for(i in 1:N){ target +=log_mix(lambda[t,1], normal_lpdf(obs[i,t]| y_overall_mean[t]+mu1[t], sigma[1]), normal_lpdf(obs[i,t]| y_overall_mean[t]+mu2[t], sigma[2])); } } you’re computing both y_overall_mean[t]+mu1[t] …

Oh, I take that back, I see now that you’re only using 6 spline bases, which is well below the 100ish that I found was necessary to see a speedup for the sparse version.

… and I think it is better to save all those N terms in a vector and then add that to target at once. When doing all those tricks, you can watch the startup message from stan where it tells you the # of seconds for doing 1k gradient evaluations. This is a rough estimate of speedup when compared to …

Running model in a HPC and would like to save intermediate outputs

Modeling

mike-lawrence March 10, 2018, 4:04pm 9

As Martin says, you could probably benefit from sparse representations; I have a demo in the context of splines here

Topic		Replies	Views
Using Stan on a computing cluster. Any advice? CmdStan	20	5221	January 10, 2019
How to improve model sampling speed when applied to high-dimension data Modeling fitting-issues , performance , cmdstanr	8	302	August 21, 2025
Stuck at Warmup iteration with no error : CmdStanR CmdStan techniques , fitting-issues	48	3572	April 21, 2020
How to speed up my Stan code? Modeling rstan , fitting-issues	8	784	June 14, 2021
Cmdstanpy, mpi speedup Developers	26	529	November 19, 2024

Running model in a HPC and would like to save intermediate outputs

Related topics