Hi everyone, So I’m having an issue with stan_lmer. I’m trying to run a model with about 200k observations and around 435 random intercepts. The exact model is of the form stan_lmer(y ~ X + (1|id)) The model doesn’t progress after hours. I’ve tried running the model on a subset of the data. If I …

Morning, Can you post your full model with the number of chains, iters, and such? Thanks.

For something that simple yet with such large data, you probably want to use cmdstanr and write the model directly: data{ int nXY; //number of observations in X/Y vector[nXY] Y ; // Y (outcome) observations vector[nXY] X ; // X (covariate) observations int nZ ; // number of levels in Z int<low…

For that much data with that simple a model, do you really need to do MCMC? Does using vb give a reasonable answer?

At the moment brms is likely to scale to larger datasets a little better than rstanarm, especially if you use the cmdstanr backend. Additionally, with the cmdstanr backend you can use within-chain parallelisation via the threads argument. Also, if you have a discrete GPU in your system you can use …

I have a very similar situation to OP. I haven’t noticed much difference between brms with cmdstanr vs rstan, but I have found using threads=threading(2) tends to slow down my brm(y ~ a + b + c + (1|id), chains=4, cores=4) sampling by a factor of 2-3 on a 12-physical-core Ryzen. [image] andrjoh…

[image] abs: What is the main difference between writing that model manually and using brms to generate one, such that manually writing it would make it perform faster? Nothing directly, but you have more control (ex. for the OPs data, they probably want a “centered” parameterized model, and …

Are people actually using MCMC on models like y ~ a + b + c + (1|id) with n>10k (or >100k) in practice (coded directly in Stan or otherwise)? It seems like it would take days or weeks, at least using the techniques I’ve tried so far (namely cores=4, threads=threading(2), and noncentered parametriz…

That’s about the scale of a dataset I’m currently working on, except I’m doing: x ~ (a+b+c+d+e+f+g)^3 + (1+(a+b+c+d+e+f+g)^3 | q | id) y ~ (a+b+c+d+e+f+g)^3 + (1+(a+b+c+d+e+f+g)^3 | q | id) z ~ (a+b+c+d+e+f+g)^3 + (1+(a+b+c+d+e+f+g)^3 | q | id) But aside from over-regularization of the correlat…

Stan_lmer really slow

Modeling

mike-lawrence May 13, 2021, 11:45pm 8

Note that if you use raw stan, you can do things like this and this, whereas I’m not sure that brms includes those performance tweaks.

Topic		Replies	Views
Is it possible to run Bayesian hierarchical model with 10million observations? General rstan , hierarchical-model , brms	20	3192	June 28, 2021
Stan_glmer speed for large sample sizes rstanarm	4	2213	April 19, 2021
Scalability of bayesian glmm Modeling techniques , performance	7	1414	January 15, 2019
STAN scaling limitations General mixture	15	1705	February 18, 2021
Code takes too long to run, despite small dataset size Modeling	7	746	April 11, 2022

Stan_lmer really slow

Related topics