# Stan\_lmer really slow

**URL:** <https://discourse.mc-stan.org/t/stan-lmer-really-slow/20964>\
**Category:** Modeling\
**Created:** [February 24, 2021, 1:48pm UTC](https://discourse.mc-stan.org/t/stan-lmer-really-slow/20964 "2021-02-24T13:48:24Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![adamramey](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/adamramey/32/4419_2.png) [@adamramey](https://discourse.mc-stan.org/u/adamramey)\
**Post date:** [February 24, 2021, 1:48pm UTC](https://discourse.mc-stan.org/t/stan-lmer-really-slow/20964/1 "2021-02-24T13:48:24Z")

</div>

Hi everyone,

So I’m having an issue with stan\_lmer. I’m trying to run a model with about 200k observations and around 435 random intercepts. The exact model is of the form

stan\_lmer(y ~ X + (1|id))

The model doesn’t progress after hours. I’ve tried running the model on a subset of the data. If I choose samples of 1,000, 5,000, or 10,000, it runs pretty quickly. Once I get above 10,000, it chugs.

I’ve tried centering the variables - no dice.

Any advice?

---

<div class="post-metadata">

**Author:** ![Ara\_Winter](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ara_winter/32/4284_2.png) [@Ara\_Winter](https://discourse.mc-stan.org/u/Ara_Winter)\
**Post date:** [February 24, 2021, 2:45pm UTC](https://discourse.mc-stan.org/t/stan-lmer-really-slow/20964/2 "2021-02-24T14:45:17Z")

</div>

Morning,

Can you post your full model with the number of chains, iters, and such? Thanks.

---

<div class="post-metadata">

**Author:** ![mike-lawrence](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mike-lawrence/32/59_2.png) [@mike-lawrence](https://discourse.mc-stan.org/u/mike-lawrence)\
**Post date:** [February 24, 2021, 7:50pm UTC](https://discourse.mc-stan.org/t/stan-lmer-really-slow/20964/3 "2021-02-24T19:50:33Z")

</div>

For something that simple yet with such large data, you probably want to use cmdstanr and write the model directly:

```stan
data{
 int nXY; //number of observations in X/Y
 vector[nXY] Y ; // Y (outcome) observations
 vector[nXY] X ; // X (covariate) observations
 int nZ ; // number of levels in Z
 int<lower=1,upper=nZ> Z[nXY] ; // Z-label for each outcome in Y
  // centered: binary toggle for intercept centered/non-centered parameterization
  int<lower=0,upper=1> centered ;
}
parameters{
  real<lower=0> noise ; //measurement noise scale
  real betaX ; // effect of the covariate X
  real mu ; // intercept
  real<lower=0> sigma ; // scale of variability among levels of Z
  vector<
    offset = (centered ? 0 : mu)
    , multiplier = (centered ? 1 : sigma)
  >[nZ] Zval ; // value for each level of Z
}
model{
  // Priors
  noise ~ weibull(2,1) ; //must be changed to reflect domain expertise 
  betaX ~ std_normal() ; //must be changed to reflect domain expertise 
  mu ~ std_normal() ; //must be changed to reflect domain expertise
  sigma ~ std_normal() ; //must be changed to reflect domain expertise
  //hierarchical prior for Zval:
  zVal ~ normal( mu, sigma ) ;
  //likelihood (can be sped up via reduce_sum if lots of cores available!)
  Y ~ normal( zVal[Z] + betaX*X, noise ) ;
}

```

As noted in the final comment, if you have lots of CPU cores available, you can parallelize the final likelihood computation using the `reduce_sum` framework.

---

<div class="post-metadata">

**Author:** ![hhau](https://avatars.discourse-cdn.com/v4/letter/h/2bfe46/32.png) [@hhau](https://discourse.mc-stan.org/u/hhau)\
**Post date:** [February 24, 2021, 7:56pm UTC](https://discourse.mc-stan.org/t/stan-lmer-really-slow/20964/4 "2021-02-24T19:56:11Z")

</div>

For that much data with that simple a model, do you really need to do MCMC? Does using [vb](https://mc-stan.org/rstan/reference/stanmodel-method-vb.html) give a reasonable answer?

---

<div class="post-metadata">

**Author:** ![andrjohns](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/andrjohns/32/15297_2.png) [@andrjohns](https://discourse.mc-stan.org/u/andrjohns)\
**Post date:** [February 25, 2021, 7:18am UTC](https://discourse.mc-stan.org/t/stan-lmer-really-slow/20964/5 "2021-02-25T07:18:05Z")

</div>

At the moment `brms` is likely to scale to larger datasets a little better than `rstanarm`, especially if you use the [cmdstanr](https://mc-stan.org/cmdstanr/articles/cmdstanr.html) backend.

Additionally, with the cmdstanr backend you can use within-chain parallelisation via the `threads` argument. Also, if you have a discrete GPU in your system you can use this to accelerate your models

---

<div class="post-metadata">

**Author:** ![abs](https://avatars.discourse-cdn.com/v4/letter/a/c2a13f/32.png) [@abs](https://discourse.mc-stan.org/u/abs)\
**Post date:** [May 13, 2021, 11:10pm UTC](https://discourse.mc-stan.org/t/stan-lmer-really-slow/20964/6 "2021-05-13T23:10:30Z")

</div>

I have a very similar situation to OP.

I haven’t noticed much difference between brms with cmdstanr vs rstan, but I have found using `threads=threading(2)` tends to _slow down_ my `brm(y ~ a + b + c + (1|id), chains=4, cores=4)` sampling by a factor of 2-3 on a 12-physical-core Ryzen.

> [@andrjohns](#):
>
> if you have a discrete GPU in your system you can use this to accelerate your models

Is [this thread](https://discourse.mc-stan.org/t/gpu-functions-in-rstan/13722/7) the best reference on using GPU parallelism?

> [@mike-lawrence](#):
>
> For something that simple yet with such large data, you probably want to use cmdstanr and write the model directly

What is the main difference between writing that model manually and using brms to generate one, such that manually writing it would make it perform faster?

---

<div class="post-metadata">

**Author:** ![mike-lawrence](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mike-lawrence/32/59_2.png) [@mike-lawrence](https://discourse.mc-stan.org/u/mike-lawrence)\
**Post date:** [May 13, 2021, 11:43pm UTC](https://discourse.mc-stan.org/t/stan-lmer-really-slow/20964/7 "2021-05-13T23:43:12Z")

</div>

> [@abs](#):
>
> What is the main difference between writing that model manually and using brms to generate one, such that manually writing it would make it perform faster?

Nothing directly, but you have more control (ex. for the OPs data, they probably want a “centered” parameterized model, and I’m not sure that brms makes it easy to switch from non-centered, which I presume is its default). I’m probably just biased by knowing raw Stan well and brms not-well.

---

<div class="post-metadata">

**Author:** ![mike-lawrence](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mike-lawrence/32/59_2.png) [@mike-lawrence](https://discourse.mc-stan.org/u/mike-lawrence)\
**Post date:** [May 13, 2021, 11:45pm UTC](https://discourse.mc-stan.org/t/stan-lmer-really-slow/20964/8 "2021-05-13T23:45:50Z")

</div>

Note that if you use raw stan, you can do things like [this](https://discourse.mc-stan.org/t/case-study-preview-speeding-up-stan-by-reducing-redundant-computation/14228) and [this](https://discourse.mc-stan.org/t/rows-dot-product-is-intended-as-a-shorthand-rather-than-compute-optimization-right/20798/11), whereas I’m not sure that brms includes those performance tweaks.

---

<div class="post-metadata">

**Author:** ![abs](https://avatars.discourse-cdn.com/v4/letter/a/c2a13f/32.png) [@abs](https://discourse.mc-stan.org/u/abs)\
**Post date:** [May 13, 2021, 11:47pm UTC](https://discourse.mc-stan.org/t/stan-lmer-really-slow/20964/9 "2021-05-13T23:47:11Z")

</div>

Are people actually using MCMC on models like

```
y ~ a + b + c + (1|id)

```

with n\>10k (or \>100k) in practice (coded directly in Stan or otherwise)? It seems like it would take days or weeks, at least using the techniques I’ve tried so far (namely `cores=4`, `threads=threading(2)`, and noncentered parametrization).

---

<div class="post-metadata">

**Author:** ![mike-lawrence](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mike-lawrence/32/59_2.png) [@mike-lawrence](https://discourse.mc-stan.org/u/mike-lawrence)\
**Post date:** [May 14, 2021, 12:04am UTC](https://discourse.mc-stan.org/t/stan-lmer-really-slow/20964/10 "2021-05-14T00:04:40Z")

</div>

That’s about the scale of a dataset I’m currently working on, except I’m doing:

```
x ~ (a+b+c+d+e+f+g)^3 + (1+(a+b+c+d+e+f+g)^3 | q | id)
y ~ (a+b+c+d+e+f+g)^3 + (1+(a+b+c+d+e+f+g)^3 | q | id)
z ~ (a+b+c+d+e+f+g)^3 + (1+(a+b+c+d+e+f+g)^3 | q | id)

```

But aside from [over-regularization of the correlations](https://discourse.mc-stan.org/t/feedback-requested-on-possible-subtle-pathology-of-typical-multivariate-specification-or-maybe-bug/21388) that I’m still working on, it at least fits in a few hours. Mind you, all my variables are dichotomous and thereby benefit greatly from the reduced redundant computation trick I linked above.

Oh, and I guess it helps that while the raw number of observations is in the 30k+ range, there are only 60 levels of id.

---

<div class="post-metadata">

**Author:** ![mike-lawrence](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mike-lawrence/32/59_2.png) [@mike-lawrence](https://discourse.mc-stan.org/u/mike-lawrence)\
**Post date:** [May 14, 2021, 12:07am UTC](https://discourse.mc-stan.org/t/stan-lmer-really-slow/20964/11 "2021-05-14T00:07:49Z")

</div>

[See also here](https://discourse.mc-stan.org/t/psa-a-case-where-stan-blows-lme4-glmer-away/20067).