# Summary method slow for large models

**URL:** <https://discourse.mc-stan.org/t/summary-method-slow-for-large-models/18147>\
**Category:** General\
**Created:** [September 23, 2020, 2:56am UTC](https://discourse.mc-stan.org/t/summary-method-slow-for-large-models/18147 "2020-09-23T02:56:00Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![walkerharrison](https://avatars.discourse-cdn.com/v4/letter/w/96bed5/32.png) [@walkerharrison](https://discourse.mc-stan.org/u/walkerharrison)\
**Post date:** [September 23, 2020, 2:56am UTC](https://discourse.mc-stan.org/t/summary-method-slow-for-large-models/18147/1 "2020-09-23T02:56:00Z")

</div>

I’m building a large model (millions of rows, tens of thousands of parameters) and calling the `summary` method after the model is done fitting takes 30-60 minutes.

Technically, I’m doing this in cmdstanr, so I’ll show the simple code below, but I’ve had similar experiences in rstan.

```
mod <- cmdstan_model("...")
fit <- mod$sample(data = ...)
mod_summary <- fit$summary() # this takes surprisingly long

```

Is it the diagnostics that are taking a while? I can’t imagine the means and quantiles would take very long, even with the size of the model. Is there anything I can do to speed it up? Can I opt out of some of the features of the summary if (for now) I’m only interested in the posterior means/medians?

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [September 23, 2020, 3:32am UTC](https://discourse.mc-stan.org/t/summary-method-slow-for-large-models/18147/2 "2020-09-23T03:32:11Z")

</div>

how many chains are you running?

do you want to get means/medians for all variables or just some of them?

cmdstanr feeds the assembled sample to the posterior package’s `summarize_draws` function - [Summaries of draws objects — draws\_summary • posterior](https://mc-stan.org/posterior/reference/draws_summary.html)

> Can I opt out of some of the features of the summary if (for now) I’m only interested in the posterior means/medians?

yes - you should be able to pass in args “mean”, “median”

you could try timing how long calling the `draws` function takes; then immediately call `summarize_draws` - if it’s an I/O bottleneck, the the former will take 30-60 minutes.

and if you’re only interested in select variables, the `draws` function lets you specify which variables to read in; this should speed up I/O and use way less memory - at which point, you can use methods in the posterior package directly.

---

<div class="post-metadata">

**Author:** ![bbbales2](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bbbales2/32/77_2.png) [@bbbales2](https://discourse.mc-stan.org/u/bbbales2)\
**Post date:** [September 23, 2020, 3:39am UTC](https://discourse.mc-stan.org/t/summary-method-slow-for-large-models/18147/3 "2020-09-23T03:39:54Z")

</div>

> [@mitzimorris](#):
>
> the `draws` function lets you specify which variables to read in; this should speed up I/O and use way less memory

I was just messing with a 7k parameter model. `$draws()` was pretty slow until I did this.

I think this only loads the `beta` variable and only does certain processing on it:

```no-highlight
fit$draws("beta") %>%
  summarise_draws(median, mad, rhat, ess_bulk)

```

`summarise_draws` comes from the posterior package ([GitHub - stan-dev/posterior: The posterior R package](https://github.com/stan-dev/posterior))

---

<div class="post-metadata">

**Author:** ![walkerharrison](https://avatars.discourse-cdn.com/v4/letter/w/96bed5/32.png) [@walkerharrison](https://discourse.mc-stan.org/u/walkerharrison)\
**Post date:** [September 23, 2020, 3:49am UTC](https://discourse.mc-stan.org/t/summary-method-slow-for-large-models/18147/4 "2020-09-23T03:49:28Z")

</div>

The model is 4 chains, 1000 warm-up, 1000 post warm-up.  
Unfortunately I need means and medians for basically every variable, but I will try passing in those args.

Do you have any insight into why the `draws` function would be slow – is it not simply reading the .csvs created during the fit? Am I just underestimating the data size because I’ve never actually loaded something that was 100k x 4000?

FWIW, I usually use the `fread` function from the `data.table` package on the unfinished csvs if I’m interested in the samples before the model is done fitting, and while it’s not “fast,” I think it’s a good bit quicker than `$draws`.

---

<div class="post-metadata">

**Author:** ![jonah](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/jonah/32/200_2.png) [@jonah](https://discourse.mc-stan.org/u/jonah)\
**Post date:** [September 23, 2020, 4:37am UTC](https://discourse.mc-stan.org/t/summary-method-slow-for-large-models/18147/5 "2020-09-23T04:37:00Z")

</div>

Internally the draws method is calling `read_cmdstan_csv()` function which currently uses the vroom package to read in the csv. It’s faster than the standard read.csv() but yeah I think it’s slower than fread(). There’s been some related discussion here:

> <https://github.com/stan-dev/cmdstanr/issues/299>
>
> Describe the bug
> CSV read speed drastically degrades when the number of params is large(\>10^5) .
> To Reproduce
> @billgillespie and I ran into a...

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [September 23, 2020, 5:06am UTC](https://discourse.mc-stan.org/t/summary-method-slow-for-large-models/18147/6 "2020-09-23T05:06:36Z")

</div>

> [@walkerharrison](#):
>
> Am I just underestimating the data size because I’ve never actually loaded something that was 100k x 4000?

multiply again by 8 to get the number of bytes - that’s 3.2 GB

---

<div class="post-metadata">

**Author:** ![bbbales2](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bbbales2/32/77_2.png) [@bbbales2](https://discourse.mc-stan.org/u/bbbales2)\
**Post date:** [September 23, 2020, 4:27pm UTC](https://discourse.mc-stan.org/t/summary-method-slow-for-large-models/18147/7 "2020-09-23T16:27:01Z")

</div>

As a workaround, if `fread` is faster, then I think you can:

1. Load up the individual chains with `fread`
2. Glue them together with `posterior::bind_draws(df1, df2, along = "chain")` (so posterior can know they are different chains)
3. Pass that into `posterior::summarize_draws()` to do the summarizing (or do custom summarizing or whatever)

This makes me think it might be handy to have a thin option on our reading functions, but I don’t know how that would work. If the analysis functions start getting choked up you could do some thinning after step 1 above (that’ll make your ESS estimates wonky though, but presumably you don’t need those to do your analysis).

---

<div class="post-metadata">

**Author:** ![walkerharrison](https://avatars.discourse-cdn.com/v4/letter/w/96bed5/32.png) [@walkerharrison](https://discourse.mc-stan.org/u/walkerharrison)\
**Post date:** [October 11, 2020, 11:35pm UTC](https://discourse.mc-stan.org/t/summary-method-slow-for-large-models/18147/8 "2020-10-11T23:35:01Z")

</div>

reviving this thread because I’m now experiencing some long waits for the `summary` method on an `optimize` object. Unless I’m mistaken, there aren’t giant csvs to be read in since we’re optimizing and not sampling – any clue why this method is also slow or ways I might speed it up?  
@jonah @mitzimorris

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [October 12, 2020, 3:28am UTC](https://discourse.mc-stan.org/t/summary-method-slow-for-large-models/18147/9 "2020-10-12T03:28:42Z")

</div>

hi Walker,

is the `optimize` method itself reasonably fast?

why call `summary` on an optimize object?  
the `optimize` method is returning a single vector containing the penalized MLE (i.e., posterior mode).

it looks like CmdStanR hands this single draw off to the `posterior` package’s `summarize-draws` method - that’s all I know - I hope one of the R devs can help out here.

---

<div class="post-metadata">

**Author:** ![walkerharrison](https://avatars.discourse-cdn.com/v4/letter/w/96bed5/32.png) [@walkerharrison](https://discourse.mc-stan.org/u/walkerharrison)\
**Post date:** [October 12, 2020, 3:38am UTC](https://discourse.mc-stan.org/t/summary-method-slow-for-large-models/18147/10 "2020-10-12T03:38:22Z")

</div>

Yes the `optimize` method takes only a few minutes (large model, 300K parameters).

I was using the `summary` method because the [intro to cmdstanr](https://mc-stan.org/cmdstanr/articles/cmdstanr.html) does so:

```
fit_mle <- mod$optimize(data = data_list, seed = 123)
fiit_mle$summary()

```

Although it appears I can just call `mle` to get a vector of the solution.

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [October 12, 2020, 5:20pm UTC](https://discourse.mc-stan.org/t/summary-method-slow-for-large-models/18147/11 "2020-10-12T17:20:48Z")

</div>

how long does it take to get a response via `mle`?  
both `summary` and `mle` read in the csv file. if `summary` is taking way longer than `mle`, this needs to be investigated. (just opened an issue on this: [https://github.com/stan-dev/cmdstanr/issues/315](https://github.com/stan-dev/cmdstanr/issues/315))

---

<div class="post-metadata">

**Author:** ![rok\_cesnovar](https://avatars.discourse-cdn.com/v4/letter/r/7bcc69/32.png) [@rok\_cesnovar](https://discourse.mc-stan.org/u/rok_cesnovar)\
**Post date:** [October 12, 2020, 5:34pm UTC](https://discourse.mc-stan.org/t/summary-method-slow-for-large-models/18147/12 "2020-10-12T17:34:46Z")

</div>

Yeah, this is the same issue as [https://github.com/stan-dev/cmdstanr/issues/299](https://github.com/stan-dev/cmdstanr/issues/299)  
It seems that vroom or how we use vroom has issues with that many parameters ( has issues = runs very slow).

Working on replacing it with fread very soon.

---

<div class="post-metadata">

**Author:** ![zcai](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/zcai/32/10242_2.png) [@zcai](https://discourse.mc-stan.org/u/zcai)\
**Post date:** [May 31, 2021, 12:13am UTC](https://discourse.mc-stan.org/t/summary-method-slow-for-large-models/18147/13 "2021-05-31T00:13:27Z")

</div>

Thanks and nice recommendation. Here is what I tried, reduced draw time from 50s (model only took 23s to fit with 4 chians,1000 warm up and 1000 samples) to 3s, if I just use `fit$draws`

```
        cmdstanfiles <- list()
        for (f in fit$output_files()) {
          cmdstanfiles[[f]] <- fread(cmd= paste0("grep -v '^#' ", f))
        }
        var_of_interest <- cmdstanfiles %>% bind_rows() %>% dplyr::select(starts_with("var_of_interest")) %>% as.matrix()

```