Avoid loo memory error by removing "ineffective samples" after sampling?

Interfaces brms
1

I fit a model using brms. If I set iter high, running loo fills memory, ultimately failing with the error

`Error: cannot allocate vector of size 5.6 Gb`.

I have 32GB of physical RAM and 24GB cache. loo completes just fine if iter is lower, but then the effective sample size is too low. Running loo(fit, pointwise=TRUE) never completes (I stopped it after 8 hours on four 3.2 GHz CPUs).

The effective samples size for some parameters is ~15 times lower than the total number of samples. So I wonder if the memory problem could be avoided by removing “ineffective samples”? A simple solution would be thinning, but perhaps there are more clever solutions to keep even more effective samples.

So my question is: is there a way to do loo here? Thanks in advance!

2

Sounds like you should just be thinning either at the time of model fitting or at the time of computing loo by using argument nsamples.

3

Yes! loo(fit, nsamples=4500) did the trick.Thank you!

I struggle to find documentation for nsamples in loo. I only found this old one from brms 2.1 docs. For future googlers, nsamples has to be a multiple of the number of chains. If not, it fails with

 Error: Number of rows in matrix not equal to length(chain_id).

The sampling is not deterministic, so for (very) low nsamples you will get slightly different statistics for each run.

1 Like
4

The doc is in extract_draws which you should be guided to in the loo doc but it turns out this is not the case. I will fix it.

2 Likes