Error message when using compare_models function for loo objects of hierarchical models

Interfaces rstanarm
1

Hi all - first time poster, long time admirer.

I’m trying to compare hierarchical models using the compare_models function in rstanarm. I’ve fit the models using stan_glmer and applied leave-one-out using the loo function of the loo package. All models use the same dataset, same # observations, same response variable, same family of distributions, same transformation applied to the response variable.

When I call compare_models to compare all models, I receive the following error message:
“Error: Not all models have the same y variable.”

By leaving out the most complex model, I was able to run compare_models successfully (no error message). So that’s the model that’s causing the problems.

I checked this page (https://github.com/stan-dev/rstanarm/blob/master/tests/testthat/test_loo.R), but I have not committed one of the example “sins” listed under this error message.

Other (maybe) helpful info:
I did end up having use the “k_threshold=0.7” argument to two of my models. The problematic model (the most complex one) is one of the models that uses the k_threshold argument. The only other difference between the problematic model and the others is that I use a different “random effect” term in it. Here are two of my model specifications - one that does NOT throw an error message (model 1) and the other that DOES throw an error message (model 2):

model 1
stan_glmer(growth ~ diam1 + gap +
dens.c + dens.h +
mycBAc5 + mycBAh5 +
(1 | plot) + (1 | species),
prior = student_t(df=7, location=0, autoscale=TRUE),
prior_intercept = student_t(df=7, location=0, autoscale=TRUE),
data = dat)

model 2 (PROBLEMATIC)
stan_glmer(growth ~ diam1 + gap +
dens.c + dens.h +
BAc5 + BAh5 +
(BAc5+BAh5 | species) + (1 | plot),
prior = student_t(df=7, location=0, autoscale=TRUE),
prior_intercept = student_t(df=7, location=0, autoscale=TRUE),
data = dat)

I have quite complex models that take a while to run, so not sure it will be helpful to include my data for reproducibility (they will take hours to run). My hope is that my error is apparent from my description, but if not, I can try to link to my .RData file … somehow.

Greatly appreciate any guidance. Apologize if I’m not including enough info - I can work with responders to provide needed info.

Operating System: Windows 10
Interface Version: rstanarm 2.15.2
Compiler/Toolkit:

2

Did you try without “k_threshold=0.7” argument? Using that argument makes rstanarm to run MCMC for those leave-one-out posteriors and it makes the loo R object different, and it’s possible that we didn’t test it with glmer and this kind of formula. If you can try without that argument and report what happens, that would be helpful.

3

Thanks so much for the quick reply! When I run the loo function without using the k_threshold argument, I get a warning:

“Warning message:
Found 4 observation(s) with a pareto_k > 0.7. We recommend calling ‘loo’ again with argument ‘k_threshold = 0.7’ in order to calculate the ELPD without the assumption that these observations are negligible. This will refit the model 4 times to compute the ELPDs for the problematic observations directly.”

If I ignore the warning and run compare_models, it works. How terrible is it to ignore the warning if I want to publish these results?

Weirdly, there is one other model that resulted in the same warning message (with one outlier instead of four), so I used the k_threshold argument on it, and the compare_models function works for that model. Do you happen to know why?

4

I’m guessing there is a bug in our code. Can you make a small data set, which you could share, so we can test?
Also it helps if you can figure out minimal model formula which fails. If making data smaller and model simpler make k<0.7, then just run with so low k_threshold that at least 2 k’s are above the threshold, so we can test this.

Meanwhile when you are waiting us to fix this, run loo with problematic model without k_threshold and with k_threshold. If the results are in loo1 and loo2, then compare
loo1$pointwise[,“elpd_loo”] and loo2$pointwise[,“elpd_loo”]
if the difference in pointwise elpd_loo’s for those which had k>0.7 are small then you can publish the results. If the pointwise differences for those are large, you can compute comparison yourself by looking at the equations in http://link.springer.com/article/10.1007/s11222-016-9696-4 (preprint https://arxiv.org/abs/1507.04544)

5

I greatly appreciate your help - thanks so much for the advice on how to move forward with publishing the results!

I’ve been working for some time on reproducing the error message with a smaller dataset with no luck (when using the k_threshold argument for one model but not another model, the compare_models function is working).

Do you have any suggestions? I could save a .RData file with the problematic loo objects into a public Dropbox folder perhaps?

6

Ok. Can you also make an issue in github Issues · stan-dev/loo · GitHub ?
You can briefly describe the problem there and link to this thread, too.

7

Great, I appreciate your flexibility. Here is the link to the public folder: https://www.dropbox.com/sh/jdmjy2w55fd24pm/AABL0Q4Myl9rA38F850AGmUTa?dl=0

I’ll work on creating the issue in github as well.

8

I replied in github issue
https://github.com/stan-dev/loo/issues/54

I’ll summarize here after the problem has been solved.