K-Fold with brm_multiple model?

Interfaces brms
1

Hi,

I’m trying to compare two models fitted on imputed data.
I’ve used mice to impute 5 datasets and used brm_multiple to fit the models.
Everything fine, until I try to perform a K-fold on these models. Then I get an error saying that newdata must be a list of data.frames.
I’ve tried to pass manually a list of data.frames by newdata argumen but I still get the same error.

Is there a way to run K-fold on brm_multiple models?
If not, can you please suggest a way to compare two models?

Thanks!
pi

Please also provide the following information in addition to your question:

  • Operating System: Win 10
  • brms Version: 2.8.7
2

See https://github.com/paul-buerkner/brms/issues/652

1 Like
3

Great! Thanks!

4

I see that interpretation can be tricky. Can you please suggest a better way to compare the models (like test against null?).

Thanks!

5

I don’t know yet. We are working on that but at the moment I would be especially cautious with all cross-valdiation like methods.

6

Thanks for the quick reply :)

I’ll try to find something in the literature.
In the meanwhile, I’ll check that the interpretation of the model makes sense within the context.

Thanks!

7

I’ve installed the v. 2.8.0 but it’s still giving me the same error.
Should I fit the model again with the new version?

Thanks!

8

You need to install the github version of brms. See https://github.com/paul-buerkner/brms for instructions.

9

Ok, I’ll try that :)

EDIT: now it seems to work. So basically it will use only the first imputed dataset.
I’ll test swapping the order to check how much the results vary across the imputed datasets.