If I would like to set the chain of the brms model as 4, and use brm_multiple function to run the same multi-level model on 10 imputed datasets, and I need run the brm_multiple function for other different imputed dataset many times, Can I use the brm_multiple and parallel at same time? For example, I have 48 cores, how to set the CPU is rational?
There’s an old issue on GitHub that might point you in the right direction:
Both
kfoldandbrm_multiplehave future automatically activated to run models in parallel, whilebrmcan run chains in parallel iffuture = TRUE. That is you have two layers of futures on top of each other for the former two functions if you also activate future in brm. I am not sure how future handles this situation exactly, but it is definitely built to deal with it (and it’s surely documended somewhere).
I read this to mean that what you want to do is possible, but probably not without some additional effort from you. I would try setting up a nested topology with the future package. Your upper layer could make use of 10 cores (1/imputed dataset) with each lower layer using 4 cores (1/chain). This uses 40 cores, leaving 8 idle. Maybe not ideal, but at least a step in the right direction. I think you may need recreate some of the behavior of brms::brm_multiple by calling brms::brm() for each imputation, then using brms::combine_models() to combine the output appropriately given the nested structure of your output.
1 Like
stantargets also has parallelization capabilities and could be an alternative approach. I don’t have experience with what you’d need to do, except to say that you could export the Stan code from brms using the make_stancode() function.