Good idea also about simulating the ground truth. I have done that (that’s one of the reasons I came across your very helpful document bc I wanted to check whether others had done the same), but it didn’t really help me understand why bridgesampling threw the errors it throws despite the high number of samples (I’ve used bridgesampling for other projects with similar models, similar amounts of data, and fewer posterior samples, and didn’t run into this problem).
Following @andrewgelman’s suggestion, I have now applied loo to the models, instead. That resulted in a warning (based on the pareto-k threshold) that 253 observations (about 10%) in the model are problematic. I then reran loo with moment_match = T. This ran for about 10 hours on four cores and then return an error message:
Error : $ operator is invalid for atomic vectors
In addition: Warning messages:
1: In doTryCatch(return(expr), name, parentenv, handler) :
restarting interrupted promise evaluation
2: In doTryCatch(return(expr), name, parentenv, handler) :
restarting interrupted promise evaluation
3: In doTryCatch(return(expr), name, parentenv, handler) :
restarting interrupted promise evaluation
4: In doTryCatch(return(expr), name, parentenv, handler) :
restarting interrupted promise evaluation
5: In doTryCatch(return(expr), name, parentenv, handler) :
restarting interrupted promise evaluation
6: In parallel::mclapply(X = I, mc.cores = cores, FUN = function(i) loo_moment_match_i_fun(i)) :
scheduled core 1, 4 encountered error in user code, all values of the job will be affected
Error: Moment matching failed. Perhaps you did not set 'save_all_pars' to TRUE when fitting your model?
But fwiw, save_all_pars is set to T; or rather, under the new syntax of brm: save_pars = save_pars(all = T)
. So, I am at a loss as to what’s going on. The model with samples is about 400 MB but here’s a link in case anybody is willing to check what they get if they loo it: Dropbox - TFJ_sp_combined_brms.RDS - Simplify your life