Bayes factor using brms

Interfaces brms
,
21

Good idea also about simulating the ground truth. I have done that (that’s one of the reasons I came across your very helpful document bc I wanted to check whether others had done the same), but it didn’t really help me understand why bridgesampling threw the errors it throws despite the high number of samples (I’ve used bridgesampling for other projects with similar models, similar amounts of data, and fewer posterior samples, and didn’t run into this problem).

Following @andrewgelman’s suggestion, I have now applied loo to the models, instead. That resulted in a warning (based on the pareto-k threshold) that 253 observations (about 10%) in the model are problematic. I then reran loo with moment_match = T. This ran for about 10 hours on four cores and then return an error message:

Error : $ operator is invalid for atomic vectors
In addition: Warning messages:
1: In doTryCatch(return(expr), name, parentenv, handler) :
  restarting interrupted promise evaluation
2: In doTryCatch(return(expr), name, parentenv, handler) :
  restarting interrupted promise evaluation
3: In doTryCatch(return(expr), name, parentenv, handler) :
  restarting interrupted promise evaluation
4: In doTryCatch(return(expr), name, parentenv, handler) :
  restarting interrupted promise evaluation
5: In doTryCatch(return(expr), name, parentenv, handler) :
  restarting interrupted promise evaluation
6: In parallel::mclapply(X = I, mc.cores = cores, FUN = function(i) loo_moment_match_i_fun(i)) :
  scheduled core 1, 4 encountered error in user code, all values of the job will be affected
Error: Moment matching failed. Perhaps you did not set 'save_all_pars' to TRUE when fitting your model?

But fwiw, save_all_pars is set to T; or rather, under the new syntax of brm: save_pars = save_pars(all = T). So, I am at a loss as to what’s going on. The model with samples is about 400 MB but here’s a link in case anybody is willing to check what they get if they loo it: Dropbox - TFJ_sp_combined_brms.RDS - Simplify your life

22

I would first start with a simpler model (varying intercepts only, plus at most varying slopes for the target parameter), and check if that gets you somewhere. Fitting a “maximal” model under conditions of sparse data means you are just getting back the priors you specified for all those abstract variance components. Then one can try building up to more complex models.

By anova I meant the frequentist likelihood ratio test. I just wanted to know what the estimates of the effects are and how much uncertainty there is (SE) and what the likelihood ratio test says.

1 Like
23

Hello Sir,

Perhaps increase the available memory —that’s if the moment matching is running on remote server with the option to assign more space per nodes/cores. I eventually started associating these errors ( … invalid atomic vector & moment match failed) and warning item 6 — with the algorithm taking up more memory than I assigned on the super computing server I was using. In my case – moment matching LOO eventually needed 40 GB to run uninterrupted.

1 Like
24

Thank you for the pdf link.

25

It’s just amazing to me how hard it is to compute a meaningful Bayes factor for what we might consider to be absolutely bog-standard psycholinguistic data-sets.

3 Likes