Feedback on BRMS multivariate multilevel analysis for article

Interfaces brms
1

Dear Users,

I have a problem and hope there is someone here that can help.
I have written an article that uses a BRMS multivariate multilevel analyses to track mental model growth of three separate complex skills in three conditions, acros 2 schools over 3 time points.
However, my supervisors are not Bayesians and are frankly a little scared.
Is there anyone here that can give me feedback, me and my supervisors could finally get on with our lives.

Thank you very much, i hope someone can find the time

2

Ah those supervisors ;-)

I am not sure what kind of feedback do you have in mind? Reading the whole paper seems a little bit too much.

3

Some feedback on the just the analysis and result section would be great.
I used the WAMBS checklist paper to make sure i covered all of my bases so:

  • Did i forget to report anything?
  • Did i report something incorrectly?
  • Wat are the things you would / i need to change before publishing.

The main goal here is to give my supervisors some security, as they dont know what a prior, Markov chain, Parento K or r2 are. :)

4

How about posting the paper as a preprint on e.g. OSF or psyarxiv? It would be easier to provide feedback if it was available–and I’d be happy to take a look (but agree with Paul that reading the whole paper might be a lot to ask for.) I think OSF has some kind of commenting function now so that’s a plus as well.

5

Hei!

Well, OSF did not let you edit end gdocs messed with the layout so here is a word link that should work. I only posted the part that needs a Bayesian eye ;)

https://1drv.ms/w/s!AumTyFWX-WDOgQUSYlpYMwrhckxE

6

There’s a lot of detail there, but right off the bat I would suggest that you describe the model in more detail (right now you spend a lot [too much] space on things like convergence, but I didn’t see what the response variables and predictor variables were, and what the model actually is).

I see that you have compared the same model to itself across different numbers of samples. There’s no reason to do that.

Generally, I would spend quite a bit of time on clarity of presentation.

1 Like
7

Additional quick comments

  • did you really use The Gelman and Rubin (1992) diagnostic or the more recent version of split-Rhat diagnostic implemented in Stan (and then the correct reference would be BDA3)?
  • 130 000 iterations is awfully lot for dynamic HMC in Stan. What is n_eff? Later tables show “Eff.Sample” which I guess is n_eff. If n_eff is 10 000 with 4x80000 post-warmup iterations , you may have problems with your model.
  • You are showing way too many digits of LOOIC. Given SE you would need to show max 3 digits, and thus there is no statistical difference between 4000 and 130000 iterations.
  • It would be good to include some posterior predictive checking (see, e.g. ppc in bayesplot)
2 Likes
8

Thank you Avi & Matti,
I’m very glad you found the time to give it a quick look!