I have a problem and hope there is someone here that can help.
I have written an article that uses a BRMS multivariate multilevel analyses to track mental model growth of three separate complex skills in three conditions, acros 2 schools over 3 time points.
However, my supervisors are not Bayesians and are frankly a little scared.
Is there anyone here that can give me feedback, me and my supervisors could finally get on with our lives.
Thank you very much, i hope someone can find the time
Ah those supervisors ;-)
I am not sure what kind of feedback do you have in mind? Reading the whole paper seems a little bit too much.
Some feedback on the just the analysis and result section would be great.
I used the WAMBS checklist paper to make sure i covered all of my bases so:
- Did i forget to report anything?
- Did i report something incorrectly?
- Wat are the things you would / i need to change before publishing.
The main goal here is to give my supervisors some security, as they dont know what a prior, Markov chain, Parento K or r2 are. :)
How about posting the paper as a preprint on e.g. OSF or psyarxiv? It would be easier to provide feedback if it was available–and I’d be happy to take a look (but agree with Paul that reading the whole paper might be a lot to ask for.) I think OSF has some kind of commenting function now so that’s a plus as well.
Well, OSF did not let you edit end gdocs messed with the layout so here is a word link that should work. I only posted the part that needs a Bayesian eye ;)
There’s a lot of detail there, but right off the bat I would suggest that you describe the model in more detail (right now you spend a lot [too much] space on things like convergence, but I didn’t see what the response variables and predictor variables were, and what the model actually is).
I see that you have compared the same model to itself across different numbers of samples. There’s no reason to do that.
Generally, I would spend quite a bit of time on clarity of presentation.
Additional quick comments
- did you really use The Gelman and Rubin (1992) diagnostic or the more recent version of split-Rhat diagnostic implemented in Stan (and then the correct reference would be BDA3)?
- 130 000 iterations is awfully lot for dynamic HMC in Stan. What is n_eff? Later tables show “Eff.Sample” which I guess is n_eff. If n_eff is 10 000 with 4x80000 post-warmup iterations , you may have problems with your model.
- You are showing way too many digits of LOOIC. Given SE you would need to show max 3 digits, and thus there is no statistical difference between 4000 and 130000 iterations.
- It would be good to include some posterior predictive checking (see, e.g. ppc in bayesplot)
Thank you Avi & Matti,
I’m very glad you found the time to give it a quick look!