# Feedback on BRMS multivariate multilevel analysis for article

**URL:** <https://discourse.mc-stan.org/t/feedback-on-brms-multivariate-multilevel-analysis-for-article/6094>\
**Category:** brms\
**Created:** [October 19, 2018, 12:13pm UTC](https://discourse.mc-stan.org/t/feedback-on-brms-multivariate-multilevel-analysis-for-article/6094 "2018-10-19T12:13:00Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kevin\_Ackermans](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/kevin_ackermans/32/2870_2.png) [@Kevin\_Ackermans](https://discourse.mc-stan.org/u/Kevin_Ackermans)\
**Post date:** [October 19, 2018, 12:13pm UTC](https://discourse.mc-stan.org/t/feedback-on-brms-multivariate-multilevel-analysis-for-article/6094/1 "2018-10-19T12:13:00Z")

</div>

Dear Users,

I have a problem and hope there is someone here that can help.  
I have written an article that uses a BRMS multivariate multilevel analyses to track mental model growth of three separate complex skills in three conditions, acros 2 schools over 3 time points.  
However, my supervisors are not Bayesians and are frankly a little scared.  
Is there anyone here that can give me feedback, me and my supervisors could finally get on with our lives.

Thank you very much, i hope someone can find the time

---

<div class="post-metadata">

**Author:** ![paul.buerkner](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/paul.buerkner/32/3303_2.png) [@paul.buerkner](https://discourse.mc-stan.org/u/paul.buerkner)\
**Post date:** [October 19, 2018, 1:10pm UTC](https://discourse.mc-stan.org/t/feedback-on-brms-multivariate-multilevel-analysis-for-article/6094/2 "2018-10-19T13:10:35Z")

</div>

Ah those supervisors ;-)

I am not sure what kind of feedback do you have in mind? Reading the whole paper seems a little bit too much.

---

<div class="post-metadata">

**Author:** ![Kevin\_Ackermans](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/kevin_ackermans/32/2870_2.png) [@Kevin\_Ackermans](https://discourse.mc-stan.org/u/Kevin_Ackermans)\
**Post date:** [October 19, 2018, 1:35pm UTC](https://discourse.mc-stan.org/t/feedback-on-brms-multivariate-multilevel-analysis-for-article/6094/3 "2018-10-19T13:35:00Z")

</div>

Some feedback on the just the **analysis and result** section would be great.  
I used the WAMBS checklist paper to make sure i covered all of my bases so:

- Did i forget to report anything?
- Did i report something incorrectly?
- Wat are the things you would / i need to change before publishing.

The main goal here is to give my supervisors some security, as they dont know what a prior, Markov chain, Parento K or r2 are. :)

---

<div class="post-metadata">

**Author:** ![matti](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/matti/32/13631_2.png) [@matti](https://discourse.mc-stan.org/u/matti)\
**Post date:** [October 19, 2018, 1:40pm UTC](https://discourse.mc-stan.org/t/feedback-on-brms-multivariate-multilevel-analysis-for-article/6094/4 "2018-10-19T13:40:17Z")

</div>

How about posting the paper as a preprint on e.g. [OSF](https://osf.io) or psyarxiv? It would be easier to provide feedback if it was available–and I’d be happy to take a look (but agree with Paul that reading the whole paper might be a lot to ask for.) I think OSF has some kind of commenting function now so that’s a plus as well.

---

<div class="post-metadata">

**Author:** ![Kevin\_Ackermans](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/kevin_ackermans/32/2870_2.png) [@Kevin\_Ackermans](https://discourse.mc-stan.org/u/Kevin_Ackermans)\
**Post date:** [October 19, 2018, 2:05pm UTC](https://discourse.mc-stan.org/t/feedback-on-brms-multivariate-multilevel-analysis-for-article/6094/5 "2018-10-19T14:05:35Z")

</div>

Hei!

Well, OSF did not let you edit end gdocs messed with the layout so here is a word link that should work. I only posted the part that needs a Bayesian eye ;)

[https://1drv.ms/w/s!AumTyFWX-WDOgQUSYlpYMwrhckxE](https://1drv.ms/w/s!AumTyFWX-WDOgQUSYlpYMwrhckxE)

---

<div class="post-metadata">

**Author:** ![matti](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/matti/32/13631_2.png) [@matti](https://discourse.mc-stan.org/u/matti)\
**Post date:** [October 19, 2018, 4:44pm UTC](https://discourse.mc-stan.org/t/feedback-on-brms-multivariate-multilevel-analysis-for-article/6094/6 "2018-10-19T16:44:55Z")

</div>

There’s a lot of detail there, but right off the bat I would suggest that you describe the model in more detail (right now you spend a lot [too much] space on things like convergence, but I didn’t see what the response variables and predictor variables were, and what the model actually is).

I see that you have compared the same model to itself across different numbers of samples. There’s no reason to do that.

Generally, I would spend quite a bit of time on clarity of presentation.

---

<div class="post-metadata">

**Author:** ![avehtari](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/avehtari/32/5935_2.png) [@avehtari](https://discourse.mc-stan.org/u/avehtari)\
**Post date:** [October 19, 2018, 8:24pm UTC](https://discourse.mc-stan.org/t/feedback-on-brms-multivariate-multilevel-analysis-for-article/6094/7 "2018-10-19T20:24:19Z")

</div>

Additional quick comments

- did you really use The Gelman and Rubin (1992) diagnostic or the more recent version of split-Rhat diagnostic implemented in Stan (and then the correct reference would be BDA3)?
- 130 000 iterations is awfully lot for dynamic HMC in Stan. What is n\_eff? Later tables show “Eff.Sample” which I guess is n\_eff. If n\_eff is 10 000 with 4x80000 post-warmup iterations , you may have problems with your model.
- You are showing way too many digits of LOOIC. Given SE you would need to show max 3 digits, and thus there is no statistical difference between 4000 and 130000 iterations.
- It would be good to include some posterior predictive checking (see, e.g. ppc in bayesplot)

---

<div class="post-metadata">

**Author:** ![Kevin\_Ackermans](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/kevin_ackermans/32/2870_2.png) [@Kevin\_Ackermans](https://discourse.mc-stan.org/u/Kevin_Ackermans)\
**Post date:** [October 20, 2018, 7:19am UTC](https://discourse.mc-stan.org/t/feedback-on-brms-multivariate-multilevel-analysis-for-article/6094/8 "2018-10-20T07:19:48Z")

</div>

Thank you Avi & Matti,  
I’m very glad you found the time to give it a quick look!