# (brms) Loo pit overlay differences when mu and sigma are in the transformed parameters block and loglik/yrep are in the generated quantities
**URL:** https://discourse.mc-stan.org/t/brms-loo-pit-overlay-differences-when-mu-and-sigma-are-in-the-transformed-parameters-block-and-loglik-yrep-are-in-the-generated-quantities/36813
**Category:** brms
**Tags:** loo
**Created:** [October 3, 2024, 6:42pm UTC](https://discourse.mc-stan.org/t/brms-loo-pit-overlay-differences-when-mu-and-sigma-are-in-the-transformed-parameters-block-and-loglik-yrep-are-in-the-generated-quantities/36813 "2024-10-03T18:42:06Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@Brendan\_Alexander](https://discourse.mc-stan.org/u/Brendan_Alexander)
#### Post date: [October 3, 2024, 6:42pm UTC](https://discourse.mc-stan.org/t/brms-loo-pit-overlay-differences-when-mu-and-sigma-are-in-the-transformed-parameters-block-and-loglik-yrep-are-in-the-generated-quantities/36813/1 "2024-10-03T18:42:06Z")
Hello all, I have been having a problem with my loo-pit diagnostics.
I have a stan model and a brms model that are supposed to be equal. The point is that it’s an exercise in writing stan models and confirming that they’re written correctly by comparing the results to those of brms.
My default assumption is that when model results are different then brms is correct.
Both my stan model and brms models are supposed to be approximately identical except for the fact that my stan model creates mu and sigma terms in the transformed parameters block which I then use in the model block for the likelihood and the generated quantities block to create log\_lik and yrep.
The two models produce essentially equal parameter estimates and the pp\_check(brms.fit)/ppc\_dens\_overlay(stan.fit) produce plots that look identical to me.
The loo() results are approximately the same, but not exactly.
The pp\_check(brms.fit, type=“loo\_pit\_overlay”)/ppc\_loo\_pit\_overlay() plots produce wildly different results: the brms loo\_pit\_overlay result indicates that there are no glaring model problems whereas the stan model results in a classic frowny face.
I assume that this is an error on my part and that there are 2 possibilities:
1. I am calculating yrep incorrectly.
2. I am calculating log\_lik incorrectly.
I cannot share my data, however, I have attached code to reproduce a similar effect using the Oats data from nlme in the attached file.
This code fits two models: a brms model and a stan model that uses slightly modified brms code (mu and sigma are moved to the tparms block and log\_lik and yrep are created in the generated quantities block).
This example shows that the resulting loo pit overlay plots are different and that it does not seem to be a difference in pp\_check vs. ppc\_loo\_pit\_overlay.
This is the brms loo-pit output:
Here is the stan loo-pit output where I calculate log\_lik and yrep in the generated quantities:
For the life of me I cannot figure out why these plots are so different when they should be about the same (if not identical).
Thanks!
Software
- Windows 10 64bit
- R 4.2.1
- rstan version 2.35.0.9000 (Stan version 2.35.0)
- brms version 2.21.10
- bayesplot version 1.11.1.9000
[pit\_ex.R](https://discourse.mc-stan.org/uploads/short-url/dWEcI84JWRtU0rKUa7WFWuebyLs.R) (5.1 KB)
---
### Author:  [@avehtari](https://discourse.mc-stan.org/u/avehtari)
#### Post date: [October 5, 2024, 3:07pm UTC](https://discourse.mc-stan.org/t/brms-loo-pit-overlay-differences-when-mu-and-sigma-are-in-the-transformed-parameters-block-and-loglik-yrep-are-in-the-generated-quantities/36813/2 "2024-10-05T15:07:03Z")
This is due to a very unfortunate design choice in RStan to make `extract()` by default to return the draws in randomly permutated order. Thus, your `yrep` and `lw` don’t have the same order and loo-pit output is nonsense. You can use `extract(, permute=FALSE)`, but the another very unfortunate design choice makes the output of the extract to have different structure. If I use
```no-highlight
yrep.stan <- as_draws_matrix(rstan::extract(fit2, pars="yrep", permute=FALSE))
```
I get the same plot as from `brms`.
---
### Author:  [@Brendan\_Alexander](https://discourse.mc-stan.org/u/Brendan_Alexander)
#### Post date: [October 5, 2024, 3:26pm UTC](https://discourse.mc-stan.org/t/brms-loo-pit-overlay-differences-when-mu-and-sigma-are-in-the-transformed-parameters-block-and-loglik-yrep-are-in-the-generated-quantities/36813/3 "2024-10-05T15:26:33Z")
I would have never guessed that in a million years.
Thanks for the clear explanation!