# Loo package

**URL:** <https://discourse.mc-stan.org/t/loo-package/3098>\
**Category:** General\
**Tags:** loo\
**Created:** [January 24, 2018, 4:37pm UTC](https://discourse.mc-stan.org/t/loo-package/3098 "2018-01-24T16:37:58Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![Assem](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/assem/32/1706_2.png) [@Assem](https://discourse.mc-stan.org/u/Assem)\
**Post date:** [January 24, 2018, 4:37pm UTC](https://discourse.mc-stan.org/t/loo-package/3098/1 "2018-01-24T16:37:58Z")

</div>

Hello everyone! I am new in statistic and have an issue with loo package usage.  
Particularly, after model implementation in stan, I wanted to use loo package to check  
the predictive performance of my model.  
the problem is estimated elpd and lpd are very large, for example, elpd is -19708 with  
248.5 standard error.  
Does it mean that model is not accurate or I can use this model?  
all k-pareto values are good less than 0.5

Thanks in advance

---

<div class="post-metadata">

**Author:** ![bgoodri](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bgoodri/32/4451_2.png) [@bgoodri](https://discourse.mc-stan.org/u/bgoodri)\
**Post date:** [January 24, 2018, 5:03pm UTC](https://discourse.mc-stan.org/t/loo-package/3098/2 "2018-01-24T17:03:53Z")

</div>

The elpd is primarily intended to compare with other models for the same outcome variable. If you have a lot of data, the elpd can be very large in absolute magnitude but may be bigger or smaller than the elpd of another model.

---

<div class="post-metadata">

**Author:** ![Assem](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/assem/32/1706_2.png) [@Assem](https://discourse.mc-stan.org/u/Assem)\
**Post date:** [January 24, 2018, 5:11pm UTC](https://discourse.mc-stan.org/t/loo-package/3098/3 "2018-01-24T17:11:14Z")

</div>

Thank you!  
I was worried about that because the estimated effective number of predictors p\_loo was more than 500 which is impossible in my dataset and SE was large.  
Thanks again

---

<div class="post-metadata">

**Author:** ![bgoodri](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bgoodri/32/4451_2.png) [@bgoodri](https://discourse.mc-stan.org/u/bgoodri)\
**Post date:** [January 24, 2018, 5:30pm UTC](https://discourse.mc-stan.org/t/loo-package/3098/4 "2018-01-24T17:30:55Z")

</div>

The estimator of `p_loo` can be flaky (it is supposed to be improved in 2.0) but that doesn’t mean there is necessarily anything wrong with the elpd.

---

<div class="post-metadata">

**Author:** ![Assem](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/assem/32/1706_2.png) [@Assem](https://discourse.mc-stan.org/u/Assem)\
**Post date:** [January 24, 2018, 5:35pm UTC](https://discourse.mc-stan.org/t/loo-package/3098/5 "2018-01-24T17:35:16Z")

</div>

Ok!

---

<div class="post-metadata">

**Author:** ![avehtari](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/avehtari/32/5935_2.png) [@avehtari](https://discourse.mc-stan.org/u/avehtari)\
**Post date:** [January 24, 2018, 8:50pm UTC](https://discourse.mc-stan.org/t/loo-package/3098/6 "2018-01-24T20:50:53Z")

</div>

> [@Assem](#):
>
> I was worried about that because the estimated effective number of predictors p\_loo was more than 500 which is impossible in my dataset and SE was large.

Yes, these are reasons to be worried. Usually impossible p\_loo values and large SE are associated with high k values or error in log\_lik computation. Since you had all k\<0.5, I would suspect error in your code. Could you tell more about your data and model? For example how many observations n and what model and likelihood? If you can show your model code, it would be easiest to check that there is nothing suspicious there.

---

<div class="post-metadata">

**Author:** ![Assem](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/assem/32/1706_2.png) [@Assem](https://discourse.mc-stan.org/u/Assem)\
**Post date:** [January 25, 2018, 5:20am UTC](https://discourse.mc-stan.org/t/loo-package/3098/7 "2018-01-25T05:20:18Z")

</div>

Sure, probably there are mistakes in my code. I believe I failed in log\_lik description.  
I have 4000 observations, 6 predictors including 5 categorical and one continuous variables.  
The model is logistic regression with binomial distribution, and I took binomial as likelihood.  
the model code is as follow:

data {  
int N;// number of observations  
int n[N];//trials  
int y[N];//successes  
int M;  
matrix[N,M] x;//predictor matrix  
}  
parameters {  
vector[M] Beta;  
real sigma;//between-observation variance  
real gamma[N];//random effect

}  
transformed parameters {  
vector[N] mu;// probability  
for (i in 1:N) {  
mu[i] =inv\_logit(dot\_product(x[i],Beta)+gamma[i]);  
}  
}  
model {  
Beta~normal(0,1000);//priors are vague  
sigma ~ inv\_gamma(0.001,0.001);  
gamma~normal(0,sigma);  
y~binomial(n,mu);  
}  
generated quantities {  
real log\_lik[N];  
for (i in 1:N) {  
log\_lik[i] = binomial\_lpmf(y[i]| n[i],mu[i]);  
}  
}

---

<div class="post-metadata">

**Author:** ![avehtari](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/avehtari/32/5935_2.png) [@avehtari](https://discourse.mc-stan.org/u/avehtari)\
**Post date:** [January 25, 2018, 9:10am UTC](https://discourse.mc-stan.org/t/loo-package/3098/8 "2018-01-25T09:10:27Z")

</div>

Code seems valid (there are ways to make it faster, and priors could be better).

The total number of parameters in your models is M+N+1=4007, so observing p\_loo around 500 is not impossible. gamma is also parameter. Without prior on gamma, p\_loo would around 4000, but the prior is regularizing it. Prior is also regularizing enough that PSIS-LOO works, that is, the posterior of gamma[i] is not too much different with or without the observation (y[i],n[i]).

> [@Assem](#):
>
> I was worried about that because the estimated effective number of predictors p\_loo was more than 500 which is impossible in my dataset and SE was large.

To summarize: Based on the code and values you gave, p\_loo\>500 is not impossible.

---

<div class="post-metadata">

**Author:** ![Assem](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/assem/32/1706_2.png) [@Assem](https://discourse.mc-stan.org/u/Assem)\
**Post date:** [January 25, 2018, 9:18am UTC](https://discourse.mc-stan.org/t/loo-package/3098/9 "2018-01-25T09:18:37Z")

</div>

Thanks for your reply.  
Does it mean that by changing priors I could get more proper p\_loo values?

---

<div class="post-metadata">

**Author:** ![avehtari](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/avehtari/32/5935_2.png) [@avehtari](https://discourse.mc-stan.org/u/avehtari)\
**Post date:** [January 25, 2018, 9:49am UTC](https://discourse.mc-stan.org/t/loo-package/3098/10 "2018-01-25T09:49:49Z")

</div>

> [@Assem](#):
>
> Does it mean that by changing priors I could get more proper p\_loo values?

The current p\_loo\>500 is proper. I think you have enough data to learn the posterior of the parameters of prior for gamma well, and you just have such overdispersion compared to binomial that distribution for gamma is wide which is reflected in p\_loo\>500. Since Pareto k diagnostic was ok, this completely proper result.

---

<div class="post-metadata">

**Author:** ![Assem](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/assem/32/1706_2.png) [@Assem](https://discourse.mc-stan.org/u/Assem)\
**Post date:** [January 25, 2018, 9:51am UTC](https://discourse.mc-stan.org/t/loo-package/3098/11 "2018-01-25T09:51:37Z")

</div>

Ok, got it. Thank you

---

<div class="post-metadata">

**Author:** ![Assem](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/assem/32/1706_2.png) [@Assem](https://discourse.mc-stan.org/u/Assem)\
**Post date:** [January 27, 2018, 4:59pm UTC](https://discourse.mc-stan.org/t/loo-package/3098/12 "2018-01-27T16:59:46Z")

</div>

There was mistake in the code above…  
I mean my mistake.  
The problem is I declared likelihood as binomial, and log likelihood as binomial\_lpmf.  
Seems it is right, however, I realized that I got good k values when I wrote log likelihood as binomial\_logit\_lpmf.  
Can I use it in this way, or I’m making it incorrect?

---

<div class="post-metadata">

**Author:** ![avehtari](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/avehtari/32/5935_2.png) [@avehtari](https://discourse.mc-stan.org/u/avehtari)\
**Post date:** [January 27, 2018, 8:22pm UTC](https://discourse.mc-stan.org/t/loo-package/3098/13 "2018-01-27T20:22:46Z")

</div>

> [@Assem](#):
>
> I declared likelihood as binomial, and log likelihood as binomial\_lpmf.

That was correct.  
Since you had `mu[i] =inv_logit(dot_product(x[i],Beta)+gamma[i]);`  
it’s correct to use `y~binomial` and `log_lik[i] = binomial_lpmf`  
If you use `binomial_logit_lpmf` then you don’t need `inv_logit` separately.

---

<div class="post-metadata">

**Author:** ![Assem](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/assem/32/1706_2.png) [@Assem](https://discourse.mc-stan.org/u/Assem)\
**Post date:** [January 28, 2018, 1:56am UTC](https://discourse.mc-stan.org/t/loo-package/3098/14 "2018-01-28T01:56:32Z")

</div>

I knew it. But with binomial\_lpmf I got not satisfying pareto k values.  
Is it better to use k-fold cross validation in this case?  
Thank you very much for reply

---

<div class="post-metadata">

**Author:** ![Alexander](https://avatars.discourse-cdn.com/v4/letter/a/bc8723/32.png) [@Alexander](https://discourse.mc-stan.org/u/Alexander)\
**Post date:** [February 27, 2019, 9:12am UTC](https://discourse.mc-stan.org/t/loo-package/3098/15 "2019-02-27T09:12:24Z")

</div>

Hi everyone!

I have a short question about using the loo package.  
I would like to calculate the LOOIC of the fit of a hierarchical model that consists of one group, n subjects with m trials each and s posterior samples. I’m struggling with the definition of the likelihood: Would you consider a n x s or m x s matrix for loo (i.e., likelihood based on subject-level or on trial-level)?

Thank you in advance!  
Alex

---

<div class="post-metadata">

**Author:** ![avehtari](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/avehtari/32/5935_2.png) [@avehtari](https://discourse.mc-stan.org/u/avehtari)\
**Post date:** [February 27, 2019, 1:06pm UTC](https://discourse.mc-stan.org/t/loo-package/3098/16 "2019-02-27T13:06:47Z")

</div>

> [@Alexander](#):
>
> . I’m struggling with the definition of the likelihood: Would you consider a n x s or m x s matrix for loo (i.e., likelihood based on subject-level or on trial-level)?

You can choose depending whether you want to test how well your model generalizes to new subjects or new trials. If you have subject or trial specific parameters (e.g. random effects) it is possible that PSIS-LOO approximation in loo package fails and you may need to use K-fold-CV. Se also this paper by Merkle, Furr and Rabe-Hesketh [[1802.04452] Bayesian comparison of latent variable models: Conditional vs marginal likelihoods](https://arxiv.org/abs/1802.04452)

---

<div class="post-metadata">

**Author:** ![Alexander](https://avatars.discourse-cdn.com/v4/letter/a/bc8723/32.png) [@Alexander](https://discourse.mc-stan.org/u/Alexander)\
**Post date:** [February 27, 2019, 3:52pm UTC](https://discourse.mc-stan.org/t/loo-package/3098/17 "2019-02-27T15:52:37Z")

</div>

Thank you very much! This is great help.

Alex