I will do in a few hours and post here.
Thanks
I did compare four models
fit1<-bodn ~ 1
fit2<-bodn ~ chf + (1 | id)
fit3<-bodn ~ chf + (1 | id), family = cumulative, data = bodn)
fit4<-p <bodn ~ mo(chf) + (1 | id), family = cumulative, data = bodn)
using
model_weights(fit1,fit2,fit3,fit4,weights=loo)
fit 4 had higher values than others.
But when I used “loo_compare” I got the message that I better use k-fold cross-validation with k=10 instead of LOO.
cross-validation is the next to learn!
Hi
fit1 <- add_criterion(fit, criterion = "loo")
fit2 <- add_criterion(fit, criterion = "loo")
fit3 <- add_criterion(fit, criterion = "loo")
fit4 <- add_criterion(fit, criterion = "loo")
Some of them might need reloo = TRUE. Then you do
loo_compare(fit1, fit2, fit3, fit4)
Thanks for the hint. I repeat them and will put an update here.
Hi,
For three models as below I ran loo_compare:
p2<-get_prior( rec_n_r_bodn ~ mo(rec_comb_sevchf)+(1|idno),family=cumulative(),
data =bodi)
p2$prior[1] <- "normal(0,1)" # sets a N(0,1) on \beta
p2$prior[3] <- "normal(0,100)" # for each cutpoint
p2$prior[18] <- "cauchy(0,2.5)" # for sd
p2$prior[21] <- "dirichlet(2,2)" # sets weakly regularizing dirichlet for 1-3 Likert scale response
fit_bodn2<-brm(
formula = rec_n_r_bodn~ mo(rec_comb_sevchf)+(1|idno),family=cumulative,
data =bodi,prior=p2,chain=5,thin=1,control = list(adapt_delta =0.99,max_treedepth=15))
p3<-get_prior( rec_n_r_bodn ~ rec_comb_sevchf+(1|idno),family=cumulative(),
data =bodi)
p3$prior[1] <- "normal(0,1)" # sets a N(0,1) on \beta
p3$prior[3] <- "normal(0,100)" # for each cutpoint
p3$prior[18] <- "cauchy(0,2.5)" # for sd
fit_bodn3<-brm(
formula = rec_n_r_bodn~ rec_comb_sevchf+(1|idno),family=cumulative,
data =bodi,prior=p3,chain=5,thin=1,control = list(adapt_delta =0.99,max_treedepth=15))
p4<-get_prior( rec_n_r_bodn ~ (1|idno),family=cumulative(),
data =bodi)
p4$prior[1] <- "normal(0,1)" # sets a N(0,1) on \beta
p4$prior[16] <- "cauchy(0,2.5)" # for sd
fit_bodn4<-brm(
formula = rec_n_r_bodn~ (1|idno),family=cumulative,
data =bodi,prior=p4,chain=5,thin=1,control = list(adapt_delta =0.99,max_treedepth=15))
fit2 ← add_criterion(fit_bodn2, criterion = “loo”,reloo = TRUE)
fit3 ← add_criterion(fit_bodn3, criterion = “loo”,reloo = TRUE)
fit4 ← add_criterion(fit_bodn4, criterion = “loo”,reloo = TRUE)
loo_compare(fit2, fit3, fit4)
model_weights(fit2, fit3, fit4, weights = "loo")
fit2 fit3 fit4
9.013943e-01 9.860567e-02 6.967917e-48
loo_compare(fit2, fit3, fit4)
elpd_diff se_diff
fit2 0.0 0.0
fit3 -2.2 4.0
fit4 -108.5 15.9
Computed from 5000 by 2622 log-likelihood matrix
Estimate SE
elpd_loo -4368.7 46.4
p_loo 900.7 21.1
looic 8737.4 92.8
------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 2375 90.6% 263
(0.5, 0.7] (ok) 213 8.1% 89
(0.7, 1] (bad) 21 0.8% 18
(1, Inf) (very bad) 13 0.5% 5
omputed from 5000 by 2622 log-likelihood matrix
Estimate SE
elpd_loo -4366.5 46.6
p_loo 902.9 21.5
looic 8733.0 93.2
------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 2389 91.1% 282
(0.5, 0.7] (ok) 193 7.4% 149
(0.7, 1] (bad) 29 1.1% 18
(1, Inf) (very bad) 11 0.4% 4
See help('pareto-k-diagnostic') for details.
Can I interpret that model 2 is better as it has better weight and regarding LOO_ic model 2 and 3 are almost the same?
I read these posts as well but a bit lost in interpretation:
Thanks
when I tried reloo=TRUE it took about one day to run it, the message was :
“40 problematic observation(s) found”
“The model will be refit 40 times.”
Hi,
your output from:
clearly shows that model fit2 and fit3 are more or less equivalent concerning out of sample predictions. However, they are both much better than fit4. We can see that when using a z-score of 2.57 (99%) and comparing fit4 with fit2:
> -108.5 + c(-1,1) * 2.57 * 15.9
[1] -149.363 -67.637
since it doesn’t cross 0.
What worries me is that you had so many problematic observations. I’m not really sure reloo helps you in this case. I would love if @avehtari could provide some insights.
February 27Hi,
your output from:
ssalimi:
loo_compare(fit2, fit3, fit4)
clearly shows that model
fit2andfit3are more or less equivalent concerning out of sample predictions. However, they are both much better thanfit4. We can see that when using a z-score of 2.57 (99%) and comparingfit4withfit2:
> -108.5 + c(-1,1) * 2.57 * 15.9 [1] -149.363 -67.637since it doesn’t cross
0.
What worries me is that you had so many problematic observations. I’m not really sure
reloohelps you in this case. I would love if @avehtari could provide some insights.
Visit Topic or reply to this email
Thanks, I have the same concern as you mentioned. My data is repeated measures, I decided to look at the first visit measures first. I got much better results for the first visit( I call it baseline). For next visits I will develop lag models and can share for discussion here. Thanks for all your inputs.
High Pareto k’s can be due to model misspecification or very flexible model, see LOO package glossary — loo-glossary • loo. p_loo’s are quite high, so it’s possible that it’s just very flexible model. It would be helpful to see plot of individual Pareto k’s and individual elpd values, but I can’t remember directly how to do it when add_criterion() has been used instead loo().
Quite soon we’ll have moment-matching-psis-loo which is much faster than reloo=TRUE, but improves accuracy when there are many high Pareto k’s. It’s possible to use the branch already if you don’t want to wait, but we’ll have better instructions soon available (ping @paul.buerkner, here’s nice example to test)
model_weights() tells also that predictions are very similar as combining predictions from fit2 and fit3 doesn’t improve compared to using just fit2
I’m confused why loo outputs show high Pareto k’s if you did use reloo=TRUE? Did you copy paste loo results before using reloo? Or @paul.buerkner does brms show the original Pareto k’s?
1 Like
[quote=“avehtari, post:29, topic:13165”]
I’m confused why loo outputs show high Pareto k’s if you did use reloo=TRUE? Did you copy-paste loo results before using reloo? Or @paul.buerkner does brms show the original Pareto k’s?
I did the loo in two different ways, first when I used reloo=TRUE it said that there is a problem in 40 observations and tried to refit the model 40 times that took 2 days and I had to terminate it.
Next, I used just loo and got Pareto k’s message.
Thanks for your inputs.
Now I understand. The new moment matching loo will be great for speeding this. Stay tuned, it will be available soon.
Thanks a lot.