Warnings during pp_average

#1

Hi everyone,

First time using model averaging. I’m finding some unusual warnings during pp_average procedures, with brms models:

pp_average(m3.1.2.3.1_logs,m3.1.2.3.2_logs,

  • m3.1.2.3.4_logs,m3.1.2.3_logs,m3.1.2.3.3_logs, method=“fitted”,newdata=df)
    Warning messages:
    1: Some Pareto k diagnostic values are too high. See help(‘pareto-k-diagnostic’) for details.

2: Some Pareto k diagnostic values are too high. See help(‘pareto-k-diagnostic’) for details.

3: In log(z) : NaNs produzidos
4: Some Pareto k diagnostic values are too high. See help(‘pareto-k-diagnostic’) for details.

5: Some Pareto k diagnostic values are too high. See help(‘pareto-k-diagnostic’) for details.

6: Some Pareto k diagnostic values are too high. See help(‘pareto-k-d> Blockquote
agnostic’) for details.

The individual loo results from models have good Pareto k estimates (k<0.7),
as can be seen:

loo.m3.1.2.3.1_logs

Computed from 20000 by 216 log-likelihood matrix

     Estimate   SE

elpd_loo 225.0 16.5
p_loo 29.3 6.2
looic -449.9 33.0

Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 209 96.8% 9
(0.5, 0.7] (ok) 7 3.2% 601
(0.7, 1] (bad) 0 0.0%
(1, Inf) (very bad) 0 0.0%

All Pareto k estimates are ok (k < 0.7).
See help(‘pareto-k-diagnostic’) for details.

loo.m3.1.2.3.2_logs

Computed from 20000 by 216 log-likelihood matrix

     Estimate   SE

elpd_loo 230.8 18.6
p_loo 31.4 8.7
looic -461.7 37.2

Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 210 97.2% 3
(0.5, 0.7] (ok) 6 2.8% 435
(0.7, 1] (bad) 0 0.0%
(1, Inf) (very bad) 0 0.0%

All Pareto k estimates are ok (k < 0.7).
See help(‘pareto-k-diagnostic’) for details.

loo.m3.1.2.3.3_logs

Computed from 20000 by 216 log-likelihood matrix

     Estimate   SE

elpd_loo 226.1 17.7
p_loo 22.0 5.0
looic -452.2 35.5

Monte Carlo SE of elpd_loo is 0.2.

Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 211 97.7% 30
(0.5, 0.7] (ok) 5 2.3% 989
(0.7, 1] (bad) 0 0.0%
(1, Inf) (very bad) 0 0.0%

All Pareto k estimates are ok (k < 0.7).
See help(‘pareto-k-diagnostic’) for details.

loo.m3.1.2.3.4_logs

Computed from 20000 by 216 log-likelihood matrix

     Estimate   SE

elpd_loo 231.3 19.3
p_loo 32.4 9.4
looic -462.5 38.5

Monte Carlo SE of elpd_loo is 0.4.

Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 208 96.3% 9
(0.5, 0.7] (ok) 8 3.7% 662
(0.7, 1] (bad) 0 0.0%
(1, Inf) (very bad) 0 0.0%

All Pareto k estimates are ok (k < 0.7).
See help(‘pareto-k-diagnostic’) for details.

loo.m3.1.2.3_logs

Computed from 20000 by 216 log-likelihood matrix

     Estimate   SE

elpd_loo 223.7 17.7
p_loo 23.2 5.1
looic -447.4 35.5

Monte Carlo SE of elpd_loo is 0.2.

Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 209 96.8% 21
(0.5, 0.7] (ok) 7 3.2% 525
(0.7, 1] (bad) 0 0.0%
(1, Inf) (very bad) 0 0.0%

All Pareto k estimates are ok (k < 0.7).
See help(‘pareto-k-diagnostic’) for details.

This models represent different patterns (intercept only, linear, categorical and additive) of a response variable along distances through a forest edge.
Does anyone have an idea of what was my mistake?

Thanks in advance!

Gustavo

Please also provide the following information in addition to your question:

  • Operating System: Windows 10
  • brms Version: 2.5.0
#2

Which of those warnings do you find unusual?

#3

Probably I was misunderstanding the warnings, which were only pointing for the values of k parameter between 0.5 and 0.7 (as shown for individual models).

Dont know if is valid to put another (related) question in the same thread, but there it comes:

I’m performing a model selection with LOO for different models that represent different patterns of the response variable (actually more than 20 response variables with an individual model selection for each) in respect to the (almost always) same predictor (distance of a forest edge): mean only, linear, nonlinear formula, categorical (sides of the edge) and additive (splines). But for many of those, the difference of looic between the best model and the other ones are less than 2x SE, which brought me to model averaging and stacking. The question is if is still valid to perform model averaging to models that are based on the same predictor.

Thanks for the help!

#4

As long as your models have the same response values and all models either have a discrete or all have a continuous likelihood, model comparison or averaging via loo can in principle be performed.

#5

Thanks!