LOO in Meta-analysis with mixed aggregate and individual data

[timdisher]

@avehtari

If there would be many studies (arms?), I would go for leave-one-study-out (as I did in that demo, as all studies had only aggregated data). Joint log score for whole study is also better in discriminating different models.

If there are not many studies (like you seem to have 4), I would separately do predictive checking and calibration checking for IPD studies. For a small number of aggregated studies there is not much information to do predictive and calibration checking. Although joint log score for studies is more discriminative than pointwise individual log scores, if you have only 4 studies, then the uncertainty in the cross-validation comparison will be high.

This is helpful thanks. This dataset is just a small tutorial set where everything runs quickly so in practice we often have maybe 1 dataset with IPD and then 30-40 aggregate studies (in larger networks). I see from this post the re-aggregating the IPD studies is an option but may come with consequences re: high Pareto-k’s. Since I think it would be a heavy lift to hack together exact loo (for me), do you think the issue of mixing loo for IPD and loso for aggregate studies is likely to lead to decision differences in practice or is it more of an academic/interesting problem?

@avehtari

All above was assuming cross-validation computation just works. If you get high Pareto-k’s easiest choice is to use K-fold-CV instead.

I’m also interested in seeing the loo() outputs which you did not include in your first post.

Have pasted these below.

print(loo_combined) # ll matrix is 4000 x 3858 consistent with one obs per IPD patient + one per agd arm
#> 
#> Computed from 4000 by 3858 log-likelihood matrix.
#> 
#>          Estimate   SE
#> elpd_loo  -1604.8 37.4
#> p_loo         7.7  1.2
#> looic      3209.6 74.8
#> ------
#> MCSE of elpd_loo is NA.
#> MCSE and ESS estimates assume MCMC draws (r_eff in [0.4, 1.5]).
#> 
#> Pareto k diagnostic values:
#>                          Count Pct.    Min. ESS
#> (-Inf, 0.7]   (good)     3856  99.9%   282     
#>    (0.7, 1]   (bad)         2   0.1%   <NA>    
#>    (1, Inf)   (very bad)    0   0.0%   <NA>    
#> See help('pareto-k-diagnostic') for details.

print(loo_agd_only) # ll matrix is 4000 x 4 consistent with one obs per study arm
#> 
#> Computed from 4000 by 4 log-likelihood matrix.
#> 
#>          Estimate  SE
#> elpd_loo    -16.4 0.8
#> p_loo         3.6 0.1
#> looic        32.7 1.6
#> ------
#> MCSE of elpd_loo is NA.
#> MCSE and ESS estimates assume MCMC draws (r_eff in [0.5, 0.6]).
#> 
#> Pareto k diagnostic values:
#>                          Count Pct.    Min. ESS
#> (-Inf, 0.7]   (good)     0      0.0%   <NA>    
#>    (0.7, 1]   (bad)      3     75.0%   <NA>    
#>    (1, Inf)   (very bad) 1     25.0%   <NA>    
#> See help('pareto-k-diagnostic') for details.

print(loo_agd_as_ipd) # ll matrix is 4000 x 1297
#> 
#> Computed from 4000 by 1297 log-likelihood matrix.
#> 
#>          Estimate   SE
#> elpd_loo   -670.5 17.6
#> p_loo         3.9  0.2
#> looic      1341.0 35.2
#> ------
#> MCSE of elpd_loo is 0.0.
#> MCSE and ESS estimates assume MCMC draws (r_eff in [0.8, 1.2]).
#> 
#> All Pareto k estimates are good (k < 0.7).
#> See help('pareto-k-diagnostic') for details.