Calculating the likelihood for loo in a binomial mixture model

Excellent, its more clear now.

loo-cv and WAIC are estimating the same thing, and there is no need to compute them both. PSIS-LOO in loo package has better diagnostics than WAIC, and thus is more reliable. In easy cases PSIS-LOO and WAIC give very similar results. In difficult cases, WAIC is more likely fail and if the results differ PSIS-LOO is more accurate or gives clear indication that both are failing. Thus computing WAIC is not giving you any additional information.

Both loo-cv and WAIC are not just measuring some arbitrary model fit. In usual case tphey both correspond to estimating the predictive performance for a single observation in one of the current sites. Model fit is only for how well would you predict the observation for the next visit in one of the current sites.

I now noticed that you have

binomial_logit_lpmf(y[i] | max_y[i] + j - 1, logit_p[i]);

which already sums over the visits.
In the model block

target += log_sum_exp(lp);

is matched in the generated quantities block with (although variable name has changed from lp to ll)

log_lik[i] = log_sum_exp(ll);

As the each log_lik[i] includes the likelihood contribution from all the observation for site i, using these log_lik values for LOO (or WAIC) corresponds to leave-one-site-out predictive performance estimate.

2 Likes