Excellent, its more clear now.
loo-cv and WAIC are estimating the same thing, and there is no need to compute them both. PSIS-LOO in loo package has better diagnostics than WAIC, and thus is more reliable. In easy cases PSIS-LOO and WAIC give very similar results. In difficult cases, WAIC is more likely fail and if the results differ PSIS-LOO is more accurate or gives clear indication that both are failing. Thus computing WAIC is not giving you any additional information.
Both loo-cv and WAIC are not just measuring some arbitrary model fit. In usual case tphey both correspond to estimating the predictive performance for a single observation in one of the current sites. Model fit is only for how well would you predict the observation for the next visit in one of the current sites.
I now noticed that you have
binomial_logit_lpmf(y[i] | max_y[i] + j - 1, logit_p[i]);
which already sums over the visits.
In the model block
target += log_sum_exp(lp);
is matched in the generated quantities block with (although variable name has changed from lp
to ll
)
log_lik[i] = log_sum_exp(ll);
As the each log_lik[i]
includes the likelihood contribution from all the observation for site i
, using these log_lik
values for LOO (or WAIC) corresponds to leave-one-site-out predictive performance estimate.