I was wondering if the output of loo could be recovered later, by appending a generated quantities block. For example, assume we have a baseline stan model (“model_baseline.stan”) that does NOT contain a GQ block. Another model (“model_GQ.stan”) is identical to the baseline but with a GQ block that generates a vector log_lik of pointwise likelihoods.
loo output in 1 step:
model1 ← cmdstan_model(‘model_GQ.stan’) # contains GQ block with log_lik
fit1 ← model1$sample(data, seed, chains, save_warmup = TRUE, …) # output files contain log_lik
loo1 ← loo(fit1$draws(“log_lik”)) # get loo output
Now, I wish to split this process into 2 steps:
model2 ← cmdstan_model(‘model_baseline.stan’) # no GQ block
fit2 ← model2$sample(data, seed, chains, save_warmup = TRUE, …) # output files do NOT contain log_lik
fit2loo ← model1$generate_quantities(fit2, data, seed) # generate log_lik values later
loo2 ← loo(fit2loo$draws(“log_lik”)) # get loo output later
The problem is that loo2 does not seem to match with loo1. Is there something I’m missing? I have verified that the log_lik values generated are the same in both cases. Yet, the loo output is very different.
I wish to split the process because the single-step approach seems to take too long. I’d rather compute the posterior first and analyse it while log_lik array is being generated for loo criterion.
Edit: The problem could be because fit1 seems to be a CmdStanMCMC object while fit2 seems to be a CmdStanGQ object.