Incorrect recovery using LOO and WAIC fit indices

[Rehab]

@avehtari

Sorry I’m not familiar with “2PL IRT” so I have to ask more questions. What are the observations? I can see y has dimensions NxJ, but what are N and J? Do you want to leave out y[i,j]' or y[i,] or y[,j]`? Could you also intent the code for easier reading? It’s now difficult to see what lines are inside each loop.

y[i,j] means responses of ( examinees i = 1, …, N ) to a set of ( items j = 1,…, J )
I think I should leave out a pair of (examinee, item) each time. I’m not sure.

@avehtari

In addition it seems that now inside your for loop

log_lik[i,j] = log( exp(log(pi[k]) + bernoulli_lpmf(y[i,j] | p[k])) 

is re-assigning log_lik[i,j] K times, and the final values is comouted using p[K] , while in model block you are collecting these terms to lps[k] and log_sum_exp :ing them.

As far as I understand, I’m collecting them in the model block in order to compute the log of the target posterior distribution. While in the generated block, we need the log likelihood for each pair of (examinee, item). Or should I also collect them in the generated block?

K means that there is k subpopulations under the unified population. It is similar that my population consists of people who have k distincts features. For example, low ability and hight ability groups.

@avehtari

I think in generated quantities you should declare

matrix[N,J] log_lik1;
vector[N*J] log_lik;

then inside the loop have

log_lik1[i,j] = ...something...;

and after the loop

log_lik = to_vector(log_lik1)

which is good format for loo package.

I’ll modify the code and add this part.

Thank you for taking the time to look at my code.