Leave-one-group-out CV

Interfaces brms
10

I’m curious to hear your general approach for doing leave-one-group-out k-fold CV in Stan. I’ve been trying to do the same thing, and my approach seems to be an ugly hack. I roughly follow this approach: https://datascienceplus.com/k-fold-cross-validation-in-stan/#share-wrapper, by including a holdout vector as data. I also include an index vector for the start locations (jj_start) of where the group index changes (i.e. first row for a new group) as data. I then fix all the groups level parameters of the group I’m holding out in transformed parameters to 0 (we need to do this right?), and do some ugly chopping of my data to get group summed log_lik in generated quantities for J groups (people in my case with repeated measures).

generated quantities{
vector[J] log_lik;
for (j in 1:J){
  {
    int start; 
    int end; 
    start = jj_start[j];
    if(j<J){
      end = (jj_start[j+1]-1);
    }else{
      end = N;
    }
      log_lik[j] = normal_lpdf(y[start:end] | a_person[j] + block(x,start,1,end-start+1,K) * beta, sigma);
  }
}}

I’m not even sure this is right, and I know it can’t be the best way to do leave-one-group-out CV. I tried to dig through brms code to get a better idea, but haven’t figured any of that out. Hopefully you’ve found a better way!