Your model is factorizable, but the prior structure problem is same as in section “Exact LOO-CV with re-fitting” in Leave-one-out cross-validation for non-factorizable models vignette. “The solution is to model y_i as a missing observation and estimate it along with all of the other model parameters.” Instead of exact LOO-CV, you are trying to do exact k-fold-CV, so you can use the same approach.