Usage of loo package with a multiple outcome model

General
, ,
1

I have fitted a multiple outcome model using Stan. I need to check the possibility of doing K-fold cross validation for that model.
In my dataframe, each subject can be treated as a one row and each row contain the information about multiple outcomes.

One possibility that I can do is, first divide the rows into K folds and then perform the K-fold cross validation by running the multiple outcome model for K no of times and calculate the error based on test data. (In my model I have the conditional independence assumption for each outcome. So I have modelled each outcome separately and I can evaluate the model for each outcome separately)

For an example for two outcomes, the model for each outcome can be fitted separately as follows:

y1 ~ bernoulli_logit_glm(x1, alpha1 beta1);
y2 ~ bernoulli_logit_glm(x2, alpha2 , beta2);

The above approach may not be efficient if either the value of K is very large and/or the dataset is large. So I am wondering whether I can approximate leave-one-out cross-validation using PSIS-LOO with the help of loo package like mentioned here:

Based on the above tutorial, first we need to extract the log likelihood and feed it into the loo package. This works for a single outcome. However in my situation I may need to define multiple log likelihoods for each outcome.

I am not sure how whether loo package allows to specify multiple log likelihoods.
So is this something that I can do with the use of loo package ?

Any advice will be really helpful.

Thank you.

2

You can compute separate log_lik for each outcome and concatenate

    vector[N_1] log_lik_1;
    vector[N_2] log_lik_2;
    vector[N_1 + N_2] log_lik;
...
    log_lik = append_row [log_lik_1, log_lik_2];

and then call loo() for the joint log_lik.

Alternatively you can use parameter_name option of extract_log_lik() to get separately log_liks for each utcome and call loo() for them separately.

If the dataset is very large, see also Using Leave-one-out cross-validation for large data