LOO hierarchical models: very low p_loo without warnings nor errors

I am struggling to understand the estimate of p_loo after fitting a hierarchical model and performing leave-one-out cross-validation. The model is built to fit biological data that follow a skew normal distribution. This distribution has four parameters which have been transformed into hyperparameters in order to understand more precisely how each hyperparameter depends on the biological variables of interest. The biological variables are seven and five are dummy, one is continuous and one is the interaction between a dummy variable and the continuous variable.

Each of the four hyperparameters is defined as a linear regression with one intercept and seven slopes (there are seven biological variables), so eight parameters for each hyperparameter. In total, the model contains 32 parameters (four hyperparameters times eight parameters). All of this is translated into the stan file:

data {
  int<lower=0> N;                //sample size
  int<lower=0> K;                //number of dummy variables
  matrix[N, K] X;                //matrix with dummy variables 
  vector[N] ratio;               //continuous independent variable
  int<lower=0, upper=1> y[N];    //binary dependent variable
}

parameters {
  real<lower=-4, upper=4> alpha_intercept;
  real<lower=0, upper=1> b_intercept;
  real<lower=-2, upper=2> c_intercept;
  real<lower=0.0001, upper=3> d_intercept;
  vector[K] alpha_coeff;       //slopes hyperparameter alpha
  vector[K] b_coeff;           //slopes hyperparameter b
  vector[K] c_coeff;           //slopes hyperparameter c
  vector[K] d_coeff;           //slopes hyperparameter d
}

transformed parameters {
  vector[N] alpha_hyp;
  vector[N] b_hyp;
  vector[N] c_hyp;
  vector[N] d_hyp;
  vector[N] y_hat;
  alpha_hyp = alpha_intercept + X * alpha_coeff;    //linear regression alpha
  b_hyp = b_intercept + X * b_coeff;                //linear regression b
  c_hyp = c_intercept + X * c_coeff;                //linear regression c
  d_hyp = d_intercept + X * d_coeff;                //linear regression d
  for (i in 1:N) {
    y_hat[i] = 0.01 + b_hyp[i] * exp(-0.5 * ((ratio[i] - c_hyp[i]) / d_hyp[i])^2) * (1 + erf(alpha_hyp[i] * (ratio[i] - c_hyp[i]) / (1.414214 * d_hyp[i])));
  }
}

model {
  alpha_coeff ~ normal(0, 1);        //prior regression coefficients hyperparameter alpha
  b_coeff ~ normal(0, 1);            //prior regression coefficients hyperparameter b
  c_coeff ~ normal(0, 1);            //prior regression coefficients hyperparameter c
  d_coeff ~ normal(0, 1);            //prior regression coefficients hyperparameter d
  alpha_hyp ~ gamma(12, 7);          //prior hyperparameter alpha
  b_hyp ~ beta(2, 2);                //prior hyperparameter b
  c_hyp ~ normal(-0.1, 0.5);         //prior hyperparameter c
  d_hyp ~ gamma(7, 8);               //prior hyperparameter d
  y ~ bernoulli_logit(logit(y_hat));
}

generated quantities {
  vector[N] log_lik;
  for (n in 1:N) {
    log_lik[n] = bernoulli_logit_lpmf(y[n] | logit(0.01 + b_hyp[n] * exp(-0.5 * ((ratio[n] - c_hyp[n]) / d_hyp[n])^2) * (1 + erf(alpha_hyp[n] * (ratio[n] - c_hyp[n]) / (1.414214 * d_hyp[n])))));
  }
}

The fitting ends without warnings nor errors as well as the loo cross-validation (I have followed this vignette. And, this is the loo output:

Computed from 24000 by 4330 log-likelihood matrix

. Estimate SE
elpd_loo -2224.2 27.1
p_loo 4.7 0.1
looic 4448.5 54.2

Monte Carlo SE of elpd_loo is 0.0.
All Pareto k estimates are good (k < 0.5).

My reasoning behind this large discrepancy between the p_loo (4.7) and the real number of parameters (32) is:

  1. Most of the posterior distributions are centred around zero.
  2. Only the four intercepts show a posterior distribution whose 95% confidence intervals do not overlap zero.

Nevertheless, there are also five-six parameters that are estimated to be close to zero but without the 95% CI spanning over zero. If my reasoning is correct, I would have expected the p_loo to be around seven or eight. Am I making sense?

  • What diagnostics can I run to explore this issue (no divergent transitions and low Pareto k estimates)?
  • Does the stan file reflect what is described in the first paragraph?

I have tried to reduced the number of hyperparameters and thus the number of parameters. For example, with only one hyperparameter, the model contains the three parameters that were not transformed into hyperparameters plus one intercept and seven slopes of the hyperparameters. So, the total number of parameters for the model with one hyperparameter is 11 (3 + 8). This returns a very similar p_loo of the one above which would make sense if my earlier reasoning is correct.

Any comments, suggestions or requests for further explanations are very appreciated.

I think everything is fine.

It’s more relevant how the posterior is different from prior and how strong posterior dependencies are.

Compare prior to posterior and examine posterior dependencies.

Looking at your model, you have very strong constraints on intercepts (lower and upper should be used only for logical or physical constraints like variances being positive and probabilities between 0 and 1, and otherwise smooth priors should be used)

What is the equation for y_hat? Couldn’t figure out, but seems like something with strong dependency between parameters.

In generated quantities you could re-use y_hat as it is transformed parameter.

Why do you have logit twice in y ~ bernoulli_logit(logit(y_hat)); ?