'fewer than 0.001 effective samples' warnings for posterior predictions from generated quantities block?

Modeling
1

Q: Can I simply ignore the following warning (because it is about the posterior predictions, r_pred, from the generated quantities block rather than about the estimated parameters)?

The following parameters had fewer than 0.001 effective samples per transition:
  r_pred[1,1], r_pred[4,1], r_pred[5,1], r_pred[6,1], r_pred[8,1], r_pred[9,1], r_pred[10,1], r_pred[11,1], r_pred[12,1], r_pred[13,1], r_pred[14,1], r_pred[15,1], r_pred[16,1], r_pred[17,1], r_pred[18,1], 
...
r_pred[67,48], r_pred[68,48], r_pred[69,48], r_pred[70,48], r_pred[72,48], r_pred[73,48], r_pred[74,48], r_pred[75,48], r_pred[76,48], r_pred[77,48], r_pred[78,48], r_pred[79,48]
Such low values indicate that the effective sample size estimators may be biased high and actual performance may be substantially lower than quoted.

The generated quantities block is like:

generated quantities {
  
  vector[N2] r_pred[J];  
...

  for (j in 1:J) {
    
    alpha[j,1] = mu_alpha;  
    for (n in 2:N2) {
      alpha[j,n] = alpha[j,n-1] - theta*Real2[j,n-1];  
      }
      
    u2[j,1:N] = u[j,1:N];
    for (n in (N+1):N2) {
      u2[j,n] = alpha[j,n] + beta*u2[j,n-1] + sd_y*normal_rng(0,1);
      }
    r_pred[j] = u2[j] + Real2[j] + D2[j];
    } //end of for (j in 1:J) {
} //end of generated quantities

where

  • J is the number of entities (in my hierarchical latent AR1 model);
  • N is the training sample size (number of calendar quarters);
  • N2 is the full sample size (incl. holdout);
  • Real2 and D2 are deterministic transformations of both observed covariate data and estimated parameters;
  • mu_alpha, beta, theta, and sd_y are estimated parameters;
  • alpha is a transformed parameter;
  • r_pred is the posterior prediction of r, which is the observed outcome;
  • u is the latent variable estimated from the training sample and
  • u2 is its counterpart for the full sample.
2

No. It means your predictions are based on the equivalent of like 4 independent draws and

Such low values indicate that the effective sample size estimators may be biased high and actual performance may be substantially lower than quoted.

3

Many thanks for your reply. My estimated parameters are all fine in terms of the effective sample sizes and Rhat. Now that the posterior preductions r_pred are not usable, how should I improve it?

I have only a maximum of 60 quarters, which are split into 48 (ie, 80%) for training and 12 quarters for computing the r_pred for holdout testing.

4

@bgoodri: I am not sure what the OP should take from your reply (and lack the knowledge to tell it myself). Do you mean a) Using the predictions for r_pred is problematic, but the other parameters are mostly fine or b) the low n_eff for r_pred indicates that the model as a whole has issues and all its predictions are suspect (because their n_eff is biased upwards)

2 Likes
5

It is at least (a), but I am suspicious as to whether all the other parameters are fine. It is not common to have many effective samples on the coefficients and basically no effective samples on posterior predictions.

2 Likes
6

Thanks for your reply. I had a more detailed look, and it seems like the posterior predictions r_pred[49:60] for the holdout sample (quarters 49 to 60) are usable – with comparable levels of n_eff (see the second column from the right).

Capture
Capture769×322 53.3 KB

Apparently, the warnings are false alarms in this case due to r_pred[1:48] being the in-sample predictions derived from directly assigning the latent AR1 process u2[j,1:48] = u[j,1:48] to faciliate the construction of the posterior predictions from u2[j,49] onward (where u2[j,49:60] are the holdout-sample forecasts of the latent process and u[j,1:48] is the estimated parameter part of that process).

Q: Do you think that since I will only use the holdout-sample posterior predictions to assess the forecasting performance of the model, I am all right in this case?

7

What is stopping you from generating posterior predictions for the first 48?

8

Saving some computation is one concern. But most importantly, if u[j,48] is the best estimate from the observed reality for quarter 48 (after considering the complex dynamics and other parts of the model for quarter 1 to 48 on the whole), I tend to believe that my best forecast for u[j,49] should be based on that u[j,48] estimate (given the AR1 process), rather than a newly generated in-sample prediction for u[j,48].

What have I overlooked?

9

The Bayesian approach is not to predict 49 conditional on the expectation of 48 but to integrate over the uncertainty in 48 (and the previous 47 observations) when predicting 49. And likewise for 50, 51, …

2 Likes