Highly correlated parameter draws when adding additional predictors

Modeling
, ,
1

Hello, I’ve ran into a weird issue with my model that I can’t make sense of.

I have a poisson regression model that I has a lot of predictors (21). The weird thing is, is that if I include all 21 predictors the parameters become highly correlated (.98 - .99), with accompanying warning that Bulk and Tail ESS are too small. However, if I remove any predictor (just one is enough), this behavior disappears and the model estimates normally (without any excessive correlation of the parameters).

I am, quite frankly, not sure what to make of this. All the dependent variables are standardized to mean=0; sd=1, with Normal(0, 1) priors on all coefficients.

The data set is quite large with thousands of observations.
None of the predictors are highly correlated between themselves, so multicollinearity should not be an issue.

Here’s my code:

test_model <- brm(
  dependent_var ~ 
    independent_var_1 + 
    independent_var_2 + 
    independent_var_3 + 
...
    independent_var_20 +
    independent_var_21,
  data = test_data,
  family = poisson(),
  prior = my_prior,
  chains = 6, cores = 6, iter = 3000,
  control = list(adapt_delta = 0.9999, max_treedepth = 15))
1 Like
2

Hi, sorry for not getting to you earlier, this is a good question!

So my best guess is that one of the predictors is redundant - note that this could happen without any of them being pairwise correlated. E.g. if I have predictor matrix as:

1 0 1
0 1 1
1 1 2
0 0 0

then no pairwise correlations are big, and any two predictors will work fin, but the third column is the sum of the previous two and thus using all three will introduce collinearity. One can construct similar examples of arbitrary size where any subset of size N is OK while any subset of size N + 1 is not.

You can also try fitting with brm(bf(dependant_var ~ ...., decomp = "QR"), ...) to use the QR decomposition which should help if the redundancy is not complete. However the resulting coefficients will no longer correspond to individual predictors, so you would need to use model predictions to interpret the model.

Best of luck with your model!