I am facing a similar problem. I have described in details my dataset elsewhere.
To construct my reference model, I performed a SPCA using all my 164906 predictors. To account for measurement error on the dependent variable, I’m following the strategy from this book. The measurement error is not related to any of the predictors.
I used the principal components as predictors and I fit the model using brms as:
fit <- brm(y | resp_se(se_y, sigma=TRUE) ~ ., data=data.cv, family = gaussian(),prior=priors_me)
In projpred, I use init_refmodel for my custom reference model and I define dis which, in my understanding, is what was discussed above. However, in the formula argument of init_refmodel, “function calls on the right-side of the | are currently not allowed in projpred” (from the manual), so I cannot use the same formula as in brms. I construct the reference model as:
ref.spca <- init_refmodel(fit, data = data.full, formula = y ~ ., family = gaussian(), extract_model_data = extractor_cust, dis = as.matrix(fit)[, "sigma"])
The performance of the reference model when fit with resp_se() is worse than the submodels.
However, this is not the case when I follow exactly the same procedure but my reference model was fit without resp_se().
fit <- brm(y ~ ., data=data.cv, family = gaussian(),prior=priors_me)
The search procedure is always the same:
cvvsfast <- projpred::cv_varsel(ref.spca, method = "L1", nclusters_pred = 100, validate_search=FALSE, nterms_max=100, search_terms = NULL, ndraws = 400).
Is my understanding correct? How can I properly account for measurement error on the dependent variable during the procedure?