Can anyone provide any information about how the acc statistic and the corresponding CIs are calculated during the variable selection procedure with projpred?
Many thanks
Hi @T_nick,
acc is the classification accuracy, i.e., the proportion of correctly classified observations. For this, the response category (“class”) with highest probability (the probabilities are model-based) is taken as the prediction (“classification”) for an observation. I think we need to explain this in the documentation, thanks for asking.
1 Like
Thanks for the prompt response and the nice explanation! Can you please also provide some clarity about the calculation of CIs of acc values?
Oh yes, I forgot that. For the acc, the CIs are based on a normal approximation. The relevant lines are https://github.com/stan-dev/projpred/blob/560616ba87420beea21cfbee6620dbacbe63f961/R/summary_funs.R#L434-L445 and https://github.com/stan-dev/projpred/blob/560616ba87420beea21cfbee6620dbacbe63f961/R/summary_funs.R#L471-L474 where you can see that essentially, the SE of the “correct classification proportion” (i.e., the SE of the mean of the observation-wise zero-one values indicating correct classification) is calculated and after that, this SE is used to build a normal-approximation CI.
Perfect this is exactly what I needed! Thanks again
1 Like