Any other efficient methods for model comparsion(or model identification), like the LOOIC

Modeling
,
1

Hi, everyone, I have compared fifteen candidate models using both R-hat and LOOIC in Rstudio. The winning model had an R-hat value below 1.1 and the lowest LOOIC score among the other fourteen models. Additionally, I performed a posterior predictive check (PPC) to assess whether the winning model fit the actual behavior data well, using 4K posterior samples. During the model fitting process, each candidate model is fitted with four independent MCMC chains using 1K iterations (after 2K iterations for initial warmup per chain), resulting in 4K valid posterior samples.

I am wondering if there are other methods that could provide stronger evidence for model comparison or identification. Some references suggest using model recovery analysis to mitigate the risk of model misidentification. In this approach, for each model, a randomly selected sample from the 4K valid posterior samples is used to generate a synthetic dataset for all valid participants. Each synthetic dataset, based on a specific generating model, is then used to fit each of the alternative models, and the best-fitting model is identified using LOOIC scores. This process is repeated 100 times to compute the percentage of cases in which each model is identified as the best model based on synthetic datasets from a particular generating model. A higher percentage assigned to the generating model suggests better model identifiability.

However, when I attempted to apply this method to six representative models, RStudio was unable to complete the process due to memory demands. In reality, this me Given these constraints, I would greatly appreciate any suggestions for more computationally efficient methods for model comparison or identification in Bayesian computational modeling, similar to LOOIC.

Thank you in advance for your insights!

2

I think the the final model you choose will have artificially low number of parameters and in a grand sense will have posterior distributions that are too narrow. I think more realistic results will be obtained by fitting a single larger model, with skeptical priors on some of the parameters.

2 Likes
3

It would be good to know also standard errors for the differences compared to the best model, for example, provided by loo_compare() function. See also [2008.10296] Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison

Depending on how flexible model you have, you may need to use cross-validation checks instead. See, e.g. Roaches cross-validation demo and Nabiximols treatment efficiency case study

How string LOO-CV is, depends on those standard errors for the differences

1 Like
4

I like this comment from @harrelfe. Skeptical (i.e. well-reasoned, thoughtful) priors are important if you’re trying to embed your actual scientific inferences into the probability model, and it sounds like that’s what you’re going for. There’s no substitute for ‘Thinking about things seriously for a while’ as part of the model comparison process. I think @harrelfe may also be pointing you towards the ‘continuous model expansion’ approach (with a single larger model, instead of comparison of many discrete models to compare).

The winning model had an R-hat value below 1.1

It sounds like some of your models had R-hat values that are too high. Is this right? Even 1.5 is questionable. You’ll want to make sure each model is sampling well and converging on a single distribution before you try to compare models.