Multiple model comparison using LOOIC

General
1

Hello everyone,

I am using the fantastic tools of the arviz package and I compare 3 competing models for modeling a time-series using cmdstanpy. I test about 80K models and I look at models with standard error of the difference between the LOO values that are greater than 2. Is there some correction for type I errors I should or could apply?

Thanks!

2

I don’t understand what you mean by comparing 3 and testing 80k models, but this paper might be related Efficient estimation and correction of selection-induced bias with order statistics | Statistics and Computing

EDIT: Changed the link to just published version (open access)

1 Like
3

Hi Aki, thanks for replying!

My data is 80 thousand time series vectors and I fit each of them indenendently with 3 different models. For each time series, I compare the 3 models to decide on the winning model so I perform model comparison 80K times. Is there a way to account for the total number of model comparisons?
I hope this is clearer now!

4

What is the goal? Choose one model for each 80K time series, or one common model for them? What is the difference between the models, that is, if the one of them is encompassing model, maybe think more carefully about the priors and you don’t need to do model selection at all (and it can be even harmful).

5

Currently, I am choosing one model for each time series. All time series are coming from one participant that was sampled multiple times. Each time series is a connectivity measure between different brain regions across time points. My goal is to determine what time series are static (random changes around the mean; model 1) and what are dynamic (changes in time are explained by exogenous factors; models 2 and 3). Hence the different models that I compare for each time series (each time series is modeled independently with its own prior).

6

For such “hypothesis testing”, I would guess that LOO is not very sensitive as brain connectivity data are often noisy, and I would recommend instead to use the encompassing model with good priors and examine the posterior of dynamic model parameters directly. Nabiximols and Betablockers case studies illustrate this. Even better if you could build a hierarchical model as I would expect there is some structure in the 80K timeseries.

7

Thank you, Aki. With all my love to Stan, I believe my case will be easier solved with more typical methods like regularized regression models. Thanks for your advice!