# Compare two models with real and simulated data for statistical inference

**URL:** <https://discourse.mc-stan.org/t/compare-two-models-with-real-and-simulated-data-for-statistical-inference/20593>\
**Category:** General\
**Tags:** loo\
**Created:** [February 3, 2021, 7:28am UTC](https://discourse.mc-stan.org/t/compare-two-models-with-real-and-simulated-data-for-statistical-inference/20593 "2021-02-03T07:28:21Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ivan\_Camponogara](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ivan_camponogara/32/3787_2.png) [@Ivan\_Camponogara](https://discourse.mc-stan.org/u/Ivan_Camponogara)\
**Post date:** [February 3, 2021, 7:28am UTC](https://discourse.mc-stan.org/t/compare-two-models-with-real-and-simulated-data-for-statistical-inference/20593/1 "2021-02-03T07:28:21Z")

</div>

Hi

I run an experiment that includes several conditions. I then made a model and estimated the posterior distribution for each condition.

Now, I have several hypotheses that I would like to test. I simulated a dataset for each hypothesis (based on the number of subjects, conditions, and trials I used in my experiment). I assigned a different mean value for each condition according to my hypotheses. Then, for each simulated dataset (i.e., my hypotheses) I made a Bayesian model (same used for the real data).

Now, I would like to know which model (i.e., my hypotheses) better explains the data. Do you think a “pairwise” model comparison (model with real dataset Vs. model with the hypothesis) will work for testing my hypotheses? In such a case, can I use the loo\_compare() command and see which “hypothesis model” is closer to my data, and make some inferences on that?

Best

---

<div class="post-metadata">

**Author:** ![torkar](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/torkar/32/1355_2.png) [@torkar](https://discourse.mc-stan.org/u/torkar)\
**Post date:** [February 9, 2021, 8:16am UTC](https://discourse.mc-stan.org/t/compare-two-models-with-real-and-simulated-data-for-statistical-inference/20593/2 "2021-02-09T08:16:40Z")

</div>

Hi Ivan, sorry for the late reply… :)

If I understand you correctly you create 1,..,n models, which you then want to compare with LOO to check which one has the best relative out of sample prediction capability? As long as they use the same data and come from the same family, e.g., exponential group of families, you should be able to compare them using LOO.

Please tell me if I’ve misunderstood something :)

---

<div class="post-metadata">

**Author:** ![Ivan\_Camponogara](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ivan_camponogara/32/3787_2.png) [@Ivan\_Camponogara](https://discourse.mc-stan.org/u/Ivan_Camponogara)\
**Post date:** [February 11, 2021, 5:02am UTC](https://discourse.mc-stan.org/t/compare-two-models-with-real-and-simulated-data-for-statistical-inference/20593/3 "2021-02-11T05:02:15Z")

</div>

Hello!

Thanks for the reply. So I have three sets of data. Dataset\_original, which is my original sampled data, is composed of three conditions (A, B, C). Dataset 1 and 2 are two simulated datasets where, based on my hypotheses, I assigned different values to the three conditions. For instance, in hypothesis 1 I hypothesized that condition A is higher than B and C and simulated the data accordingly (dataset 1), and my hypothesis 2 is that condition B is higher than C and A, and simulated the data accordingly (dataset 2)  
I ran the same model (dep\_var ~ predictor + (1 + predictor | subject )) on the 3 datasets (same dependent variable and predictor), the one containing the real data, and the other 2 containing the simulated data based on mine hypotheses. I have now 3 models: model\_original, which is the model created on my original sampled data, and models 1 and 2 that are based on the simulated data (datasets 1 and 2 respectively). Now, I would like to know if I can run a pairwise comparison between model\_original and model 1 and 2 through the LOO command, and make inferences about the outputs. For instance, I compare model\_original with model 1 and model\_original with model 2. I see that the out of sample predictions of the first comparison (model\_original vs model 1) is 100 and of the second comparison (model\_original vs model 2) is 300. Can I infer that my hypothesis 1 is more likely true than hypothesis 2?

---

<div class="post-metadata">

**Author:** ![torkar](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/torkar/32/1355_2.png) [@torkar](https://discourse.mc-stan.org/u/torkar)\
**Post date:** [February 11, 2021, 6:26am UTC](https://discourse.mc-stan.org/t/compare-two-models-with-real-and-simulated-data-for-statistical-inference/20593/4 "2021-02-11T06:26:04Z")

</div>

If I understand you correctly your models use different datasets? Then it would be hard to compare them using LOO - I can’t understand what the results would mean :)

---

<div class="post-metadata">

**Author:** ![Ivan\_Camponogara](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ivan_camponogara/32/3787_2.png) [@Ivan\_Camponogara](https://discourse.mc-stan.org/u/Ivan_Camponogara)\
**Post date:** [February 11, 2021, 9:08am UTC](https://discourse.mc-stan.org/t/compare-two-models-with-real-and-simulated-data-for-statistical-inference/20593/5 "2021-02-11T09:08:36Z")

</div>

Many thanks. You confirmed my thoughts. There should be another way to perform a model comparison then. Thanks again!