# \#simulation-based-calibration

**URL:** https://discourse.mc-stan.org/tag/simulation-based-calibration/105.md

[Latest](https://discourse.mc-stan.org/latest.md) · [Categories](https://discourse.mc-stan.org/categories.md) · [Tags](https://discourse.mc-stan.org/tags.md)

---

## [SBC derived quantities with vector of parameters](https://discourse.mc-stan.org/t/sbc-derived-quantities-with-vector-of-parameters/41618)

<div class="topic-metadata">

**Author:** [@jkbest2](https://discourse.mc-stan.org/u/jkbest2)\
**Replies:** 3\
**Last updated:** [September 28, 2026, 5:20pm UTC](https://discourse.mc-stan.org/t/sbc-derived-quantities-with-vector-of-parameters/41618 "2026-09-28T17:20:54Z")

</div>

I’d like to include a derived quantity in my SBC run that uses a parameter vector as input. The simple version of the function takes the vector log\_lambda as its argument: tot\_abund \<- function(log\_lambda) { sum(rpois…

---

## [How to use fit models to create new data](https://discourse.mc-stan.org/t/how-to-use-fit-models-to-create-new-data/36132)

<div class="topic-metadata">

**Author:** [@JAQuent](https://discourse.mc-stan.org/u/JAQuent)\
**Replies:** 1\
**Last updated:** [August 5, 2024, 11:04am UTC](https://discourse.mc-stan.org/t/how-to-use-fit-models-to-create-new-data/36132 "2024-08-05T11:04:23Z")

</div>

I want to run simulations for my projects in order to determine sample size for future data collection and how different prior perform, for which I want to use models that I already fit. Imagine I fit a model like this: …

---

## [Extending Rstan's Simulation Based Calibration](https://discourse.mc-stan.org/t/extending-rstans-simulation-based-calibration/20135)

<div class="topic-metadata">

**Author:** [@Dashadower](https://discourse.mc-stan.org/u/Dashadower)\
**Replies:** 4\
**Last updated:** [July 25, 2024, 11:23pm UTC](https://discourse.mc-stan.org/t/extending-rstans-simulation-based-calibration/20135 "2024-07-25T23:23:43Z")

</div>

When trying to do simulation based calibration, I personally felt that having to create multiple stan files was hard to keep track, especially when you’re making small changes here and there to the model. Rstan’s sbc() …

---

## [Binomial simulated based calibration (SBC)](https://discourse.mc-stan.org/t/binomial-simulated-based-calibration-sbc/34908)

<div class="topic-metadata">

**Author:** [@nicksc79](https://discourse.mc-stan.org/u/nicksc79)\
**Replies:** 3\
**Last updated:** [April 26, 2024, 11:18am UTC](https://discourse.mc-stan.org/t/binomial-simulated-based-calibration-sbc/34908 "2024-04-26T11:18:48Z")

</div>

I want to develop a binomial SBC by applying this code followed the instructions of Stan’s manual. But the error i am getting is “Ill-typed arguments to ‘~’ statement. No distribution ‘bernoulli\_logit’ was found with th…

---

## [How can we explain why SBC works?](https://discourse.mc-stan.org/t/how-can-we-explain-why-sbc-works/31889)

<div class="topic-metadata">

**Author:** [@danielschad](https://discourse.mc-stan.org/u/danielschad)\
**Replies:** 4\
**Last updated:** [July 1, 2023, 10:01pm UTC](https://discourse.mc-stan.org/t/how-can-we-explain-why-sbc-works/31889 "2023-07-01T22:01:27Z")

</div>

We (Daniel Schad, Bruno Nicenboim & Shravan Vasishth) have a manuscript on arXiv (\[2203.02361\] Data aggregation can lead to biased inferences in Bayesian linear mixed models and Bayesian ANOVA: A simulation study) using …

---

## [Could SBC rank uniformity be attacked by unstable equilibrium point in generator?](https://discourse.mc-stan.org/t/could-sbc-rank-uniformity-be-attacked-by-unstable-equilibrium-point-in-generator/29960)

<div class="topic-metadata">

**Author:** [@hyunji.moon](https://discourse.mc-stan.org/u/hyunji.moon)\
**Replies:** 1\
**Last updated:** [January 17, 2023, 4:39pm UTC](https://discourse.mc-stan.org/t/could-sbc-rank-uniformity-be-attacked-by-unstable-equilibrium-point-in-generator/29960 "2023-01-17T16:39:24Z")

</div>

Justification for SBC rank uniformity are two: theoretical: measure theory that excludes computational/numerical considerations (integration, optimization error) experimental: mostly with multivariate normal I was cur…

---

## [How do I evaluate my model, in terms of bias, coverage, etc.?](https://discourse.mc-stan.org/t/how-do-i-evaluate-my-model-in-terms-of-bias-coverage-etc/29450)

<div class="topic-metadata">

**Author:** [@grouper](https://discourse.mc-stan.org/u/grouper)\
**Replies:** 17\
**Last updated:** [January 4, 2023, 8:38am UTC](https://discourse.mc-stan.org/t/how-do-i-evaluate-my-model-in-terms-of-bias-coverage-etc/29450 "2023-01-04T08:38:00Z")

</div>

Ok, I’ve spent a few days looking for the best practice of evaluating my model in this case. With a frequentist MLE model, what I do is: simulate N datasets with known \\theta, find the MLE for \\theta. Calculate biases…

---

## [Validating hierarchical GP with SBC](https://discourse.mc-stan.org/t/validating-hierarchical-gp-with-sbc/29858)

<div class="topic-metadata">

**Author:** [@Lisar](https://discourse.mc-stan.org/u/Lisar)\
**Replies:** 3\
**Last updated:** [January 3, 2023, 5:11pm UTC](https://discourse.mc-stan.org/t/validating-hierarchical-gp-with-sbc/29858 "2023-01-03T17:11:04Z")

</div>

Hi all, I’m trying to validate hierarchical Gaussian process regressions using the SBC package. My real-world dataset is vegetation data, which was collected at many locations within many plots, and I’m interested in est…

---

## [How are Bayesian deconvolution, Bayesian inverse problem, Inverse Bayesian, Prior elicitation different?](https://discourse.mc-stan.org/t/how-are-bayesian-deconvolution-bayesian-inverse-problem-inverse-bayesian-prior-elicitation-different/29004)

<div class="topic-metadata">

**Author:** [@hyunji.moon](https://discourse.mc-stan.org/u/hyunji.moon)\
**Replies:** 1\
**Last updated:** [November 8, 2022, 6:07pm UTC](https://discourse.mc-stan.org/t/how-are-bayesian-deconvolution-bayesian-inverse-problem-inverse-bayesian-prior-elicitation-different/29004 "2022-11-08T18:07:01Z")

</div>

This question started from discussion in naming of SBC package on how to name family of optimizers such as HMC, VI: estimator or computational algorithm. I am confused by similar concepts with different names in the sub…

---

## [New preprint on model/algorithm validation with SBC - It is stronger than you thought!](https://discourse.mc-stan.org/t/new-preprint-on-model-algorithm-validation-with-sbc-it-is-stronger-than-you-thought/29372)

<div class="topic-metadata">

**Author:** [@martinmodrak](https://discourse.mc-stan.org/u/martinmodrak)\
**Replies:** 0\
**Last updated:** [November 7, 2022, 7:40am UTC](https://discourse.mc-stan.org/t/new-preprint-on-model-algorithm-validation-with-sbc-it-is-stronger-than-you-thought/29372 "2022-11-07T07:40:16Z")

</div>

Me along with @hyunji.moon, @Dashadower, @paul.buerkner, @nhuurre, Kateřina Faltejsková, @andrewgelman and @avehtari have a new preprint on SBC: Simulation-Based Calibration Checking for Bayesian Computation: The Choice …

---

## [Naming in the SBC package](https://discourse.mc-stan.org/t/naming-in-the-sbc-package/23838)

<div class="topic-metadata">

**Author:** [@martinmodrak](https://discourse.mc-stan.org/u/martinmodrak)\
**Replies:** 12\
**Last updated:** [June 10, 2022, 5:50pm UTC](https://discourse.mc-stan.org/t/naming-in-the-sbc-package/23838 "2022-06-10T17:50:36Z")

</div>

I am working with @hyunji.moon, @Dashadower and Teemu Säilynoja (AFAIK not on Discourse) on a revamp of the API of the SBC package to bring the power of SBC to the masses. I would be super happy to get help with naming o…

---

## [Simulating fake data for multi-species occupancy model (Dorazio et al's butterflies)](https://discourse.mc-stan.org/t/simulating-fake-data-for-multi-species-occupancy-model-dorazio-et-als-butterflies/26766)

<div class="topic-metadata">

**Author:** [@julrich](https://discourse.mc-stan.org/u/julrich)\
**Replies:** 2\
**Last updated:** [March 13, 2022, 11:26pm UTC](https://discourse.mc-stan.org/t/simulating-fake-data-for-multi-species-occupancy-model-dorazio-et-als-butterflies/26766 "2022-03-13T23:26:42Z")

</div>

Hi everyone, I’m in the process of learning to use STAN to build a multi-species, multi-site occupancy model but have a few questions with simulating fake data for my model. My real data is similar in form to the examp…

---

## [SBC with histogram/ecdf vs treating credible intervals as confidence intervals](https://discourse.mc-stan.org/t/sbc-with-histogram-ecdf-vs-treating-credible-intervals-as-confidence-intervals/26641)

<div class="topic-metadata">

**Author:** [@bnicenboim](https://discourse.mc-stan.org/u/bnicenboim)\
**Replies:** 6\
**Last updated:** [March 9, 2022, 6:25pm UTC](https://discourse.mc-stan.org/t/sbc-with-histogram-ecdf-vs-treating-credible-intervals-as-confidence-intervals/26641 "2022-03-09T18:25:38Z")

</div>

Ok, this is not really about Stan, but I guess that there are enough SBC experts around to answer me :) (Take into account that maybe my question is super naive.) Cook, Gelman, and Rubin (2006) write the following Th…

---

## [Reciprocal truncated normal distribution seems to work well but the SBC hists are terrible](https://discourse.mc-stan.org/t/reciprocal-truncated-normal-distribution-seems-to-work-well-but-the-sbc-hists-are-terrible/26585)

<div class="topic-metadata">

**Author:** [@bnicenboim](https://discourse.mc-stan.org/u/bnicenboim)\
**Replies:** 3\
**Last updated:** [March 1, 2022, 4:57am UTC](https://discourse.mc-stan.org/t/reciprocal-truncated-normal-distribution-seems-to-work-well-but-the-sbc-hists-are-terrible/26585 "2022-03-01T04:57:35Z")

</div>

Hi all, I’m fitting a truncated reciprocal distribution to simulated data. It’s a weird distributions, but it has been argued that it’s useful for reaction times and I anyways want to use it for a tutorial. It requires …

---

## [Check your models! SBC 0.1.1 is out!](https://discourse.mc-stan.org/t/check-your-models-sbc-0-1-1-is-out/26343)

<div class="topic-metadata">

**Author:** [@martinmodrak](https://discourse.mc-stan.org/u/martinmodrak)\
**Replies:** 1\
**Last updated:** [February 15, 2022, 12:50pm UTC](https://discourse.mc-stan.org/t/check-your-models-sbc-0-1-1-is-out/26343 "2022-02-15T12:50:01Z")

</div>

Do you use simulated data to check that your model works correctly? If no, then you should :-) If yes, then you are just a small step away from being able to validate the implementation of your model more thoroughly with…

---

## [How random variable type can change SBC?](https://discourse.mc-stan.org/t/how-random-variable-type-can-change-sbc/25846)

<div class="topic-metadata">

**Author:** [@hyunji.moon](https://discourse.mc-stan.org/u/hyunji.moon)\
**Replies:** 1\
**Last updated:** [January 2, 2022, 11:22pm UTC](https://discourse.mc-stan.org/t/how-random-variable-type-can-change-sbc/25846 "2022-01-02T23:22:51Z")

</div>

Assuming “random variable as type” concept which Julia is proud of as introduced here is the same as rvar in posterior package, may I ask whether viewing prior as rvar would have any value or use-case? I think this chang…

---

## [Simulation-based calibration (SBC) and thinning](https://discourse.mc-stan.org/t/simulation-based-calibration-sbc-and-thinning/25128)

<div class="topic-metadata">

**Author:** [@cao](https://discourse.mc-stan.org/u/cao)\
**Replies:** 4\
**Last updated:** [November 8, 2021, 3:51pm UTC](https://discourse.mc-stan.org/t/simulation-based-calibration-sbc-and-thinning/25128 "2021-11-08T15:51:33Z")

</div>

Hi all, In the Stan reference manual states that “it should be emphasized that the only reason to thin a sample is to reduce memory requirements”. This is a common view I see mentioned in the Stan forums. However, in F…

---

## [Expressing data-averaged posterior](https://discourse.mc-stan.org/t/expressing-data-averaged-posterior/24995)

<div class="topic-metadata">

**Author:** [@hyunji.moon](https://discourse.mc-stan.org/u/hyunji.moon)\
**Replies:** 8\
**Last updated:** [November 8, 2021, 3:00pm UTC](https://discourse.mc-stan.org/t/expressing-data-averaged-posterior/24995 "2021-11-08T15:00:50Z")

</div>

data-averaged posterior Hi, I was wondering how can the two following procedures with different orders of applying g and gathering samples be reflected in the mathematical expression of data-averaged posterior. 1. 1 tim…

---

## [Comparing theory and stan implementation on sbc](https://discourse.mc-stan.org/t/comparing-theory-and-stan-implementation-on-sbc/16913)

<div class="topic-metadata">

**Author:** [@hyunji.moon](https://discourse.mc-stan.org/u/hyunji.moon)\
**Replies:** 7\
**Last updated:** [October 29, 2021, 2:58pm UTC](https://discourse.mc-stan.org/t/comparing-theory-and-stan-implementation-on-sbc/16913 "2021-10-29T14:58:08Z")

</div>

I was comparing rstan sbc.R and stan user manual(+paper) and had some questions: 1. Rank consistency vignettes has: int ranks\_\[1\] = {pi \> pi\_}; but user manual and original paper, Validating Bayesian Inference Algorit…

---

## [Bayesian power analyses and sequential analytic designs - feedback on paper](https://discourse.mc-stan.org/t/bayesian-power-analyses-and-sequential-analytic-designs-feedback-on-paper/24293)

<div class="topic-metadata">

**Author:** [@JimBob](https://discourse.mc-stan.org/u/JimBob)\
**Replies:** 0\
**Last updated:** [September 8, 2021, 9:08am UTC](https://discourse.mc-stan.org/t/bayesian-power-analyses-and-sequential-analytic-designs-feedback-on-paper/24293 "2021-09-08T09:08:08Z")

</div>

Hi everyone, I have written a tutorial paper on power analyses using brms. The basic approach follows that proposed by Kruschke, but extends it in a few ways - for example, including the possibility of sequential design…

---

## [Bayesian power analysis for sample size planning](https://discourse.mc-stan.org/t/bayesian-power-analysis-for-sample-size-planning/24144)

<div class="topic-metadata">

**Author:** [@Agus](https://discourse.mc-stan.org/u/Agus)\
**Replies:** 4\
**Last updated:** [September 20, 2021, 9:09pm UTC](https://discourse.mc-stan.org/t/bayesian-power-analysis-for-sample-size-planning/24144 "2021-09-20T21:09:55Z")

</div>

Hi! This is sort of a “higher-level” question, but I think there are a lot of people in this forum who might have stumbled upon the very same issue and could share their own experiences or insights! So, I prepared a re…

---

## [Simulation-based calibration: Some challenges and directions for future research](https://discourse.mc-stan.org/t/simulation-based-calibration-some-challenges-and-directions-for-future-research/24240)

<div class="topic-metadata">

**Author:** [@andrewgelman](https://discourse.mc-stan.org/u/andrewgelman)\
**Replies:** 1\
**Last updated:** [September 19, 2021, 8:36am UTC](https://discourse.mc-stan.org/t/simulation-based-calibration-some-challenges-and-directions-for-future-research/24240 "2021-09-19T08:36:12Z")

</div>

Hi, I thought this might interest many Stan users: Simulation-based calibration: Some challenges and directions for future research « Statistical Modeling, Causal Inference, and Social Science It’s following up on the …

---

## [SBC StanConnect 2021](https://discourse.mc-stan.org/t/sbc-stanconnect-2021/21798)

<div class="topic-metadata">

**Author:** [@hyunji.moon](https://discourse.mc-stan.org/u/hyunji.moon)\
**Replies:** 15\
**Last updated:** [September 13, 2021, 11:36pm UTC](https://discourse.mc-stan.org/t/sbc-stanconnect-2021/21798 "2021-09-13T23:36:12Z")

</div>

Hello, @martinmodrak and I would like to organize a StanConnect session on Simulation-based calibration (SBC). SBC is relatively young but a versatile diagnosis tool that could calibrate mathematical models (prior, ob…

---

## [SBC StanConnect Tutorial - discussion thread](https://discourse.mc-stan.org/t/sbc-stanconnect-tutorial-discussion-thread/24176)

<div class="topic-metadata">

**Author:** [@martinmodrak](https://discourse.mc-stan.org/u/martinmodrak)\
**Replies:** 3\
**Last updated:** [September 13, 2021, 10:41pm UTC](https://discourse.mc-stan.org/t/sbc-stanconnect-tutorial-discussion-thread/24176 "2021-09-13T22:41:08Z")

</div>

This is a thread to discuss anything about the tutorial on SBC that will be held was held at SBC StanConnect. The tutorial materials (which you can follow on your own) are available at SBC Tutorial - Index Are you unsur…

---

## [User-defined function calls in transformed data block (for SBC, with differential equations)](https://discourse.mc-stan.org/t/user-defined-function-calls-in-transformed-data-block-for-sbc-with-differential-equations/24187)

<div class="topic-metadata">

**Author:** [@Matthew\_Spencer1](https://discourse.mc-stan.org/u/Matthew_Spencer1)\
**Replies:** 3\
**Last updated:** [September 4, 2021, 4:10pm UTC](https://discourse.mc-stan.org/t/user-defined-function-calls-in-transformed-data-block-for-sbc-with-differential-equations/24187 "2021-09-04T16:10:38Z")

</div>

I have a working stan model (tested using cmdstan 2.25.0) that uses the ode\_rk45 differential equation solver to generate multinomial probabilities \\mathbf z\_t for observations at many different times t, in many differen…

---

## [SBC pre-tutorial assistance thread](https://discourse.mc-stan.org/t/sbc-pre-tutorial-assistance-thread/24113)

<div class="topic-metadata">

**Author:** [@hyunji.moon](https://discourse.mc-stan.org/u/hyunji.moon)\
**Replies:** 8\
**Last updated:** [August 30, 2021, 4:51pm UTC](https://discourse.mc-stan.org/t/sbc-pre-tutorial-assistance-thread/24113 "2021-08-30T16:51:14Z")

</div>

SBC StanConnect tutorial staff (@martinmodrak, @Dashadower, Teemu, and I) will use this thread to assist you with SBC tutorial before the conference. Please do not hesitate to raise issues or ask questions ranging from i…

---

## [Simulation Based Calibration (SBC) for non-linear model](https://discourse.mc-stan.org/t/simulation-based-calibration-sbc-for-non-linear-model/24074)

<div class="topic-metadata">

**Author:** [@JLC](https://discourse.mc-stan.org/u/JLC)\
**Replies:** 4\
**Last updated:** [August 30, 2021, 4:22pm UTC](https://discourse.mc-stan.org/t/simulation-based-calibration-sbc-for-non-linear-model/24074 "2021-08-30T16:22:25Z")

</div>

The upcoming SBC event led to me trying to implement an SBC approach to a non-linear model I’m working on. The predictors are matrices of observed predictors over time relative to an observed outcome. As the model is n…

---

## [Summarising Rhat values over multiple variables/fits](https://discourse.mc-stan.org/t/summarising-rhat-values-over-multiple-variables-fits/23957)

<div class="topic-metadata">

**Author:** [@martinmodrak](https://discourse.mc-stan.org/u/martinmodrak)\
**Replies:** 5\
**Last updated:** [August 17, 2021, 7:53am UTC](https://discourse.mc-stan.org/t/summarising-rhat-values-over-multiple-variables-fits/23957 "2021-08-17T07:53:46Z")

</div>

In the SBC package under development, I use the maximum Rhat value over all parameters in a fit as one of diagnostics whether the fit was OK overall. When exploring a set of all fits of the SBC runs I also look at the ma…

---

## [Call for testers of model checking/testing tutorial](https://discourse.mc-stan.org/t/call-for-testers-of-model-checking-testing-tutorial/23981)

<div class="topic-metadata">

**Author:** [@martinmodrak](https://discourse.mc-stan.org/u/martinmodrak)\
**Replies:** 0\
**Last updated:** [August 16, 2021, 10:38am UTC](https://discourse.mc-stan.org/t/call-for-testers-of-model-checking-testing-tutorial/23981 "2021-08-16T10:38:55Z")

</div>

Hi, the four of us (me, @hyunji.moon, Teemu Säilynoja and Shinyoung Kim) are preparing an online tutorial showing how to use simulated data and the SBC package to guard against bugs and other problems in your Stan model…

---

## [Using narrower priors for SBC](https://discourse.mc-stan.org/t/using-narrower-priors-for-sbc/21709)

<div class="topic-metadata">

**Author:** [@hyunji.moon](https://discourse.mc-stan.org/u/hyunji.moon)\
**Replies:** 14\
**Last updated:** [August 6, 2021, 5:19pm UTC](https://discourse.mc-stan.org/t/using-narrower-priors-for-sbc/21709 "2021-08-06T17:19:26Z")

</div>

ADMIN EDIT: Moved this to a new topic, as I believe this is of general interest (by @martinmodrak) I always wanted discuss the need for narrower SBC prior compared to general modeling situations. This results from the f…

[Next page](https://discourse.mc-stan.org/tag/simulation-based-calibration/105.md?match_all_tags=true&page=1&tags%5B%5D=simulation-based-calibration)