Hi all, As mentioned in the stan dev call I’ve been running SBC on the benchmark models: [image] stan-dev/stat_comp_benchmarks master/benchmarks Benchmark Models for Evaluating Algorithm Accuracy I’m using cmdstan called via a python script which calculates …

The first thing I check when I see such bad histograms is whether my simulation code actually matches the model. If I understand the code correctly, you basically want to ignore y and simulate your own y_sim. It is therfore my impression that you want to be using y_sim instead of y in the model blo…

I can’t believe I didn’t see that, thanks @martinmodrak ! I’ll re-run with the fixed model block. Also thanks for the tip for the half cauchy!

[image] PhilClemson: I can’t believe I didn’t see that No worries, happens to all of us :-D

Just thought I’d give an update as to where I got with this. With the fix in the model block the histograms look slightly less skewed but it still looks like the model-fitting is biased. [arma2] Here’s the updated model code: // ARMA(1, 1) functions { real half_cauchy_rng(real lb) { real p…

One thing that’s potentially problematic (but I currently can’t think completely straight about this) is that the err ~ normal(0, sigma); in the model code actually might not correspond well to the err_sim[t] = normal_rng(0, sigma_sim); because err is a non-linear function of parameters and thus som…

[image] martinmodrak: One thing that’s potentially problematic (but I currently can’t think completely straight about this) is that the err ~ normal(0, sigma); in the model code actually might not correspond well to the err_sim[t] = normal_rng(0, sigma_sim); because err is a non-linear function …

Just one more basic thing to check: are all your diagnostics in all runs all right? (divergences, neff, Rhat) When I do SBC, I usually have to set much narrower priors than I would in application to avoid producing large number of extreme datasets that than cause problems for the sampler. I’ve defin…

@martinmodrak This is a good point. Most of the diagnostics are unavailable as I’m running everything on a single-chain basis. I had assumed since the benchmark models have been extensively tested that there shouldn’t be any issues, but that only applies for the given datasets. I’ve also since real…

I would expect all diags to actually be available. Divergences are definitely accessible, but you can also definitely compute ESS and Rhat for single chains - it is less sensitive, but the code works (at least the R implementation), e.g. this works: single_chain <- matrix(rnorm(100), nrow = 100, nc…

I tried running stansummary and diagnose from CmdStan and indeed it does provide diags for a single chain. Not sure why I got it into my head that it didn’t! In any case I’ll update the SBC code to make sure these diagnostic checks are being passed.

SBC for arma model

Developers

Topic		Replies	Views
Rewriting example models with SBC Developers simulation-based-calibration	8	1405	February 25, 2019
Using narrower priors for SBC General prior-choice , simulation-based-calibration	14	1649	August 6, 2021
SBC for the benchmark models Developers simulation-based-calibration	5	1081	April 7, 2021
Simulation Based Calibration for Gaussian Process Model Modeling simulation-based-calibration	6	1259	July 14, 2019
SBC for cmdstanr General cmdstanr , simulation-based-calibration	22	1664	December 9, 2020

SBC for arma model

Related topics