Hi everyone, I am a Stan newbie.
Are there any functions for calculating AIC, BIC and DIC values for fitted model?
This isn’t a great answer to the question you actually asked, but this paper from @avehtari talks about WAIC as well as reason why you might not want to do it. You can compute WAIC using the waic function in the loo package (in R).
As for DIC, you can compute that directly from samples (although I don’t have a link to hand).
Hope this helps at least a little.
(Edit because i forgot the link)
1 Like
I saw loo package but I forgot to mention that I’m using PyStan. Is there anything like loo for python?
I will read the paper.
There is a reference code Python GitHub - avehtari/PSIS: Pareto smoothed importance sampling (PSIS) and PSIS leave-one-out cross-validation for Python and Matlab/Octave
@ahartikainen wrote “I think ArviZ library will include PSIS-loo code.” Installation of pystan on Fedora 33 causes fbprophet to fail · Issue #278 · stan-dev/pystan · GitHub
@avehtari Thank you, I’ll take a look.
If you can find a copy of Bayesian Data Analysis (BDA3, Gelman et al.), there’s some in-detail description of both LOO-CV and WAIC (as well as DIC) around pages 172-174.
I used that to write a Python script for computing the WAIC a while ago and if you’re willing to go there it’s fairly straightforward, although tedious and requiring a lot more calculations than the DIC (the latter requires only computing the logp at the posterior mean and the expected value of logp, while the former requires computing a version of that for each data point and mcmc sample).
That said, I would find it helpful if there was a pystan function to compute LOO and WAIC, since Stan developers seem to feel strongly against DIC and (god forbid) BIC.
This would be a great project for someone to develop in Python. In R, the loo package uses the output of RStan, and has special hooks for RStan, but is otherwise pretty general I believe given the right form of MCMC output.
Right—it’s doing the equivalent of a posterior predictive calculation using full Bayes from the sample rather than a point estimate. That’s also why you’ll find
1 Like
I might have to compute a DIC myself (here’s the forum post). If we need to implement the DIC ourselves, then if anyone’s got a link to a sample data set and correct DIC values as a check on our work, I’d appreciate the post!
I defer to Aki Vehtari et al., who strongly discourage the use of DIC.
If you need it for some kind of reason you can’t control, you might try the Coda package in R. Stan’s output can be converted to Coda’s input. But I don’t know what it requires that might not be available in the RStan/Stan output. (Ack, can’t put R in parens or I get ®).