AIC, BIC, DIC in Stan


#1

Hi everyone, I am a Stan newbie.
Are there any functions for calculating AIC, BIC and DIC values for fitted model?


#2

This isn’t a great answer to the question you actually asked, but this paper from @avehtari talks about WAIC as well as reason why you might not want to do it. You can compute WAIC using the waic function in the loo package (in R).

As for DIC, you can compute that directly from samples (although I don’t have a link to hand).

Hope this helps at least a little.

(Edit because i forgot the link)


#3

I saw loo package but I forgot to mention that I’m using PyStan. Is there anything like loo for python?
I will read the paper.


#4

There is a reference code Python https://github.com/avehtari/PSIS
@ahartikainen wrote “I think ArviZ library will include PSIS-loo code.” https://github.com/stan-dev/pystan/issues/278#issuecomment-405917515


#5

@avehtari Thank you, I’ll take a look.


#6

If you can find a copy of Bayesian Data Analysis (BDA3, Gelman et al.), there’s some in-detail description of both LOO-CV and WAIC (as well as DIC) around pages 172-174.
I used that to write a Python script for computing the WAIC a while ago and if you’re willing to go there it’s fairly straightforward, although tedious and requiring a lot more calculations than the DIC (the latter requires only computing the logp at the posterior mean and the expected value of logp, while the former requires computing a version of that for each data point and mcmc sample).

That said, I would find it helpful if there was a pystan function to compute LOO and WAIC, since Stan developers seem to feel strongly against DIC and (god forbid) BIC.


#7

This would be a great project for someone to develop in Python. In R, the loo package uses the output of RStan, and has special hooks for RStan, but is otherwise pretty general I believe given the right form of MCMC output.

Right—it’s doing the equivalent of a posterior predictive calculation using full Bayes from the sample rather than a point estimate. That’s also why you’ll find