Any examples of using LFO-CV with cmdstanpy?

Interfaces
1

Just wondering if anyone has an example of using Leave Future Out CV with cmdstanpy? I see the vingette here which uses brms and R heavily. If not, I would be happy to implement it! I just don’t want to recreate the wheel…

Thanks!

2

CmdStanPy is pretty new and we don’t have the equivalent of loo implemented in Python (at least as far as I know). We’re looking for more CmdStanPy case studies to publish as vignettes or put in our case study collection, so please let us know if you do replicate that R/brms tutorial in Python with CmdStanPy.

1 Like
3

Would you want me to publish a PR to the cmdstanpy repo? Or is there a better place to put it?

4

A PR for what? There’s a whole loo package in R. Are you proposing to code that in Python? If so, it should probably also be a separate package from CmdStanPy, so that it could also be used with PyMC or PyStan or Pyro.

5

Oh, I was thinking of using arviz’s loo and just recreating Paul, Jonah and Aki’s example about LFO-CV. Just using arviz and cmdstanpy.

I could share that code here, as part of this thread, or I could put it somewhere else (hence the PR request). There is not an examples/ directory there, so adding the example to that repo is probably not in the cards.

1 Like
6

ArviZ docs could take this kind of example

7

CmdStanPy would definitely benefit by having an example of this in the documentation (where we do have an examples folder)

1 Like
8

Cool. I didn’t know ArviZ had an implementation of loo.