Cookiecutter-cmdstanpy: a template for cmdstanpy projects

That’s a big challenge - I was surprised by what abstract code I ended up writing (and implicitly asking any user to understand) just in order to allow optionally varying the Stan program, priors and covariates. I’m fairly sure there’s room for improvement there but I guess there are also some things it’s just not worth making a template for. My intuition is that I do enough repeated work that some kind of template must be worthwhile but it’s hard to tell in advance what would be the most useful.

Definitely!

I’ve found the best approach to be to subclass cmdstanpys classes. But the again I wanted more than what cmdstanpy currently provides.

subclass cmdstanpys classes

cool! just curious - what did you want and which classes did you subclass?

Hm, only CmdStanModel and CmdStanMCMC and just for a few things, mainly for convenience. Except for the following:

  • I wanted to (semi-)gracefully be able to kill runs/chains. Whatever else I did always resulted in the chains staying running, messing up my timings. The solution was to reimplement (copy&paste) CmdStanMCMC's sample method, grabbing the ThreadPoolExecutor instance and the subprocesses, passing them to the subclassed CmdStanMCMC instance which then had all the information to terminate rogue chains, and to work with the partial results.
  • I wanted to be able to pool all warmup samples (from the last metric-window) and compute the covariance matrix. This is definitely a non-portable hack, because it depends on all parameters being unconstrained (the metric works on the unconstrained values, while stan only gives me the constrained values).

Other than that, I wanted the following things for convenience. I only looked at the source files for the above two classes, so I might have just missed some stuff:

  • Be able to work with fit objects where only some of the chains finished (see above).
  • Get per chain timings. Did I miss some functionality?
  • Collapse the stan summary to a boolean (i.e. everything alright or not).
  • Sometimes the ordering of arrays of vectors didn’t seem to be right. But I’m not sure anymore.
  • Randomly pick a draw to initialize a new run.
  • Inspect the source code and get functions/model blocks.
  • Restart sampling from a fit object.
  • Do some weird incremental warmup.
  • Addendum: I wanted the model to be able to infer some of the data from other parts of the data, like for example the length of an array.
  • Also: being able to sample from the prior.

Final edit:

  • Also, the sample function does not seem to play nice with numpys ndarray for the stepsize, and I think I could not simply pass multiple metrics or inits
1 Like

ArviZ might work in this case (tbh, there could be errors too)

I think these can be accessed from the stdout files?

Maybe ArviZ summary might be easier to work with?

Was this after you combined multiple chains? See arviz.InferenceData output, does that look correct?

I think a simple function could work in this case? (It might be easier to get inits from arviz.InferenceData)

(source → cpp?)

Was there any problems?

In Stan code?

? (fixed_param + a custom .stan is probably easiest way to do this)

I think I already fixed these issues. But if this is not yet working, then we need to fix it.

no, you didn’t miss that - it’s not scraped out of the Stan csv file; that could be done and added to the RunSet object.

Ari’s correct that it’s also in the stdout files.

what do you mean by “restart sampling”?

  • continue adaptation? in which case, initializing with stepsize and mass matrix should work.
  • get more samples? initializing with stepsize and mass matrix, no_warmup, will produce more samples. but there’s a problem in that we don’t save the state of the RNG - the seed + chain_id is the initial offset, but then during a run, we record how many times it’s called. to properly continue, one would need to know both the seed and how far to advance the RNG.
1 Like

we recommend running multiple chains in order to be able to test convergence. because if you don’t have convergence, whatever your sample is, you have no evidence that you have a set of draws from the posterior, consequently, no reason to believe that the samples are valid.

therefore, you need to wait until all chains have finished sampling. granted, this is problematic if you’re not able to run all chains in parallel or otherwise distribute them across a cluster.

1 Like

I thought I’d post an update as I think this project is now more or less usable - at least I’ve found it useful for my own cmdstanpy projects. If anyone is looking for a cmdstanpy template, please try it out and let me know your opinions - I’d really appreciate any feedback.

You can find the project here and fetch it from the command line like this:

pip install cookiecutter
cookiecutter gh:teddygroves/cookiecutter-cmdstanpy

I made quite a few changes with the general aim of being flexible enough to allow arbitrary models, configuration
and data processing while keeping things as simple and intuitive as possible. The readme now contains a lot more explanation and some detailed examples.

Here is the new project structure (the writing directory is optional)

.
├── LICENSE
├── Makefile
├── README.md
├── analyse.py
├── data
│   ├── prepared
│   │   └── readme.md
│   └── raw
│       ├── raw_measurements.csv
│       └── readme.md
├── model_configurations
│   ├── interaction.toml
│   ├── interaction_fake_data.toml
│   └── no_interaction.toml
├── prepare_data.py
├── pyproject.toml
├── requirements.txt
├── results
│   └── runs
│       └── readme.md
├── sample.py
├── src
│   ├── data_preparation.py
│   ├── model_configuration.py
│   ├── prepared_data.py
│   ├── readme.md
│   ├── sampling.py
│   ├── stan
│   │   ├── custom_functions.stan
│   │   ├── model.stan
│   │   └── readme.md
│   └── util.py
└── writing
    ├── bibliography.bib
    ├── img
    │   ├── example.png
    │   └── readme.md
    └── report.md

The template comes with a working example model - you should be able to try it out with make analysis straight away. This makes it possible to edit the template incrementally so that it implements your analysis, with the possibility to re-run at any time in order to check your work.

The only abstractions are now ModelConfiguration and PreparedData. A ModelConfiguration is a representation of a toml file with instructions for which modes to run in (choices are currently prior, posterior and k-fold cross validatation), where to find a Stan program and prepared data and how to run cmdstanpy (including stanc_options, cpp_options and keyword arguments for CmdStanModel.sample). There are a few examples in the folder model_configurations. PreparedData is a a definition of what prepared data should look like.

By default the results are converted to arviz InferenceData and saved in netcdf format to results/runs/<model configuration name>/mode.nc.

It’s now possible for the raw data to have any format with a little editing . The readme goes into some examples of how to do this.

I was a little unsure at first whether a cmdstanpy (or more generally statistical analysis) template was a viable project, mainly because as @mitzimorris says above the goals of being both flexible and easy to use and understand kind of pull in opposite directions. However I’m now using the template for any project that threatens to get bigger than a few files and anecdotally it definitely feels like an improvement on what I was doing before. I also noticed that project templates seem to be very popular in the deep learning world, e.g. this tensorflow_template_application and this pytorch-template have 1.9k and 3.2k github stars respectively.

I’m also curious about whether there are any other statistical analysis templates out there - if anyone knows one please post it!

8 Likes

I’ve done some poking around without finding anything really fleshed out. I’ve been wanting to make a cookiecutter or other template repo for CmdStanPy but for a slightly different purpose, for packages that want to use cmdstanpy/Stan to do pre-baked kinds of analysis (along the lines of Facebook Prophet). There are lots of useful tricks to know about compiling models on install that would be great to spread around.

2 Likes

That sounds like a really nice project. I can also imagine templates for the other Stan interfaces being very useful.

Maybe at some point it might make sense to add a templates section to the Stan documentation page. Before then it probably makes sense to try and form a consensus about whether and for what kinds of project it’s a good idea to recommend using templates and what things a template should and shouldn’t cover. I’m not really sure what the right process is here - maybe write a template sepecifications design doc, maybe just encourage experienced people to write and try out templates?

Thanks, @Teddy_Groves1

@mitzimorris: Would it make sense to put links to these cookiecutter templates with CmdStanPy doc? I’d prefer to reduce clutter on our top-level web pages.

As I said in the Stan meeting, I think it’d make sense to have a range of these cookie cutters depending on goal.

Here’s the Molecular Dynamics cookie cutter I mentioned:

GitHub - MolSSI/cookiecutter-cms: Python-centric Cookiecutter for Molecular Computational Chemistry Packages.

It’s a lot more elaborate in that it contains testing, doc, and continuous integration hooks, all of which would be nice for something like a group of people working on a package.

The other question is whether this should all link in Jupyter notebooks. I was under the impression that Jupyter is the way a lot of people interact with Python (somewhere between RStudio and knitr).

1 Like

I think it would make a lot of sense to add a community/downstream section of the CmdStanPy doc which links to things like this and also fbprophet, etc

2 Likes

and Arviz!

Thanks a lot for the thoughts and link - I’ve now renamed the template “cookiecutter-cmdstanpy-analysis” and added optional CI, tests and Sphinx docs with working examples.

I think one or more Jupyter notebooks might be a good idea for a model evaluation example, supplementing or replacing the existing analyse.py script, and/or as an alternative documentation option alongside markdown and Sphinx. I personally tend to just use python scripts for evaluation and markdown for documentation as I’m always using them already and can’t be bothered to set up Jupyter as well, but maybe that is just because I haven’t found the right template yet!

1 Like

I think a demo notebook go somewhere in a project. they’re also useful during development and for testing.

1 Like

This is now done - I changed the example analysis script into a notebook and I think it’s a big improvement.

I really appreciate all the comments - if anyone has any more please share!

Indeed, I think this would be a great addition to Redirecting to new ArviZ documentation host: ReadTheDocs

One thing to look into could be Kedro.

It’s something I use in my work that has multiple plugins for different cloud services and allows you to produce your work in a cloud agnostic way.

There’s also support for Spark and Dask.

It is mainly used for creating Data Science projects and does use cookie cutter under the hood.

1 Like

Thanks! Kedro looks like a very nice example of a well implemented Python package that uses cookiecutter.

1 Like