# How do folks use python (+ pandas) and stan?

**URL:** <https://discourse.mc-stan.org/t/how-do-folks-use-python-pandas-and-stan/17761>\
**Category:** Interfaces\
**Created:** [August 31, 2020, 2:09pm UTC](https://discourse.mc-stan.org/t/how-do-folks-use-python-pandas-and-stan/17761 "2020-08-31T14:09:07Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![pj-trainor](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/pj-trainor/32/9197_2.png) [@pj-trainor](https://discourse.mc-stan.org/u/pj-trainor)\
**Post date:** [August 31, 2020, 2:09pm UTC](https://discourse.mc-stan.org/t/how-do-folks-use-python-pandas-and-stan/17761/1 "2020-08-31T14:09:07Z")

</div>

Bottom-Line Up Front: I’m struggling to use stan and python together, especially when I want to test out different models and do posterior-predictive checks. I cannot use R and its excellent stan-ecosystem, so I was wondering how other folks manage python (+ pandas) and stan.

I’m having trouble integrating stan into my python workflow, even with great tools like [cmdstanpy](https://github.com/stan-dev/cmdstanpy) and [PyStan](https://pystan.readthedocs.io/en/latest/). I frequently struggle when doing posterior-predictive checks (PPCs), where I have to:

- transform a pandas dataframe to a stan-compatible dictionary (usually tedious)
- run the sampling and get the model fit (easy!)
- use summaries of my posterior distributions, and generated quantities to compare model estimates against the original data (very tedious)

This workflow is further complicated when I want to try different models, check my priors, run more [generated quantities](https://cmdstanpy.readthedocs.io/en/latest/generate_quantities.html), etc. For example, testing a partially-pooled GLM vs. a non-pooled GLM adds substantial overhead to my code if I want to do any PPCs.

I think the crux of this issue is going between my data and stan-data (input data, sampled values, generated quantities). I think [tidybayes](http://mjskay.github.io/tidybayes/index.html) explains it well:

> **Extracting tidy draws** from the model. This often means extracting indices from parameters with names like `"b[1,1]"` , `"b[1,2]"` into separate columns of a data frame, like `i = c(1,1,..)` and `j = c(1,2,...)` . More tediously, sometimes these indices actually correspond to levels of a factor in the original data; e.g. `"x[1]"` might correspond to a value of `x` for the first level of some factor.

Unfortunately, R is not an option at my workplace, which means I can’t use all of the excellent tools that have been developed for R. (It’s not an option because nobody uses R, and there isn’t a strong-enough data analysis contingent to justify adding it to our technical stack).

So my question is: how do folks use python (+ pandas) with stan, especially when they have to explore different models and make frequent use of the generated quantities block?

---

<div class="post-metadata">

**Author:** ![ahartikainen](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/ahartikainen/32/321_2.png) [@ahartikainen](https://discourse.mc-stan.org/u/ahartikainen)\
**Post date:** [August 31, 2020, 2:16pm UTC](https://discourse.mc-stan.org/t/how-do-folks-use-python-pandas-and-stan/17761/2 "2020-08-31T14:16:53Z")

</div>

ArviZ might help you with this.

As for doing generated quantities stuff, I think you are probably going to have same problems with R (unless you use brms or similar interface).

Is the problem that you have multidimensional inputs in a table format, and then transforming them back to ndarray creates some “overhead”?

[https://arviz-devs.github.io/arviz/index.html](https://arviz-devs.github.io/arviz/index.html)

---

<div class="post-metadata">

**Author:** ![pj-trainor](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/pj-trainor/32/9197_2.png) [@pj-trainor](https://discourse.mc-stan.org/u/pj-trainor)\
**Post date:** [August 31, 2020, 2:44pm UTC](https://discourse.mc-stan.org/t/how-do-folks-use-python-pandas-and-stan/17761/3 "2020-08-31T14:44:36Z")

</div>

I haven’t heard of ArviZ, but it looks promising!

I see what you’re saying about the generated quantities. I’m not familiar with brms.

And, yes, most of the overhead comes from turning my data (a pandas dataframe) into a dictionary that I can pass to a stan model (e.g. via [cmdstanpy.CmdStanModel](https://cmdstanpy.readthedocs.io/en/latest/api.html#cmdstanmodel)) and transforming the stan-output so it is compatible with my original data (as a dataframe). A common example is extracting indices from paramters with names like `b[1,2]` into separate columns of a dataframe, and mapping them back to their original data.

I can think of a few different ways to handle this with python, but I find myself having to write much different code each time I want to try out a new model on the same data. I was wondering if others have had the same problems.

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [August 31, 2020, 3:58pm UTC](https://discourse.mc-stan.org/t/how-do-folks-use-python-pandas-and-stan/17761/4 "2020-08-31T15:58:39Z")

</div>

> [@pj-trainor](#):
>
> I think the crux of this issue is going between my data and stan-data

agreed this is tedious.  
regarding the particularly tedious job of extracting parameter indices from names like `b[1,2]`, the latest version of CmdStanPy has methods which allow you to retrieve the fitted estimates as a named variable, i.e., the structure of the estimate corresponds to the structure of the Stan program variable. the relevant methods are:

- `stan_variable(name: str)` - [https://cmdstanpy.readthedocs.io/en/latest/api.html#cmdstanpy.CmdStanMCMC.stan\_variable](https://cmdstanpy.readthedocs.io/en/latest/api.html#cmdstanpy.CmdStanMCMC.stan_variable)
- `stan_variables()`
- `stan_variable_dims()`

however, you need Arviz or similar to do downstream analysis