Promoting posteriordb into an official Stan project

Stan Governance
24

Thanks, Martin!

I have tried to go through the whole thread to make a summary of how the discussion stands. In some parts, discussions are bigger than posteriordb, and I have tried to single them out to continue those discussions outside the issue of posteriordb.

So, the main question is whether to include posteriordb under the stan-dev umbrella or not. Below are the main arguments I could deduce for and against, including posteriordb.

Argument for including posteriordb

  1. It is essential that projects that work with Stan, but are not limited to Stan, become part of the Stan project. This development will expand the userbase and potential developers connected to the Stan project.
  2. Including other PPLs in addition to Stan will enable developers and users of other PPL to also contribute to Stan.
  3. The risk of the project becoming not actively maintained is relatively small, and the potential cost for a project not being actively supported is small. Hence the risk is not that large.
  4. Verifying the estimation of expectation values in a high-level independent way is a great fit for the core goals of Stan.
  5. The general goals of posteriordb are aligned with the goals of the core Stan tools.

Argument for NOT including posteriordb

  1. Since posteriordb also includes other PPLs than Stan, it is not sure if it should be included as a Stan project. Including more vaguely-related projects will broaden Stan too much. Instead, Stan should be mainly focused on the Stan core parts. This perspective does not mean that posteriordb is a bad project. It should just not be a part of stan-dev. If posteriordb supports the more general community, it may provide some functionality to Stan, although this is not sufficient to be an appropriate Stan project.
  2. posteriordb risk feature bloating of the Stan project. We will include too many things that risk increasing maintenance burden, and this risk a decay of the Stan project in the long term.
  3. posteriordb and the use cases are currently too vague. Projects included in Stan should be more precise in its goals and how posteriordb plans to achieve these goals.

Important things to do in posteriordb (that we all agree on)

  1. The posteriordb should be language agnostic (i.e., not an RStan project).
  2. We should minimize “outside” dependencies of posteriordb. Ideally, we should only use code within stan-dev to compute statistics of interest (such as ess_bulk and ess_tail, MCSE etc.).

Comments and suggestions outside posteriordb for further discussion

  1. Should summarize_draws (posterior) be changed to reflect MCMC CLT more explicitly?
  2. How do we decide exactly what is core and what is auxiliary in the Stan project?

Comment/discussions with disagreement

  1. Should only expectations of parameters be included or more extensive statistics such as quantiles, ESS values, etc.?
  2. How exactly should testing and benchmarking be conducted?
  3. The scope of posteriordb. Should posteriordb be limited to only testing of expectation or also more explorative analysis and benchmarking?
  4. Should the project be mainly focused on Stan models and the current limitations in Stan, or should it be used more broadly. I.e., should we include models with discrete parameters?
  5. The sharpness of the posteriordb project. How precise should the project’s goals be, and how and how those goals are realized in the design of posteriordb.

I think this summarizes the ongoing discussion (but please feel free to correct this if you think I misunderstood some arguments). I think the main argument against including posteriordb is argument 1, i.e., how broad should the Stan project be?
Personally, I think this is more a matter of just deciding than something that has a true answer. Hence I believe this discussion is something the Stan Governing Board should focus on, since that also has broader implications.

The second issue is how broad or specific posteriordb should be where it seems to be disagreements. For me, I think agility is quite important, so I would prefer not to limit the project beforehand too much (except the stuff we all agree on). Depending on user needs, we should extend and improve in that direction. That said, I think all the discussions (1-5) we should keep in mind as the project develops.

I hope this can help the Stan Governing Body in the next steps of the decision process.

/Måns

3 Likes