Voting on inclusion of posteriordb

Stan Governance
1

Following the discussion at https://discourse.mc-stan.org/t/promoting-posteriordb-into-an-official-stan-project I am calling a vote on the inclusion of posteriordb as a core Stan project. For those who didn’t follow the original discussion, here is what it would mean and the main disagreements that remained. A good summary of the main arguments and points of disagreement is IMHO this post by @mans_magnusson.

To very briefly summarise the summary: posteriordb is designed to test inference algorithms across a wide range of models and data sets. It stores models and validated results to compare against and provides API to access them. Everybody agreed that having such a database is useful for Stan.

The only major disagreement from the discussion was IMHO about scope of the project, specifically about the potential for scope creep/feature bloat. The points in this direction were made by @betanalpha. If I understand Mike correctly, he would like a relatively tight specification of the project and its goals before promoting it. If I understand @mans_magnusson and others correctly, they consider such tight specification premature and not necessarily useful/desirable and are OK with a higher-level description allowing for more agility in future development, e.g. “Testing implementations of inference algorithms with asymptotically decreasing bias and variance (such as MCMC)” and “the goal of posteriordb is the best possible Bayesian inference” (the goals are currently summarised at https://github.com/MansMeg/posteriordb/blob/master/doc/use_cases.md).

There was also a concern about package architecture especially R-first + interfaces for other languages (current state) vs. full plugin architecture (proposed) and about code sharing/dependencies. Those were not completely resolved either way, but I personally believe such development decisions can safely be considered a day-to-day devel business and not be decided together with acceptance (your opinion may vary).

As a consequence of accepting the project, its developers will become Stan developers (and thus e.g. eligible to vote). The main posteriordb developer who is not already Stan developer is @mans_magnusson. There are other non-trivial contributions (jarnefeltoliver, eerolinna, who AFAIK are not users on Discourse), but since it is unclear whether they want to become part of the Stan developer community and their contribution is not obviously fundamental, they would go through the usual process (which is currently not completely defined) of approving new developers, should they want to join the ranks.

An additional goal of the vote is to record consensus on acceptance (if it is there).

The discussion leaves us with three options to vote on:

  1. Promote posteriordb into an official Stan project in its current state and goals and plans as outlined in the discussion thread.
  2. Promote posteriordb into an official Stan project conditionally, only if its scope, goals, implementation or plans are made more specific and further narrowed only to those immediately relevant to core Stan mission. A further discussion would follow on how exactly should this ‘narrowing’ look like and whether the posteriordb team would like the project to join Stan under such conditions.
  3. Do not promote posteriordb into an official Stan project

The voting period is two weeks, but with regards to holidays in some parts of the world, I would push it back a few days, so that voting is until January 9th. If any option gets more than 50% votes (not including pepole who abstain from voting) it is selected, otherwise there will be a second round among the top two options. Tagging the whole @Stan_Development_Team to consider voting, if they feel sufficiently informed about the issue (this vote is not restricted to a specific module).

The voting will be done with Discourse poll below (the options are shortened compared to the full version above, but the full version is important).

  • Promote posteriordb as is
  • Promote posteriordb conditional on narrower scope/further clarification
  • Do not promote posteriordb
  • Abstain from voting

0 voters

The “Abstain” option is the same as not voting at all, but it lets me understand how many people missed this post (and thus I should nudge them to vote) versus how many just don’t want to express opinion on the topic. You obviously don’t have to participate in any way.

Thanks everybody for their participation and patience so far.

5 Likes
2

@martinmodrak, if this is an official vote, would it be possible to do this through Helios, just like how we voted for the SGB?

Edit: I’m not trying to add more work for you. I believe someone has already set up the infrastructure to handle votes (@jonah?) via Helios. Is this vote a different class of vote than voting for the SGB?

3

The currently approved trial of the development voting procedure! expects voting via forums. Agree that Helios might be preferable if it is not too much hassle to setup, but I wanted to follow the agreed-upon procedure…

EDIT: to clarify, I think we might consider other voting methods for future votes, changing method mid-voting is IMHO hugely undesirable unless someone makes the case this particular vote is too personal for voting on the forums. I am also honestly unsure whether anonymous voting is a good option for technical votes - in contrast to votes about people where the benefit is clear.

1 Like
4

Thanks for setting this up.

Yeah it was specifically in the proposal that we would try voting via the forum and nobody objected to that. However, since this is a trial period it can definitely be changed going forward if a different venue is preferred. But I agree with Martin that we should do this initial vote the way it was laid out in the proposal.

Personally I think votes in elections and votes on technical/development issues are quite different, but that’s a good question to consider. I’m not sure if others would agree with me on that.

Personally I think voting on technical issues would be best done in public whereas it makes sense to vote in private for elections. But if enough people prefer voting in private for this too then that can be changed for future votes. It’s not hard to set it up via Helios if that ends up being what people prefer. But personally I like keeping development related votes public.

3 Likes
5

Thanks, yeah I think it’s important that everyone is at least fully aware that there’s a vote happening. If they then decide to abstain that’s totally fine.

1 Like
6

Thanks! It’s been too long since I saw that… glad it was already thought about. Thanks for putting up the actual thread. =).

2 Likes
7

I like open votes. In my years in various academic departments, I’ve seen open votes and closed votes, and open votes seem to be associated with more open discussion, while closed votes can get tangled because of the uncertainty.

8

I also didn’t realize that Helios doesn’t offer open voting. My fault. Definitely think this is better in the open.

1 Like
9

I have to admit that I am frustrated with this vote, especially given that @martinmodrak posted the vote without appealing for feedback on the vote text from the main discussants. From a more global perspective don’t think developers calling votes from a moderator perspective is not particularly constructive, and on a more local perspective I don’t think that the vote text is particularly faithful to the conversation. To be clear I believe that in this context I would have no problem from vote call from @avehtari or @mans_magnusson, especially if the terms of the vote had been discussed.

In my opinion many of the states goals included in the vote text to motivate immediate inclusion of posteriordb are ambiguous as best and speculative at worst. The computational scope of Stan is extremely well-defined by mathematical constraints, and building projects that don’t respect that scope will inevitably lead to problems. With reasonable changes, however, posteriordb can align very naturally with that mathematical scope and become an extraordinarily productive contribution.

10

It is true that I took some risks by not running the exact wording by the discussants and the responsibility for any potential bias in the wording is mine. It would be useful for me and likely for others if you could point out where do you believe my vote text is not faithful to the conversation or provide an alternative (so that others could decide it they would vote differently if your wording was chosen). I think it is important that community members are open about their frustrations and thanks for voicing yours. However I don’t think that I didn’t “appeal for feedback” or that I somehow circumvented the main discussants.

To give receipts: on Dec 5th (17 days before the vote) I first posted that I believe vote is the appropriate next step, provided my understanding of the remaining disagreement and outlined the main features of the vote (you were quoted in the post so I believe you must have been notified about it). This was approved by @mans_magnusson on Dec 15th. I reiterated my intentions on Dec 18th (4 days before the vote) and explicitly asked people to let me know if they want to collaborate on the wording. I got one reaction in private which was roughly “the discussion/voting process is too slow and cumbersome” and described the outline of the vote as favoring rejection of posteriordb. Over this time frame, I didn’t get any reaction from you or an indication that you plan to react.

So I see a few possible complaints underlying your general statement:

  1. The final wording differs in an important way from my Dec 5th outline - if so, could you clarify what do you believe should be worded differently?
  2. I didn’t seek feedback vehemently enough (e.g. I didn’t remind you via a personal message). Putting some extra effort into getting feedback from people most likely to disagree is something I often try to do and consider best practice. I admit I have fallen somewhat short of my own standards here.
  3. Similarly to the previous point, I should have explicitly encouraged people to let me know if they want to react, but require more time to compose their thougths. Posting something like “Hey, I need X more time, please don’t close the discussion yet” is also something I would consider good practice in general and I think should be encouraged whenever possible.

Is any of those what you had in mind? Or am I misunderstanding something?

How to move forward?

As described in the voting procedure, the start of the vote does not mean an end to discussion and if people are convinced by Mike’s arguments or are on the fence and belive further discussion to be useful or just don’t like the process so far, I encourage them to change their vote to “Don’t include” unless their concerns are addressed. If anybody is concerned, but does not for any reason want to state their concerns publicly, please reach out to other community member you trust or to any member of the SGB to voice the concern (or simply note the presence of some concerns) on your behalf. I promise to take any such concerns seriously. There is also the option to seek formal arbitration on the voting process with the SGB.

It was my goal to represent this as the main disagreement in the vote: posteriordb devs believe the same goals are sufficiently precise for the time being and I assume people voting for option 1 agree. Do you believe my wording does not make this clear? I encourage anybody who agrees with Mike that the goals are ambiguous and should be refined prior to promoting posteriordb to choose option 2 or 3 in the vote.

My understanding is that the posteriordb devs (and likely others) consider most/all of your technical points valid and worthy of consideration in further development, but disagree that they need to be addressed before posteriordb can be promoted.

P.S.: Can someone who is not an admin/mod try to change their vote? The system let’s me change mine, but I am not 100% sure it works for non-privileged users (somewhat unintuitively you need to click the “Show vote” button to change your vote).

6 Likes
11

Hi,
just retagging the whole @Stan_Development_Team - those who already voted, because @betanalpha did express some concerns about fairness of the vote in hist post above, so you might want to check if you want to change your vote based on this critique. And those who didn’t vote, please consider voting if you feel informed about the issue. The vote closes on Jan 9th anywhere on Earth.

Happy new year everybody.

2 Likes
12

Yeah I voted to include it but I definitely still think Michael’s points should be addressed in future development.

4 Likes
13

I agree with Jonah. Also I think the vote was fair. Lots of people have been talking about posteriordb for awhile, we had a vote, and it will be good to move forward.

14

Seems a valuable project. I don’t see why it would be part of Stan. It’s not like we can run SMC, PDMPs, Gibbs, MALA… in Stan. Surely it stands on its own.

1 Like
15

TL;DR I think Michael makes some good points and a good few of them could be addressed by adopting some of the RFC style of processes

idk my man, my worry about making it a precedent that the submitter needs to gather feedback from all the discussants before making a call for a vote is that it feels almost like a pre-vote vote. What if the main discussants don’t agree on the wording? What does main discussants mean here is also sort of too vague for me as well. Maybe this voting scheme should adopt something like the RFC approach with a proposal template, time for comments, then a call for vote in the same thread.

I’m not sure I understand what’s wrong with Martin calling for the vote. Like I’m thinking in terms of the Rust RFC style, in the Rust RFC terms I think Aki and Magnus would be the leads while Martin would be the “shepard” for the issue. So I think it’s fine for Martin to post it.

That’s def a big issue. It’s also kind of why I like the RFC process more because then I think anyone voting can see the original proposal + comments on it. I’m not sure how to handle that in the discourse format

Do you a link to a past comment describing these changes?

Maybe we need like a pro/con section that anyone against the vote can add to?

4 Likes
16

For more significant changes like including new projects into Stan I think the RFC approach would be really valuable. In my opinion it’s been exceedingly successful for the more recent math projects, not only in clarifying design but also making the goals and scope explicit which in my opinion is the more significant problem here.

There is for example my last comment, Promoting posteriordb into an official Stan project - #38 by betanalpha.

In my opinion the challenge here is that some of the key technical points cannot be divorced from whether posteriordb is an appropriate Stan package.

The goals directly related to Stan, such as performance testing and validity testing require a very rigid database schema. This is based purely on the mathematics of Bayesian computation and the core functionality of Stan – users specify models and Stan tries to estimate expectations values as accurately as possible. Exactly how benchmarks are integrated into performance and validity testing, especially in the core library and across interfaces, is very much an open question that can be discussed independently of the vote. Same with the exact details of how fits are validated and how validation information is saved. But again the relevant user-facing schema – model, function, validated expectation value – is fully specified by the nature of Bayesian computation. I would hope that this isn’t controversial.

My problem with the proposal, and hence the appropriateness of a vote, is not that the proposed scope of posteriordb exceeded to this scope but rather that the proposed scope was not precisely defined, referring to unspecified goals of other projects and other speculative applications. If the full scope of the project were precisely defined then we could have a well-defined circumstance for a vote. Instead we’re voting on a whether or not posteriordb seems like a relevant project which to me is far too speculative. In fact in his last post @avehtari noted changes to the proposed scope that were motived not due to internal discussion but rather external discussion!

Perhaps most importantly what exactly is the utility of voting before the scope has been precisely defined? All a premature vote does is give the project a commitment based on an assumption is that the scope will eventually converge to something appropriate to the project. Because being a part of Stan will not bestow additional resources to the developers to which they don’t already have access, there’s no cost to waiting for posteriordb to mature to the point where it’s scope is precise and can be evaluated. If anything including the project prematurely only introduces the risk that the posteriordb developers will converge to a scope that conflicts with the goals of Stan and we have to deal with resolving those conflicts.

17

The voting period has run out and there is a clear majority. Of the 18 people voting, 15 (83%) supported option:

Which is the accepted option. Thanks everybody who participated in the discussion and voted. One thing I think got lost in the discussion is that accepting posteriordb is not only investment/commitment from the Stan community towards posteriordb, but it also means that posteriordb developers have given up on tight control of the project and gifted their work to the community, so special thanks to @mans_magnusson, @avehtari and everyone else for their work on the project.

This also means that @mans_magnusson is now officially a Stan developer, I am adding him into the Stan Development Team group here on Discousrse. I am not sure who has the neccessary GitHub privileges to make the actual transfer of the repo under stan-dev - does anyone know who should do this?

2 Likes
18

That is great news! Maybe @jonah knows? Do you need anything from me?

EDIT: No problem, @rok_cesnovar reached out. We will set this up Thursday. Thanks.