Revised SGB proposal for voting on technical issues

Stan Governance
1

@Stan_Development_Team and Stan community,

On behalf of the Stan Governing Body (@SGB) I’d like to share this revised proposal for a voting process on technical issues. We have a separate proposal coming soon that seeks to add more clarity on what Stan development is and who developers are, but the goal of this proposal is to outline a plan for how to settle potential disagreements on the technical direction of the project.

Feedback from current Stan contributors, potential Stan contributors, and anyone interested in the future of Stan development is greatly appreciated!

Background

As it stands right now, Stan does not have a formal process for managing disagreement on technical issues. Although the project has been successful to date, there is a benefit to having an agreed-upon procedure in place to manage serious disagreement on technical issues. To this end, the SGB is aiming for a lightweight solution that is not burdensome for developers, but that will provide a mechanism for reaching a decision when informal consensus appears unlikely. Visibility into how technical decisions are made also helps onboard new developers to the project.

Below is a proposal for a voting system for making technical decisions in case of disagreement among developers.

When should you call a vote?

A vote is only necessary if there is still disagreement on a major issue after ample discussion. For the most important decisions, a vote should be called even if there is no disagreement in order to create a record of the consensus.

We realize there is ambiguity on what constitutes a “major” issue or “ample” discussion. Some judgement is required on the part of the initial developer. A working definition is that an issue is “major” if it may create a substantial maintenance burden on other developers or introduces any user-facing changes that may be controversial (of course there is ambiguity about these definitions too). “Ample” discussion means that the issue has been raised, and other developers have had an opportunity to read, process, and chime in. If other developers think the vote was called too soon then they can vote “No”. A proposal voted down can be brought up for a vote again after subsequent discussion.

If you’re not sure if a vote is required, err on the side of collaborative decision making by calling a vote. However, keep in mind that voting is intended as a fallback and you should first make an attempt to understand and address the technical concerns through polite discussion.

Who can vote?

  • Votes are open to developers contributing to the relevant module (see bullet point below) but any developer outside the module can force it to a developer-wide vote by objecting. Developers calling for a vote also have the ability to nominate multiple relevant repos initially.

  • A module is a repository on stan-dev. (This is a starting point. Depending on how this goes and how often modules need to be grouped together this can be revisited.)

  • The exception to this is for major changes to the Stan language, in which case:

    • The proposed change should be posted on Discourse to allow the entire community to comment
    • The voting will be open to all modules.
    • Note: this is for major user-facing language changes and does not pertain to implementation issues, which are the purview of the language module developers. An example of a major user-facing change is a totally new syntax for declaring arrays (New array declaration syntax). An example of a minor user-facing change/addition is vectorizing binary functions (https://github.com/stan-dev/stanc3/issues/643).

  • If you are eligible to vote it is up to you whether or not to vote, but please only vote if you think you are sufficiently informed about the relevant issues.

How does voting work?

  • Platform for voting: This is TBD, but for the initial trial period we will try out the “poll” feature on Discourse (https://meta.discourse.org/t/how-to-create-polls/77548). Other suggestions are welcome.

  • The following procedure will be used:

    • Someone proposes a vote and contacts SGB
    • The SGB appoints someone to run the vote or recommend to the SGB that the vote not happen
    • That person sets up the post with the poll, notifies voters, and reports the results after the vote

  • A poll is open for 2 weeks in order to give everyone enough time. If a lazy vote fails and a majority approval is required, please create a new poll and note that it is requesting majority approval.

  • Three vote options are available:

    • +1: Agree
    • 0: Abstain (the default)
    • -1: Disagree (you must list your reasons for disagreement in a comment)

    (We may need to adjust this if a particular topic in question can’t be expressed as a yes/no vote and requires more options.)

  • Proposal phrasing: Please include a statement that makes it clear what it means to “agree” or “disagree.”

  • Lazy vote: A binding agreement is reached when there is a minimum of one +1 and no -1s. In other words, if anyone is in favor and no one is against then the vote passes.

  • Majority approval: If a vote does not pass after a lazy vote, then the issue will require majority approval, where a binding agreement is reached when there are at least two +1s and more +1s than -1s. However, majority approval should be considered a last resort. The aim should be to reach rough consensus through discussion. To facilitate reaching rough consensus, voters are allowed to change their vote during the voting window if they change their minds.

  • Every voter’s vote has equal weight and each person can only cast one vote on a given call. (The developer bringing up a topic for a vote can also cast their vote.)

Arbitration

The voting procedure is intended to help resolve developer disagreement but it is possible for disagreement to happen about the process itself. In this case there is the option for arbitration, which works as follows.

  • If at least 3 developers consider any part of the process leading to a particular vote problematic, they may ask the SGB to arbitrate. An arbitration request stops the voting period if it already started.

  • The SGB may designate a community member to serve as an arbiter either for a specific case or to generally handle all arbitration requests for a module. The arbiter will act on behalf of the SGB and notify the SGB of the resolution of any arbitration.

  • The arbiter/SGB will consult all parties involved and then make a final decision on how the vote should proceed. That is, the arbiter/SGB will not decide the technical issue itself, but rather resolve issues related to the process, e.g., when a vote is called, the order of voting on multiple proposals, the final wording of the proposal, etc.

14 Likes
2

I like it. Having a formal system for voting would give devs an opportunity to formally decide whether this is something we can agree on working together to implement, review, maintain and advise on the forums.

If I’m understanding correctly, a lazy vote only requires a minimum of +1. This may be a bit lax, as in the math library implementation (advising, reviewing, or writing code) is almost always more than one person. So my only critique is that may be more than one +1 should be required for a lazy vote. We need to have more than one dev enthusiastic about a project for it to be successful, because this is a group effort.

For example, I’d dislike for a project a colleague proposes to turn into squandered time because it was voted in lazily, and no one wants to, or has the technical background, to review and maintain it. I think we have to be committed to some extent to have some fluency in at least a few other domain areas of the project, so we can help negotiate obligations a bit.

But mostly, I’m a big fan.

3 Likes
3

overall, this is great!

questions:

  • does a “major” issue require a design doc, pre- or post-hoc?
  • do disagreements about how much testing and general proof of concept required constitute “major” issues?
4

Good point. Requiring only 1 approval could be too lax. Any suggestions for a minimum number of votes in favor?

Good question. I’m certainly open to requiring design docs but maybe we don’t need to officially require it (should definitely be strongly encouraged though). Going through the process of a design doc is a good way to work towards rough consensus and work out any issues that would cause people to vote No on your proposal. If someone doesn’t want to do a design doc that just drastically increases the chances that their proposal will get voted down because they won’t have received the feedback and buy in that the design doc process offers. But I could be totally wrong about this and maybe officially requiring them is a good idea. I’d lean towards trying this out and seeing how it goes and then revisiting the question of making it a requirement. What do you think?

Also a good question! I’m not sure. What do you think @mitzimorris? What do other people think? It would be great to get more thoughts on this.

5

Thanks, SGB, both for working this out and for posting it here for discussion!

6

Thanks! This looks great and exactly what I was after. I like that you’ve volunteered the SGB to resolve disputes.

The case where there are no -1s is handled by “Lazy vote”, so the “at least two” is redundant here. As to content, a +1 and no -1s means nobody objects, right? I don’t see a problem going forward with that. It’s going to be hard to collect abstentions.

De facto, proposals will need at least enough of a design to vote on. If it’s not clear what I’m voting on, I’m just going to vote “no”.

With “post-hoc”, I think @mitzimorris is alluding to people writing all the code for a PR and then start discussing it. It’s hard to tell someone who’s put in a lot of work that you don’t want their PR. But we’re just going to have be tough and keep doing this and all the while remind people that they should be writing proposals, issues, etc., before big wadges of code intended for a PR.

If the proposal doesn’t propose enough testing, we can vote it down. If there’s not enough of a proof of concept, we vote it down. So the voting mechanism itself should self regulate this. Different people agree on the right level of these things, which is precisely why we have voting.

Also, this shouldn’t be sidestepping the code review mechanism we have in place. Of course, that’s usually just one person, and they might be a shill. People asked me not to review your PRs. Similarly, cliques of devs can decide to approve each others’ PRs and get around general code quality standards. That’s when we go to the SGB.

So I think this is OK without clarifying specifics on how much of a proposal is presented for design, how much testing is being done, etc.

3 Likes
7

I like it as well! Agreed with all previous comments. Thanks @SGB and anyone else involved in coming up with this proposal.

1 Like
8

Thanks @jonah and the SGB, overall this looks like a great place to start and I think the a trial implementation would go a long way towards identifying any major problems.

Two comments.

I appreciated that in previous discussions “who can vote” included an explicitly statement about when one should vote. In other words not only that you’ve contributed but also feel like you are adequately informed on the issues being discussed. The challenge with the Stan project is that we’re combining state of the art statistical computation, probabilistic programming, automatic differentiation, and C++ programming and many issues will fall into intersections in which not every developer is experienced.

I would also suggest that “major user-facing changes” include the interfaces (including for example the math and algorithm APIs) in addition to the languages as experienced users will have just as much to contribute there, as well.

3 Likes
9

Just wanted to say thanks to everyone for the comments. We’ll be revisiting this soon (after StanCon is over), making some tweaks based on these suggestions, and then we’d like to start a trial implementation to see how well this works in practice.

10

After rereading this, it seems like there’s another thing that I think could use a vote… something where we record that we want something to change. There may not be a disagreement, but we should indicate to everyone with the ability to vote that something big is changing. It could be a change to the language, something that affects users (updating minimum requirements), or something beyond what I can think of at the moment.

I think what’s written (with comments from everyone) is designed for getting out of deadlocks. I’ll reply in depth about some of the other issues with the voting mechanism presented. It might still be the one that’s adopted, but it’s good to know what we’re adopting.

11

Yeah that makes sense. If there are big changes coming then even without disagreements a vote is a good way to officially record the agreement on the change. Is that what you were saying or did I misunderstand?

Thanks, that would be great.

12

Yes. Thank you for translating that to something more understandable!

1 Like
13

For these reasons, any votes regarding stat comp, probabilistic programming, or autodiff, I give my votes to @betanalpha. Any C++ issues I give my votes to @stevebronder. So they get +1 on top of their addition votes on any of these issues, which includes mine.

1 Like
14

this is called proxy-voting and I would vote against it.

3 Likes
15

fair

16

Yeah it looks like this wasn’t in the revised version but it’s important. I just added a bullet point to the proposal above that says:

  • If you are eligible to vote it is up to you whether or not to vote, but please only vote if you think you are sufficiently informed about the relevant issues.

Yeah I agree with this. What do other people think about including major interface changes in the “major user-facing changes” that anyone can vote on?

1 Like
17

Hey everyone, just wanted to check and see if anyone else has feedback on the proposal. Beginning next Monday (Sep 21) the SGB would like to start a trial period that lasts several months, so this is the last chance for comments and suggestions before we start that. Thanks!

1 Like
18

All this makes sense to me. I’d maybe add a layer of indirection, where someone proposes that something goes to a vote and a person nominated by the SGB runs the vote with the authority to ask the SGB to not allow a vote. This should short circuit someone who wants everything that isn’t going their way to be a vote.

2 Likes
19

That seems like a good idea. So the process would be

  1. Someone proposes a vote and contacts SGB
  2. SGB appoints someone to run the vote or recommend to SGB that the vote not happen
  3. That person sets up the post with the poll, notifies voters, and reports the results after the vote

@anon75146577 Did I get that right?

Does anyone have any objections to that?

3 Likes
20

based on one of @syclik’s suggestions above the latest version includes a new sentence:

2 Likes