Stan Governance


#82

This has been ongoing, but we’ve stepped back in the usual bureaucratic/parliamentary fashion of first trying to figure out how we’ll make the decision (yes, there’s an xkcd for this that I quoted before, but I’m out of steam for linking).


#83

+1. Just wanted to say that I think this is very accurate.

I haven’t found a good one. From what I understand, the fork probably won’t go far (immediately) on technical merit. It seems like it’s forked due to the disagreement with the way the situation was handled.

A few things stood out (not in any particular order):

  • the structure they used for managing governance seemed reasonable: two GitHub repos with issues that are open to read, but only the group can comment on.
  • difficulty with code of conduct is that it needs to be both as unambiguous in interpretation and it needs to be taken seriously. That’s going to be tough.
  • there are certain discussions that should be closed, but I think a key mistake in this case was that it was too closed and there should have been more open discussion, especially with the list of complaints
  • governance is hard
  • it looks like this happened due to a couple mistakes with overreactions and not so clear discussion as to how serious mistakes were / weren’t. That seems like something we should avoid.

I don’t think we’re large enough to need a complicated structure, but I think it makes sense for us to start putting some process in place. That said, I don’t want us to have process / code of conduct / governance that isn’t enforced.


#84

I understand why the Node.js circumstance induces further uncertainty in the implementation of a code of conduct, but I also think it demonstrates just how critical the issue is.

I don’t want to start a long discussion regarding the specifics of the Node.js case, other than to use it as an example of how easy it is to slip into a toxic environment. The developer in question had a long history of posting and supporting articles similar to the one that recently lead to the Google controversy — shitty arguments built out of shitty science that make bold conclusions whose immediate implications are the dismantling of any system trying to facility diversity. Ignoring the validity of these arguments, the ultimate conclusions are used to argue against the very existence of underrepresented groups in these communities, making it a very unwelcome place. At the same time the people making these arguments or supporting them hide behind mistaken calls to “free speech” (not relevant in a non-governmental setting) or “open discourse” (not relevant because of the asymmetric circumstances of the discussion. Long story short, these actions are classic dog whistles (actions easily ignored by all but the targeted parties).

What happened in the Node.js case is that numerous women who were getting into the Node.js community started complaining about the particular developer. And fairly, I would argue, given that I can’t imagine how can you feel that your contributions are considered fairly when the people in power to approve pull requests spend their free time sharing articles arguing that your efforts aren’t worthwhile. Enough people complained that a formal call for this voluntary resignation was made, the only possible resolution according to the Node.js organization bylaws. Of course the accused developer dissented and no one else in leadership had the power or desire to push him out otherwise.

I think that this highlights two very important points.

Firstly someone doesn’t have to physical or directly emotionally attack someone in the Stan community to make the community toxic. While we all know of people in academia who have gotten away with such atrocious activities, a different but similarly powerful toxicity is a vocal group of people consistently questioning the relevance of other groups and no one else in the community making it clear that such behavior is unacceptable. The immediate conclusion of this is that the intent of the code of conduct has to be enforced. When we see people making unacceptable claims on the Stan forums or issues list, even ones that seem mild, one of the devs has to respond noting that such behavior is not acceptable. Yes this will cause some people to leave, but I’d rather err in that direction rather than facilitating an healthy environment for underrepresented devs and potential devs.
This would all happen long before any calls for resignation or formal punishment of any kind is made.

Secondly this highlights the issue that it may be impossible to decouple Stan the project from some of the more prominent core developers. Andy’s blog is not under the purview of Stan — if extremely hypothetically he were to start posting papers making bad evolutionary psych arguments against female developers that would immediately reflect upon the Stan project. Even if he hasn’t done anything unacceptable within the confines of the GitHub pages or Discourse or even the weekly meetings, the damage would be done unless the Stan project not only was able to remove him but also put out a strong statement against that behavior. Same with me and my presumably private Twitter account, or the courses that we’re teaching that don’t have to be approved by the Stan project in any way. Consequently it very well may not be enough to limit behavior outside of “official” Stan media.

This second point also raises a serious issue about the ultimately sustainability of Stan. If some of us were to leave angrily or be forced out, Stan would very much become a different project. The name would be the same, but the evolution of the project could drastically in change just as much as if we forked the entire project. Which makes me less and less against a beneficial dictator (or board of dictators) for life approach, at least for high-level decisions like devs being asked to leave. If someone on the board became an issue the project might have to fork anyways, so what’s the difference?

Anyways, this stuff is hard and this stuff sucks but we have to do it or we leave ourselves open to these problems. The code of conduct didn’t cause the problems in the Node.js community — even though it ended up not being enforceable it allowed the identification of a toxic personality that was preventing people from underrepresented groups from contributing and incited a fork of the project to remove his influence. Without a code of conduct it’s likely that Node.js would have continued unforked but with a far more exclusive and toxic community about which even good devs would have shrugged and unfortunately accepted.

My take aways:

  • An unenforceable code of conduct is better than no code of conduct at all. If we get to the point where we have serious disagreement about enforcement then we’re already at a point where the entire project is threatened.

  • A legally valid, enforceable code of conduct with clear policies would be ideal, but also nigh impossible given our resources. In my opinion the argument that we have to wait for such an ideal to adopt any code of conduct is terrible given the above.

  • The spirit of the code of conduct has to be a serious concern of (at least) most of the Stan dev team and we have to be willing to make that clear by addressing any issues early and often. In particular, if complaints are made about dev behavior outside of formal Stan media then the Stan organization has to make a statement addressing the issue.

  • In that event it is unlikely we will have consensus and we’ll need some official voting procedure. While it would be great to have something more formal with set term lengths for voters and procedures for becoming a voter, an initial beneficial-dictators-for-life board would be a reasonable starting point. Again, if there is too much disagreement there then there are probably bigger issues for the project.


#85

I agree with Michael that 90% of the work here around ethical/legal issues happens before the governance structures get involved. Some of the other details I’m not so sure about because we could factor some of these responsibilities:

  1. electing a final arbitrator for ethical issues would give us a place (for now at least) where final responsibility rests. I trust us to choose a person with reasonably modern understanding of ethics and oust them if they’re not doing the job. OTOH as Michael and others have pointed out an ethical issue actually arriving on the doorstep of the final arbitrator is really a community failure to begin with so we need to do some work to address that.

  2. legal issues are just work, that’s something that can get handled by a board of 3-5 people and they don’t have to understand the technical parts of the project all that well. We have some people to choose from (and we already have such a thing via the board required by NumFocus).

  3. technical decisions are also just work, they should be made by the sub-group working on the specific sub-project. We have semi-formally made these designations for all the core components now. We could choose the decision-making sub-groups by consensus since those decisions haven’t been a source of disagreement in the first place.

  4. Turnover in contributors is unavoidable and we should learn to bring on people who could step up and handle components of the project as current contributors inevitably leave.

The CoC I agree about: 1) adopt one we can buy into wholesale; or 2) start by writing down the components we all agree on. That should at least bring unaddressed issues to the front.


#86

+1

There are also things specific to this community that might not be covered explicitly in an off-the-shelf CoC. I’m thinking of scientific issues.

For example, there is a solid group of people associated with Stan who have strong negative opinions about hypothesis tests (and other things, but let’s stick with one for the moment). I think a Stan CoC should cover the sorts of things that are in the board FAQ (no personal attacks, don’t be belligerent, etc). I personally think this is as much an issue when looking at potential things that will toxify the Stan community as the things @betanalpha listed above. (See some of the R mailing lists)

If people are scared to talk to us on scientific matters because they think they’ll get their head bitten off, we can’t grow the community or make any reasonable claim to diversity.

I’m uncertain about this. I agree with what you said, but this seems to be beyond the purview of a Stan CoC - this is a job not a lifestyle choice. But as Dianne Keaton says in Looking for Mr Goodbar (unfortunately before she’s murdered), “When your nightlife interferes with your daylife…”.

I just can’t work out how you could codify a mechanism for this that wouldn’t be massively open to abuse. (Or disastrous node.js-style failures)


#87

I don’t know how to codify the spirit of anti-toxicity we’ve managed to maintain on our mailing lists, etc…, but I wish I could bottle it.


#88

I don’t think it’s about codifying it, it’s about spelling out the types of things that, in the spirit of the anti-toxicity of the mailing lists etc, are not acceptable behaviour and a process for both reporting and dealing with people who do things that aren’t acceptable.


#89

That’s what I mean by codifying. It’s easier to say what we encourage.


#90

I think the basic point is just to treat everyone with respect. For example, rather than replying “where did you learn statistics, you idiot”, you say, “that’s a subtle point, and here are some pointers to understanding it better”. It’s just as easy and nobody’s feelings get hurt.

This becomes challenging in a few situations:

  • users who are very demanding in that they want us to help them and don’t want to put in any work themselves,

  • users who keep telling us we’re wrong or that we should do X, Y, or Z if we want to be serious about Stan (I appreciate feature requests, not being lectured about priorities), and

  • developers who disagree.

I remember the first comment I recall from some of the early code I wrote to implement multivariate normal densities, which said, roughly, “Who wrote this idiotic textbook linear algebra code?”. Uh, that was me—I just copied the definition, inverses and multiplies and all. I didn’t know what I was doing. I perservered because I’m thick skinned, have a fair bit of self confidence, and know that developers tend to be very harsh when speaking to one another about code.

We used to have our students formally review each others’ papers in our second-year M.S. and Ph.D. course at Carnegie Mellon; their ripping each others’ papers apart made the comments above seem tame.


#91

I think of that as step 2 of 3(+) in learning to review. Step 3 is where you figure out how to express direct criticism without making it personal or combative and where you figure out which criticisms should be addressed first to improve the work. It means you need some perspective on the work… which most graduate students don’t have. Probably the same problem in code review. :P


#92

Step 1 is probably figuring out you need to cough up an informed opinion.


#93

Now that summer is dead, did this conversation actually go anywhere? Can someone start a new thread(s) about this? In my glancing interactions with Stan, governance has come up more than once…


#94

Sure. Happy to start one or more if you don’t want to kick them off. I think we might have a better chance at making decisions if the discussions are more narrow. Any particular parts of this discussion you think are most pressing?


#95

Well, the first thing to sort out is what we need to sort out.

We have an advisory group for NumFOCUS already. NumFOCUS holds the trademark and funds that go to NumFOCUS.

Andrew’s empire at Columbia doesn’t fall under Stan governance or vice-versa, so that’s not relevant.

What is relevant:

  • software governance
    • how do we decide what to commit
    • when do we do releases
    • who counts as a developer
    • how do Stan subprojects coordinate
  • trademark governance
  • web page, case study, and other doc governance
  • opportunities that come into the project as a whole (consulting, teaching, papers, etc.)

For software governance, I think we should put one person in charge of each repo and let them decide how to run it. That’s why I’ve been keen to break the stan-dev/stan repo down into the language, algorithms, and services components. I suggest repos because it’s the easiest thing to control permissions for and organize within GitHub. I want one person in charge for a clear decision-making path.

Breck was suggesting we follow the Fogel book and vote on everything among developers (and maybe the developers are defined per repo).

Sean has been arguing to combine repos rather than split them, making what he’s been calling a monorepo. I’m still not understanding why that would be a good idea (he said it’s partly having to do with wanting a single history, but I don’t understand why that’s a good idea or what exactly it means given that submodule versions are tracked).


#96

“monorepo” case argued at length here: https://danluu.com/monorepo/

Everything from issue tracking to releases scales at best linearly with the number of repos. As the person doing much of that “devops” stuff, I feel the pain most acutely perhaps.

With one repo you can culturally set permissions on a per-directory basis, though it would be nice if github had something automated.


#97

Google’s (dubious) AMP project implements something like this on Github
by putting an OWNERS.yaml file in each directory. I gather they have a
bot that makes sure one of the “owners” of code in that directory
reviewed a PR. I would wager the bot is open source.

See:


#98

In what decade was that written and is it still true? I can’t even fathom how big a Google monorepo would be. How does someone download it? How do they control access to all the stuff they lock down (not that we have that problem). That is, how do they maintain a monorepo without it really just recreating the logic of multiple little repos inside for permissions, etc.

How does this fit in with having developers do work on their own repos then send it in rather than anyone working on branches on stan-dev?

that forces you to define what a “project” is for your particular team or company

We are going to have to do that anyway to decide who gets merge permission on each “project”. Negative impact of mono-repo, I’d say.

With a monorepo, projects can be organized and grouped together in whatever way you find to be most logically consistent,

For us, that’s nesting and GitHub lets you do it. No saving as far as I can tell. How do you solve the version A of stan math with version B of stan and version C of cmdstan problem with a monorepo on GitHub?

With a monorepo, projects can be organized and grouped together in whatever way you find to be most logically consistent,

I don’t see how that’s the case. We can navigate them now in the outside on GitHub and each gets its own Wiki, etc. In a monorepo, all this would be merged. We’d get one single wiki, one signel Readme, etc. at the top level of GitHub.

It seems like an advantage to actually be able to set permissions rather than relying on culture.

Multi repo setups usually have two separate levels of navigation – the filesystem idiom that’s used inside projects, and then a meta-level for navigating between projects.

Not the case with GitHub projects. On my machine, I navigate with the file system. Though that’ll only work from CmdStan down—you can’t also toss in RStan, PyStan, etc., as they all get their own links to the state of the repos underneath them. That seems like an advantage to me, though.

A side effect of that side effect is that, with monorepos, it’s often the case that it’s very easy to get a dev environment set up to run builds and tests.

Testing for each of our interface repos requires different tooling.

Are they really suggesting a mono-build, too? That seems even less workable.

with multiple repos, you need to have some way of specifying and versioning dependencies between them.

This is a feature, not a bug!

I’m going to stop here, as I’m far from convinced by these arguments, every single one of which seems wrong. I must be missing something if you see some advantage in doing things this way other than on the dev ops side.


#99

May 2015 and yes, it’s still true. Google’s monorepo is giant though I do think there exists some code outside of that repo that is secret. Everything else is for everyone at the company to read.

Unrelated - those are just github forks so they don’t all have to join your organization to submit a pull request.

I don’t think that problem exists anymore, haha. If you want you can always check out a directory at a specific tag: git checkout v2.16.0 test/unit for example.

Stop, you’re making me drool thinking about how much easier it would be to find and edit information! It seems vastly superior but I don’t understand your objection here either - you want separate wikis? Why?

Agreed.

Not sure I understand this point. Also I am not advocating for PyStan and RStan to be in the same repo.

The next line explains more. I can’t really tell if you want to engage on this topic or not - you got about 4 lines into the argument, haha. But we don’t have to take this topic seriously, I just felt summoned to actually give a reasoned walk-through of the monorepo stance (which is fairly common in industry).


#100

OK, let’s back up. I must not understand what a monorepo is. I thought you wanted everything on stan-dev to get merged into a single repo. What do you want to do?

Are you suggesting we stay on Git or move to some other version control that has support for monorepos?


#101

I don’t even really care that much, I just believe in the idea that fewer repos is better. PyStan and RStan are maintained separately. They could be in a repo with the rest of it but I think my stance is basically “let’s stop making new repos if possible” and maybe “let’s merge the Stan and Math repos if everyone is cool with it.” Sorry, home sick and mildly delirious - I think I caught your tuberculosis. I sounded yesterday just like you sounded Thursday, haha.