Technical Working Group

Developers
1

[edit, 17 August 2019; I (Bob) just want to make it clear that this proposal was never ratified by the SGB and no longer reflects my views on the way Stan should be run. If there’s an official document on how the TWG is being run, please edit this document to start with a link to that document and indicate that this disucssion is outdated. Thanks. The only thing I know of now outlining the role of the TWG is this Discourse post announcing Sean Talts as new Director.]

The Stan governing gody asked me to put together a plan for the technical working group and how the project's technical aspects will be managed. Comments are welcome.

Note: I would like to change the name of my own role from “chair of technical working group” to “technical director”, but I can’t do that without approval from the governing body. I don’t want to run the technical working group as a committee, as should be clear from the outline below.

Stan Technical Working Group

Stan governing body

The governing body is a committee charged with final authority for all decisions related to Stan and controls the direct Stan budget through NumFOCUS. It is also in charge of appointing the executive director and chair of the technical working group.

The governing body has the authority to override any decisions made by the executive director and has the authority to replace

Executive director

All operational decisions rest with the executive director, who is appointed by the governing body. The governing body may replace the executive director at any time.

The executive director has the authority to override decisions made by the technical working group chair.

Technical working group chair

The governing body appoints a chairperson of the technical working group. The governing body may replace the technical working group chair at any time.

The technical group chair has the authority to override any decisions made by a technical lead.

Technical lead

Each functional module of Stan will have a single technical lead. The lead is responsible for creating the roadmap for the module and making all technical decisions with respect to their module. The technical leads are appointed by the technical working group chair and may be replaced by the chair at any time.

The technical lead has the authority to override any decision made by a reviewer in the module they are leading.

Reviewer

With sustained non-trivial development activity (this is not calendar related, but scope of contributoion related), developers will be asked to start reveiwing pull requests by a technical lead. These initial reviews will be backed up by an existing reviewer. After this trial period, the developer will be promoted to reviewer status and given permission to approve and merge pull requests.

A reviewer has the authority to override any decisions with respect to code, documentation, and testing made by a developer or contributor as part of a pull request (including another reviewer or the chair of the technical working group).

Developer

With sustained non-trivial contribution activity, contributors will be considered to be Stan developers by the relevant technical lead(s). It is the responsibility of the technical leads to recognize developers and invite them to become part of the developer team. Developers will have authority to create branches and work directly on stan-dev.

Contributor

Anyone who has submitted a pull request that has been reviewed and merged is considered a contributor. Such contributions may range from typo fixes in the doc to a new feature in C++. There is no official process for recognizing contributors—anyone who contributes is considered a contributor.

Background references

Current appoinments

Governing body members:

  • Andrew Gelman (chair)
  • Breck Baldwin (presiding officer)
  • Daniel Lee (secretary)
  • Julie Bertrand
  • Michael Betancourt
  • Jonah Gabry
  • Daniel Simpson
  • Sean Talts
  • Lizzie Wolkovich

Executive director

  • Breck Baldwin

Technical working group chair

  • Bob Carpenter

Technical leads

Core Stan

  • Math Library: Daniel Lee
  • Language: Sean Talts
  • Algorithms: Michael Betancourt
  • Services: Mitzi Morris

Interfaces to Core Stan

  • CmdStan: Mitzi Morris
  • RStan: Ben Goodrich
  • PyStan: Allen Riddell
  • MatlabStan: Brian Lau
  • Stan.jl: Rob Goedman
  • StataStan: Robert Grant
  • MathematicaStan: Vincent Picaud

R packages and extensions

  • rstanarm: Ben Goodrich
  • brms: Paul Bürkner
  • ShinyStan: Jonah Gabry
  • rstantools: Jonah Gabry
  • loo: Jonah Gabry
  • bayesplot: Jonah Gabry

Code reviewers

Technically, reviewing (merge) privilege is on a per-module basis, but I’m not listing the modules here.

  • Jeffrey Arnold
  • Ben Bales
  • Michael Betancourt
  • Steve Bronder
  • Marcus Brubaker
  • Paul Bürkner
  • Bob Carpenter
  • Rok Češnovar
  • Jonah Gabry
  • Rob Goedman
  • Ben Goodrich
  • Robert L. Grant
  • Jiqiang Guo
  • Ari Hartikainen
  • Marco Inacio
  • Brian Lau
  • Daniel Lee
  • Charles Margossian
  • Mitzi Morris
  • Vincent Picaud
  • Allen Riddell
  • Krzysztof Sakrejda
  • Sean Talts
  • Rob Trangucci
  • Sebastian Weber
  • Matthijs Vákár
  • Yi Zhang

Developers

  • Imad Ali
  • Andrew Gelman
  • Andrew Johnson
  • Rayleigh Lei
  • Edward A. Roulades
  • Dan Simpson
  • Erik Štrumbelj
  • Aki Vehtari
2 Likes
2

I’m glad to see you’re moving this forward! I think overall the structure makes a lot of sense and it’s in line with how things already work. I have some comments about the descriptions of the roles involved that maybe will help you clarify what you’re looking for.

re: Executive director: the role is defined in terms of authority but it’s a leadership role so it would be useful to also describe it in terms of responsibility.

re: Technical Working Group Chair: this is also a leadership role but I can’t tell from the text how the leadership responsibilities are different from the executive director.

re: contributor/reviewer/developer: just because people are people I think you should probably say how people are removed from these roles;

re: reviewer: I think the main challenge for reviewers is a communications challenge in terms of reconciling decisions the technical leads make with what a PR is trying to accomplish. I don’t think describing the role in terms of authority is enough.

So maybe that’s all the same comment (?): It’s hard to tell how you’re hoping this will work because a lot of it is described in terms of power over decisions.

2 Likes
3

@Bob_Carpenter, thank you for putting that together. I’ll just ask a bunch of questions.

First, I think it’s great that there is now a Technical Working Group (TWG). As far as I understand, the TWG has the authority to make any technical decisions on behalf of Stan. So far, Stan has been developed pretty organically and we’re now at a point where it makes sense to articulate a roadmap and to encourage developers to help along that direction. (The TWG has no authority over what volunteers find interesting and want to work on, but it can state what it thinks needs to be done next for Stan in order to make Stan better.) I also see the role of the TWG as declaring what it wants to accomplish and having the technical leads determine how it accomplishes that. Please let me know if any of that isn’t correct.

I think what you outlined will work. I have questions as to how this will work. (Sorry about the length of questions… I’m just trying to motivate the question and I’m trying to ask questions that can be answered.)

  • One of the development challenges we currently have with Stan is that it’s hard to change something that touches all the interfaces. For example, let’s say we wanted to change how the adaptation information is output. In order for that to work, it would have to be updated in Stan to output differently, then RStan, CmdStan, and PyStan would have to be implemented differently to accommodate. Will the TWG have meetings where we can make decisions that affect many functional modules? The outcome I’d like to see is that the TWG determines whether something should happen, and the technical leads agree to a reasonable timeline. It doesn’t have to be done through a separate meeting, but if not, how will we come to a conclusion like that and know that a decision like that has been made and it is expected of the different technical leads to come up with a plan to implement?

  • Do you expect the technical leads to create and maintain a roadmap? I think that’s reasonable. Do you expect anything else of the technical leads? For example, do the technical leads promote people to developers and code reviewers within their functional modules?

  • Right now, each of our functional modules have different contributing guidelines, different testing requirements, different devops loads, and are fundamentally different. Do you expect the technical leads to write contributing guidelines for their functional module? Or should we adopt the same guidelines across all of Stan and have exceptions?

  • Speaking of devops, if a technical lead needs some devops resources, how are they to request that? Especially when it goes across functional modules?

  • Should the TWG have its own category on discourse? Preferably TWG has the power to write, but it’s read-all? I’d expect this to be typically low traffic, but with high traffic bursts. Personally, I’d like to know what the TWG decides and that’s written out somewhere. Say we’ve decided that in Stan X.Y.Z we’ll stop supporting compiler FOO. I think that’s not a single technical lead’s domain, but a Stan thing. It’d be nice if these sorts of decisions were documented somewhere (and marked somehow).

4

What do you think that is w.r.t. the technical working group?

The TWGC can overrule/make technical decisions, whereas the XD can overrule the TWGC.

You’re right that I should add responsibilities. I see my responsibility as TWGC as overseeing the roadmap and making sure the various parts of Stan continue to work together.

Do you think it’s another responsiblity thing? Like trying your best to make sure things work and comply with Stan’s standards? I don’t think we can really put the responsiblity for pull requests working on the reviewers, though.

I’ve sort of been assuming good faith on everyone’s part in trying to make sure things work and higher-level decisions are respected.

If you have suggestions for how to describe responsibilities or what they should be, please include.

It’s like a patent, which can stop people from doing things, but can’t make them do anything.

I think the TWG should have overall organization responsibility, but I think most of the design and code direction needs to come from the technical leads. The next thing I want to do is get all the technical leads to help articulate the road map so everyone’s on the same page, so to speak.

This makes it again sound like the TWG is a committee. I don’t want to run things that way. Of course I’d want to consult all the relevant tech leads for any inter-module decisions and hope that everyone felt free to comment. I do not want to try to run the project by committee.

I think tha’s impossible because none of us control developer resources. So we can say in what order we’d like to see things happen, but I have literally nobody reporting to me in any org chart, so there’s very little I can actually make happen.

I think we need a common roadmap where the technical leads are mainly in charge of their components. I hope it’s obvious that I want to continue to solicit input for big decisions from everyone. I just don’t want everyone to have a say in the decision.

Yes, that’s what I meant by these two statements in the proposal:

I’d very much like to improve integration and unit testing across modules. Otherwise, I think it’s horses for courses in that the modules are different. So I don’t want to impose the same standards on the loo package as on the math library.

The SGB is in charge of the budget. I’ve got some budget which is currently earmarked largely for devops, which @seantalts is managing

I want the tech leads to be in charge of their modules at this level of granularity. I think we can have some overall recommendations.

Sure. I don’t think Discourse is the place to do that, though. I’d like something a little more coherent like Wiki pages.

At the same time, I don’t want the whole project to get bogged down in paperwork. I think it’s going to be a delicate balance.

5

How would the TWG stop people from doing things? Or even having a roadmap? I’m more interested in having the TWG encourage people to do things.

Cool. That’s what I was hoping you’d say.

I wasn’t trying to suggest that it run by committee. I’m trying to figure out how this works in pragmatic terms. If the tech leads have an expectation of being part of the team, respecting the team decisions (respecting the TWGC’s decisions), encouraging devs to work on things on the roadmap, and we’re able to get together occasionally to agree to make certain decisions / compromises that benefit Stan technically, then I think we can move forward quickly. If that’s describing something as running by committee, then I guess that’s what I’m thinking of. (I’m thinking more like a sports situation: head coach and the team.) If we don’t have those expectations, then how does this work? (I’m not imaginative enough and a bit cynical to believe that we’re going to get tech leads making decisions that help Stan overall if we don’t explicitly set that as some sort of expectation.)

Maybe talk me through that example… how do we coordinate a change that needs to be implemented in Stan, CmdStan, PyStan, and RStan?

Yes, but the SGB is expecting the TWG to request resources. (Right @breckbaldwin?)

If not, the SGB really doesn’t need an operating budget and the SGB doesn’t need to raise

Everything else: make sense! (I didn’t want to scatter +1s everywhere just to mark that I agree with it.)

6

We can not merge PRs. There’s no way we can make people create PRs.

First, it requires someone to write a functional spec for the change. This doesn’t need to involve a committee and in my opinion, is better off not being done by committee. It might involve asking people involved with the various modules to comment. Then we can decide whether the functional spec is a good idea or not overall, and perhaps provide comments to update it. I don’t see that we’d ever need to have a meeting with a bunch of people in a room or on a call unless things got complicated.

Take the concrete example of standalone generated quantities. It got discussed in Stan meetings and on Discourse (not followed equally by all stakeholders) and I think everyone was behind the general idea. Then it was implemented in Stan as a service. After that, nothing has happened as far as I know. None of the interfaces prioritized implementing it. I’m actually stumped as to how to proceed at this point given that I think it’s a pretty important feature.

There have been several recent suggestions for adding more diagnostic output for ADVI. I think for things like this, we decide at the algorithms level, do it, and then let the interfaces catch up with displaying it. I don’t think we need any meetings at all. But we face the same problem as with generated quantities—the interfaces are proceeding with their own features and their own priorities.

7

Here’s a relevant recent discussion around VI and diagnostics:

It’s about adding K-hat diagnostics, changing thresholds on VI runs, etc. I’m not sure it requires a committee meeting to sort out.

8

Thanks. This is a good use case. I think this is how we currently proceed and we do get stumped because there isn’t enough coordination.

One way that we could make this work is that the committee (meaning all the technical leaders) are made aware that this is an important feature, that it must be on the roadmap, and that if their functional module is part of this feature that they should be making it clear to the developers that it is important.

I can honestly tell you that I didn’t know it was important and that it was on any roadmap. I knew that some things were merged, but I don’t know what’s currently blocking it, what remains to be done, and how I could help.

I really think it would be nice if the technical leads had a different level of commitment to the project than code reviewers. I know it can’t be forced, but it’d be nice if people volunteered to take on that role. (Without it being too onerous of a commitment.) Maybe that means maintaining the roadmap, reporting back on the key things on the roadmap, agreeing to keep tabs on the most important things on the roadmap, etc.

I’m happy to go with any alternatives, but I think the current status quo isn’t as effective as we could make it. Even small changes like having a separate channel other than the developers category would help. Thoughts?

1 Like
9

I was thinking more about the responsibilities aspect of this, here’s what I have:

Executive director: this role should probably be responsible for communicating operational issues (bring up issues like, e.g., it’s hard to run workshops b/c installation is messy, or we’re about to have a conference so let’s not break rstan, etc… ) That does affect the roadmap in a vague sort of way.

TWGC: I agree about overseeing the roadmap. In addition I think the role should include finding enough common ground with the technical leads s.t. projects move forward. Additionally, for the parts of the roadmap that the technical leads aren’t interested in pursuing the TWGC role could include identifying contributors who could implement it. I realize anybody could do that but some contributors (e.g.-Ben Bales and Matthijs come to mind) have at times been very flexible about what they work on but they’ve run into roadblocks that wouldn’t have been so serious if they were working with a more explicit roadmap. I realize that’s all cat herding but it’s one of the harder parts of overseeing long-term development so it might as well be explicit.

re: reviewers:

I think this is something like a code of conduct situation. Like you’re not responsible for making the code work when you review but you are responsible for providing clear feedback (don’t waste dev time) and checking against the roadmap or bringing in the technical lead if it’s unclear (don’t waste dev time). … I know we all want to assume people will do their best but we should be specific about what we view as important.