TWG Definition/Roles. Call for feedback/ideas/refinement

General
1

All,
A big bit of unfinished SGB (Stan Governing Body) business is the definition and creation of the Technical Working Group (TWG).

This is a SGB decision, not a referendum, but we want your input. Roughly we need to define the domains for the TWG leads and appoint them. The role of the TWG director is also up for redefinition. I will articulate this as specific text which will sound like decisions have been made but that is not so.

Changes are much easier to incorporate if they are concrete, e.g., specific text. Don’t let that stop you from commenting however, some ideas are not at that level.

This draft contains new changes not vetted with the SGB, particularly the creation of the CTO position.

Also note that downstream packages like RStanArm, BRMS, ShinyStan…are not included. Please discuss.

Proposal under discussion:

" The Technical Working Group (TWG) exists to help harmonize Stan development across varied domains. Historically the TWG domain leads have been gatekeepers of the domains of Algorithms, Language, Math, Services, CmdStan, RStan and PyStan. There is also a TWG director with final decision making authority, ability to override leads (rarely) and that position is paid $50,000/yr.

In recognition of Stan’s continued growth and reliance on community contributions we redefine the TWG as follows:

The TWG’s domains, membership and mission are defined by the SGB (Stan
Governing Body) in consultation with the Stan community by seeking at a minimum feedback on the forums at https://discourse.mc-stan.org when changes are made to the structure of the TWG. Individuals can self nominate when TWG domain lead/director positions become available and the positions will be advertised on discourse.

Domain leads and the director are primarily facilitators, community builders and stewards for their domains with the skills to consider cross domain issues. Domain leads are in service to their developers.

The leads/director can act as decision makers when needed but that should be rarely applied. In greater detail:

  1. The biggest responsibility is keeping PRs (pull requests which are software changes waiting for review) moving in a timely manner. This means approving PRs, requesting changes to the PRs, marking working in progress on PRs or closing ideally within one week of submission. Delegation is highly encouraged.

  2. Domain leads manage people working in the domain which means recruiting developers, helping less experienced contributors make useful PRs and facilitating code quality/testing criteria.

  3. Leads attend meetings, report at least twice a month at the general Stan meeting. Can be submitted in writing.

  4. Review or delegate review of design documents.

The current set of domains are, with time bursty time estimates:

  • Math. 15 hrs/month
  • Inference (was algorithms). 10 hrs/month
  • Interfaces (includes RStan, PyStan, CmdStan, CmdStanPy, CmdStanR, Services) 15 hrs/month
  • Compiler (includes language, StanC3 compiler) 15 hrs/month

The TWG director has proven to be a difficult job so the job will be split into a more standard CTO position and refining what the TWG director position means in the hopes of smoother development operations.

The CTO interfaces with the non-developer stakeholders and ultimately represents their interests. The CTO is responsible for managing high-level strategic issues like what kind of technology to use, where to make investments, and long term visions. They also serve as the highest court in the land, arbiter and tie-breaker for technical issue within the project. This is an unpaid position.

The TWG director exists to facilitate the overall smooth operation of the Stan development organization. They report to the SGB, CTO, control budgets and hire contractors. They are encouraged to have open meetings, ensure that all voices are heard and grow the developer community. Given that much of this position is administrative, the TWG director is a paid position, $50,000/year. The TWG director’s focus is to facilitate alignment across the tech leads. In greater detail:

  1. Organizing and facilitating developer meetings and/or hackathons.
  2. Figure out what each domain agrees is implementable for the next year
    and create a roadmap from that, hopefully facilitating lots of cross-team
    communication such that the roadmap is fairly aligned towards common goals
    that come from the SGB and CTO.
  3. Centralize requests for help from across the different domains such that
    when new contributors come to the project, they can turn to the director to
    be triaged to the right domain or issue.
  4. Hire and oversee contractors to deal with maintenance work
  5. Maintaining status/membership of developers in the community and
    providing a clear path to advancement for contributors.
  6. The TWG director needs to be able to end discussions and make decisions when the project seems deadlocked in consultation with the CTO. Any decision is better than no decision, as they say.

"

1 Like
2

I like the direction this is going in. I’d say possibly the CTO should be the clear last-stop on the arbitration train - here it looks like either one could be it due to the last item on the last list.

3

What happens when a lead fails to do this?

Broadly unsure about the CTO job. It doesn’t appear to have made the TWG lead smaller and it’s unclear if it has any power. Can you flesh it out? Also not sure who’d be able to do it. Also not sure “what sort of technology to use” belongs in that domain.

4

Why is the TWG job so hard?

I think the TWG director has proven to be a difficult job because it’s the wrong way to run an open-source project. I’m largely responsible for the current direction, with help from @seantalts. We both thought having a management structure more like the good companies we worked at would be more efficient in decision making. It’s hard to express how much more efficient a well-managed industry software project is than any decision making I’ve ever seen in open source or academia. I really miss that structure as my managers at SpeechWorks were way better than I was at deciding what I should work on.

While it would be nice to have efficient top-down decision making, I didn’t at all anticipate the hard feelings this would create in the community. I stepped down because I didn’t see any way I could do the job effectively. That’s partly me in that I have a hard time making decisions without understanding all the details (a critical failure for a manager), partly that I felt a conflict of interest with @andrewgelman as my boss and @mitzimorris as my partner, and partly that I thought any kind of decision I’d make would lead to the same kind of hard feelings the current roadmap has engendered.

How about the Apache way?

I’d like to see us move to an Apache-like consensus process with votes to break deadlocks.

Apache does have something like tech leads for their gazillion projects, but those seem to be mostly reporting and process roles, not decision-making roles.

I particularly want to adopt the Apache process of making all decisions online in the open rather than with subsets of people in face-to-face or online meetings. I’m not saying we shouldn’t have subsets of devs meet to discuss design, etc., just that they shouldn’t come to binding resolutions that are then presented as a consensus without first running things by the community online.

The Apache proces is similar to the “rough consensus” model used by the Internet Engineering Task Force (IETF).

The reason I’m apprehensive about the rough consensus approach is that it still leaves a manager setting the threshold for consensus. A simple majoirty vote seem easier to manage.

A nice example of how this works is the way @breckbaldwin managed the Apache 2 licensing discussion (not related to Apache process). That felt open and inclusive, yet it came to a conclusion relatively quickly.

CTO?

Like @anon75146577, I don’t understand the purpose of the CTO role or how it’s differentiated from the TWG role. I don’t think adding more C-level titles is a step in the right direction for an open-source project.

In companies I’ve been associated with that are beyond startup size, CTO is more of a sales role. The decision making chain was clearly CEO > COO > VP of engineering; the CTO was not in the chain.

In or out?

I had originally cordoned off the downstream interfaces like rstanarm and brms because I didn’t feel like I understood them well enough and because they seemed to require much less coordination because they are endpoints rather than infrastructure, so changes to them don’t have knock-on effects. I also felt I had no hope of influencing rstan or pystan given how many battles I’d lost in the past on API design.

But that was just my failure. Going forward, I would very much like to see all of the projects branded as Stan brought under the same process and governed in the same way.

If the decision is to not bring them under the Stan community process, then I would like to see them taken off of our web pages and out of the stan-dev repo. Then they would have to apply to the SGB if they wanted to use the Stan name or logo in their branding.

Who do you pay?

Why was the TWG director singled out for extra compensation? Why not the executive director (which seems like an even bigger role), or the board members or tech leads? I think singling out people to pay is going to lead to resentment among all the others acting on a volunteer basis.

It’s problematic for an open-source project to directly pay people for open-source development when most developers are contributing time on a volunteer basis. I’m not getting paid by the Stan project to work on Stan. Andrew and I raised the grant funding to pay me and the rest of the Stan team at Columbia. Same with @avehtari and @Erik_Strumbelj and @ericnovik, all of whom employ multiple Stan devs with funding they and their teams have raised.

If the project is going to pay the TWG director, I believe it should also compensate the executive director, the board members, an the tech leads for projects. These all seem like more important roles to the project than creating a YouTube channel or managing Discourse posts. I’m not saying not to pay such roles, just that if you’re going to pay some of those roles, it would seem consistent to pay the other ones.

Also, it seems like we are going to need a way forward with StanCon now that @andrewgelman wants someone other than his employees at Columbia to organize it.

What tech leads?

The commitment for math and the interfaces is way more than 15 hours/month. t is bursty around releases and initial development. I think managing stanc3, rstan, and the math lib tech leads are as big a job as the TWG director role as currently defined. Similarly, CmdStanPy and PyStan3 are a lot of work as they’re being developed. Specifically, I think they’re much more work than managing Discourse or a YouTube channel.

I don’t think there’s much to do for algorithms as we are basically just telling people to go away until they’ve proven their algorithm works; here’s a quote from the Wiki page.

With so many algorithms introduced each week the Stan development team unfortunately does not have the resources to assist with this testing, and consequently the evidence burden is the responsibility of anyone who might want their method to be considered.

I’m not sure what to do about the C++ services. It’s a tiny amount of code bridging the algrorithms and interfaces, but as such it has an outsized impact on interface development compared to any of our other C++ decisions.

Some history

As I recall, the main motivation for the TWG position was to resolve issues that we could not resolve by consenus. This seriously blocked the progress of Stan around the Stan 2 refactor and led to lot of hard feelings which I feel are still negatively affecting the project many years later.

When I briefly had the TWG director role, someone asked me to do a “roadmap” talk at the Helsinki StanCon. The way I conceived that, it was just a big feature to-do list that everyone roughly agreed on. I didn’t see it as making any binding decisions.

6 Likes
5

My understanding from the meeting where this was decided was that the TWG lead was expected to have to take responsibility for some unpopular decisions and was basically being paid to take the flak. Think danger pay, but where the danger is snarky emails from men in their 30s who shouldn’t send snarky emails. I don’t think this is what played out and I hope the SGB is reconsidering things as they refactor the post.

The existence of a tech lead hasn’t stopped the blocks. My understanding is that there is still a major one in math.

I think the project leadership is sending wildly inconsistent messages on this and it’s getting us in trouble. In particular @breckbaldwin’s unwillingness to close off any doors has extended the HINTS saga beyond its natural limit. This should be a choice for the TWG equivalent and the SGB and it’s a choice they have to make. Right now we’re just annoying people.

A period of focus on limited algorithms would make sense given that everything else is being “blown up and put back together” right now.

2 Likes
6

First off thanks for all the thought that has gone into this thread. This is exactly what I was hoping for. I’ll start at the top.

The CTO position was an attempt by me and someone else, who may out themselves if they choose, to get Bob back into the decision making process. It has failed and I remove it.

They need to be replaced by the SGB if PRs don’t flow. We have multiple instances of developers getting frustrated by the tech leads and this has to stop. We are very dependent on external to Columbia contributions, thanks folks!, and throwing up barriers is not good. Structures that make sense with 4 folks in an office banging out Stan 1.0 changes when we have 20+ contributors.

We tried something and it is not working out in certain respects. I’d like to see a more community building role from the TWG director which is borrowed from James Vasile’s advice (OpenTech Strategies). I don’t want to toss the single point of responsibility for the technical side yet. I think it almost worked this time.

Happy to discuss further.

Having been through a few controversies in the past year I’d say the trigger points for controversy are:

1 Lack of transparency
2. Lack of awareness that decisions are being made
3. Lack of being heard.

Addressing each in turn I’d say that 1) transparency has increased if you know where to look which is what 2) is about. I think we need a weekly summation of what is going on, what needs action in Stan and it should happen at the weekly meeting Thursday. 3) is what really drives the frustration, we do need to hear from everyone and make sure they know that.

I did decide to try something different with the Apache licensing conversation as Bob notes:

Thanks for identifying something that is working. A very hard part of this job is all the complaining about how ideas don’t work. You are a tough group to work with.

Back to the process, all I did was restate what @seantalts was driving for, I just crisped it up. It was not quick in my estimate, but it is the model I’d like to pursue. I came up with a tripwire version that if anyone objected to adoption we would go to majority vote (Apache 2 library inclusion into Stan Math--general policy case. Trip wire consensus decision). Rough consensus sounds good too.

The decision making process I’d like to see is;

  1. Announce upcoming decision at general Stan meeting.
  2. Discuss on discourse for a week.
  3. Talk about at next weeks Stan meeting
  4. Discuss for a few more days, kick to vote/tripwire/rough consensus/sgb depending on the issue
  5. Done in two weeks ideally

I am trying this with the TWG issue we are discussing now. I’d consider last week’s meeting to be 1), doing 2) now and 3) tomorrow (Stan General Meeting Thursday September 26, 2019 11am EDT). The final decision is actually the SGB’s to make, see Your new governing body.

This is very much my personal opinion but Stan’s responsibilities need to simplify. The downstream packages like RStanArm, BRMS and my favorite ShinyStan are bigger than Stan org. They are welcome in our repo however.

I think this is a big decision that we need to think hard about. I’ll remind all that both @andrewgelman and @avehtari are very focused on inference research while at the same time being huge Stan supporters with research groups that build Stan. We have other research groups sniffing around looking for a happy place to do inference research.

I believe that at my first Stan meeting, essentially my interview, that I actively wondered why there is not a plugin architecture for inference research, gradient based and otherwise? Not doing so given the psychological energy telling people to go away seems odd. I’ll leave this for future discussion.

Paying the TWG director was my idea. On any version it is a crap job. I’d like to remind all that economics matter.

So why pay people?

  • Rich people can volunteer, poor people cannot. If you want diversity, pay people.
  • These jobs are “work” as in not done for fun.
  • The jobs are ludicrously underpaid ??$10k for a youtube manager??, more like a “tip”, for expected value. Eventually we start paying people for the value their jobs bring.
  • We have the money, this is not a struggle for Stan and we can experiment with what works.

This is a great discussion and what we can use. I hope I have been respectful of opinions I agree and disagree with.

Breck

1 Like
7

tl;dr the following post has some factual corrections littered with a couple of additional concerns to think about. I think the big takeaway is to please read at least this open source governance survey.. My most important objection is that we need a (better?) mechanism to make forward progress than majority voting.

There are many, many open source governance models, the most successful of which have had something like a BDFL position. You opted out of all governance very early on in the SGB and TWG formation which must be very frustrating now, but we’ve done a ton of research and discussion on open source governance. Here is the bibliography of sources I read when thinking carefully and discussing the original TWG director position internally to the SGB:

I particularly recommend the Python project’s open source governance survey - they summarize quite a few governance models.

Let me take this opportunity to reiterate that the tech lead meeting I held used the IETF rough consensus technique (I am pretty sure I linked it to you last week, albeit a blog post about it and not the wikipedia page), so I mentioned that we came to rough consensus as a group on what was in that document. That doesn’t mean the roadmap was set in stone - in fact, it’s still up for discussion, though I’m hoping we’re getting close to something we can mostly all agree on. I think I could have done a better job communicating that distinction, but given that you are still characterizing that meeting and my future intentions incorrectly after our conversations about this, it seems like that additional communication didn’t matter very much.

Anyway, I really want to make sure the Stan project can actually make decisions because I want to ensure our forward progress and not just sit back on our laurels to slowly maintain code until no one is using it anymore. I’d need to see your proposed ad-hoc modifications to the Apache system fleshed out more to really think about whether it could solve deadlocks. But either way, I think an executive is attractive because that is the branch typically associated with actually executing. Usually votes are used within things like the legislative and judicial branches, which are designed to act as a conservational check and balance against the executive, to slow it down. See also design by committee.

I aimed to create the same “feature to-do list that everyone roughly agreed on,” but by actually getting those people in a room to discuss what should go on it and what we were actually going to work on . My hope there is that we’d have significantly more alignment and buy-in towards what we actually agreed upon. I’m not sure in what sense you think either roadmap is or was “binding?”

Paying the position was Breck’s idea, not mine, but I have to say I wouldn’t have taken the job without it - it’s a particularly shitty version of a normally fairly shitty job. I think we should pay positions that are 1) important to do well and 2) unlikely to receive volunteers that will be capable of doing them well (both in terms of technical and managerial skill as well as time commitment). So perhaps what I would try to do this time is 1) redesign the structure and then 2) have the SGB ask for applications. If they seem suboptimal and there are folks who would be better for the position, ask them how much it would take for them to take it and consider just paying that amount. (I’m not going to apply for any new paid positions).

That was me. I don’t actually think we should get rid of CTO the idea, but we can definitely change the name and make it clear that this position exists to break ties or deadlocks. And instead of “making major technology choices and technical decisions” we can just say that they will design or approve internal Stan APIs and, together with the TWGD, decide what the TWGD’s contractors work on. This would mean working together to co-write up essentially a list of tasks or focus areas with priorities attached. This is the most concrete part of technical vision setting and is perhaps enough in a project like this.

Bob, does that sound better?

8

After sleeping on it, majority voting is growing on me as a mechanism for tie-breaking and breaking deadlocks. But there are a lot of details missing that I think are important:

  1. Who votes? (related: who can call a vote?)
  2. When can votes be called?
  3. Who writes the text of the vote? I particularly want to avoid situations I’ve seen before in another org with incredibly leading problem statements and questions, etc.
  4. Who enforces the vote? What happens if someone ignores the result of the vote?
  5. What are the new roles for TWG members like?

I think calling votes over the internet is a great way to make enemies where they didn’t exist before, as opposed to getting folks in the same room often to talk things out. I’d like to keep in the proposal a probably-paid role for some kind of dev-organizer / project manager / something like that who would be responsible for organizing a couple of face to face meetings a year for TWG members (as well as 3 other items from that list above: recording the roadmap summarizing releases for users, hiring & managing contractors, triaging new devs)

Bob, I also want to just run a thought experiment past you. Let’s say we vote and one of the first issues is this one:

And let’s say the vote goes down - the majority of folks do not want to force them to accept governance nor do they want to kick them out. Will you accept that the decision has been made and work to support it? I think your opinion of the current system changed deeply when you realized that I was listening to the majority of stakeholders on a couple of issues that you cared about and had unpopular views on. If you are not okay with that, I would suggest taking the leadership role we all want you to have, and both the power and responsibility that comes with it.

9

This is very confrontational, bordering on a personal attack, which does not seem appropriate. And hypothetical misses the other options: A hypothetical Bob could choose to work on the part of the project that he does not disagree with, or he could choose not to continue contributing to the project, or he could try to fork the project.

10

I’m the main offender here, but I’m a little older than that.

Nor has the existence of a TWG director. Before, the problem was two different opinions among subsets of developers and nobody empowered to make a decision. The bottleneck now is satisfying the tech leads that PRs are safe. Ultimate responsibility lies with the TWG director, as I believe they have the authority to override the module leads.

This is why I think this process is wrong compared to one based on voting, where this would’ve probably already been decided in favor of inclusion.

I believe the right approach is to be friendly and welcoming to everyone, while firmly insisting on our evaluation criteria. I don’t mind the top-level message of the algorithms procedure, just the get-off-my-lawn tone. I’m not sure how to handle the case where someone wants a lot of feedback during meetings, but that’s a separate issue to be handled by whoever is moderating (not ever very clear).

I don’t want to be singled out. I want everyone (who wants to be) to be included in the process on big issues. I’m OK with having authority for decisions rest in the SGB or TWG, but as I’ve said, I think voting will work more efficiently with fewer hard feelings all around.

This is a tricky issue balancing the need to be relatively conservative in our low-level modules. Adding threading (which is the currently blocked issue) is a huge change, so I think it makes sense to take it slowly.

Yet I think the tech leads if they’re going to function as such need to have the power to block PRs they think are unsafe. The problem I see is that nobody really respects their authority, which I think is endemic to a project of open-source volunteers.

The tech leads for math and algorithms are outside of Columbia, so I don’t think that’s the issue.

I think that’s spot on.

I very much disagree about this as it’s only ever a very small set of stakeholders at the meeting. I don’t think the current math lead or algorithms lead have been at a Thursday meeting in years. Do you want to insist they come to regular meetings or give up their roles?

That’s “we”—you’re part of the group.

Why have them in our repo if they’re not official Stan projects? Where do we draw the line? Can anyone who’s a stan dev start dropping pet projects into the Stan repo and branding them as “Stan”?

It’s not that I don’t like any of these projects. It’s that I want to see a consistent system enforced. As I said for these projects, I’d like to see them brought into the Stan fold rather than kicked out.

I don’t know what you mean by “plugin architecture”, but I don’t know how we can make it any simpler than exposing log densities and gradients in C++, R, and Python. All of our evaluation tools and posterior analysis tools are open.

One big exception here is students, who are typically poor by most measures other than future potential, but often have the energy, free time, and motivation to volunteer. The volunteering can be synergistic with their other work. That’s also true for a lot of academics, who I wouldn’t classify as either rich or poor.

Breck and I get a distorted view of income from hanging out with highly skilled NYC and Silicon Valley tech programmers and managers. Even in the countries with rich middle classes like the U.S., the median family income is US$60K, which is very conservatively less than half the compensation of a junior C++ programmer at Google in NYC. In many countries, US$10K would represent a large fraction of a middle-class programmer or academic’s salary.

Another problem we face is that we need a high level of competence from Stan contributors in most cases. The kind of competence with which, barring extenuating circumstances, would land a relatively high-paying job.

One opportunity for increasing diversity would be to give something like scholarships to train people in the relevant stats and programming skills so they can start contributing to Stan. That would be super expensive compared to the likely yield for the project, but it might be worth trying. I’d be game to volunteer for something like an intensive summer school (not a boot camp—if it’s called that, I’m out).

Right—I didn’t want to be on the SGB and I opted out of the TWG for reasons I tried to articulate above.

Thanks for the links. They’ll take a while to digest. But why not make that list public along with a summary rather than internal?

I’m still having trouble reconciling these two statements. My understanding is that you considered the meeting you had of a few tech leads a quorum from which to reach rough consensus. As such, you didn’t want further feedback on things you considered already decided.

If that’s not the case, then yes, I misunderstood all the stuff you sent me, told me, and posted in now delisted dev-only Discourse posts.

Isn’t that the difficult and unpopular part of the current TWG director position? What’s left for the TWG director to do?

I’m happy to help do that without having a title or extra pay.

I absolutely do not want to be the one making final decisions on things. As I said above, I don’t think I have the proper mindset to do it and I feel I have too many conflicts of interest.

I’d just like the opportunity to make my position known without having to attend a lot of meetings, which typically leave me so frustrated I have trouble sleeping, often for days (e.g., the first couple hours of the recent roadmap discussion and pretty much any weekly Stan meeting).

So when Breck asked if I wanted every decision run by me, I said I’d like them run not only by me, but by the entire community for feedback. As an example, I keep citing the Apache 2 decision, as that’s the only big decision lately that felt open to me.

If joining the SGB the only way of influencing higher-level project decisions, I’ll probably just bow out of the decision making and stop trying to ask clarifying questions about decisions I don’t understand or seem really wrongheaded to me.

Doesn’t your proposed procedure of rough consensus involve voting over the internet or excluding most of our devs from most decisions?

Presumably whoever is involved in the rough consensus. That is, whoever’s a developer or in the community, or however that comes out. It seems like we have to do this any way. Personally, I’d like to see members of each subproject voting on that subproject and everyone voting on the project-wide decisions. Now I’d like that to just be devs, who are going to actually do the work and will be more informed about the tech used, but I can understand the desire to make it more open to

That’s tricky. And it’s going to be a problem with anything we do to decide when we’ve collected enough community input.

I think the trigger for a vote should be when we get to an impasse where we can’t make progress without a decision. That’d be tricky for things like unifying argument names across interfaces, as that’s not blocking any particular person’s progress, it’s just suboptimal for the project as a whole

I’d have been happy to write up an alternative proposal to stanfit objects (I have done it several times before) and the current one provides the model, so I think that’d be relatively easy. I assume I’d lose that vote, which I’m OK with. The only time I comment on it is when it comes around for what feels like voting again as it feels like it just did.

The Apache approach is to have a chair of each project that manages the procedure. Presumably they’d be doing it. In practice, that’d mean a tech lead wouldn’t accept a PR or would back it out if it didn’t comply.

I think we have the same problem now with people ignoring the roadmap, though I’m not clear what one could do that would violate the current roadmap. Is RStan in violation as soon as it doesn’t change its argument names to the new ones?

That seems to imply there are existing roles, which I don’t know about. I think the responsibility for executing here is on the tech leads with backup from the project director.

That’s the current status, and while I don’t like it, I’m not actively fighting against it. What I would say if you asked me before the vote went down is that if there’s a third category besides governed-by-Stan and not-endoresed-by-Stan, then it should be better defined than just saying what two things it’s not. Such as what the rough criteria are for becoming or remaining such a project.

I’d almost certainly leave any real work supporting decisions I don’t like to other people. So I wouldn’t be keen to build stanfit objects in R or Python (not that anyone would be asking me to), nor would I want to work on the services or CmdStan argument structures if they don’t get refactored the way I think is sensible.

I’m OK with the question. I think we pretty much have to assume everyone’s going to go along with whatever our governance is. I keep trying to emphasize that I don’t expect special treatment.

The real Bob is going to choose option one unless things get a lot worse than they’ve ever gotten before.

But I think the question was more about whether I’d continue to argue against decisions that had already been made by the accepted process. While I don’t want to obstruct people expressing opinions, I feel that’d be very counterproductive for the project as a whole and would certainly discourage it.

11

For inference research specifically, what Bob mentions, the plugin architecture gradient based and otherwise is available in R and Python (and in C++, but minority of inference research is done directly in C++). My group is using this “plugin” interface. There are open examples how to use it, we have helped people on discourse, but it doesn’t seem to be popular. Using the current log density plus gradient interface is easy. and my group is in progress making more examples and case studies for this.

For implementing an inference algorithm in C++, I think the C++ code is modular enough for adding other algorithms, although there is additional work to enable those in interfaces, ie there is no plug-ang-play-feature that you could put you algorithm module somewhere and interfaces would figure out how to show help and check arguments, etc.

What seems to be forgotten sometimes is that, these two above “plugin” approaches assume that algorithm needs just the log density or the log density plus gradients. However, there have been people asking support for algorithms which would require, for example, mix of log densities and log probabilities (sampling for mix of continuous and discrete parameters), Markov blankets (alternating or nested inference for different parameter groups), or streaming data (some of sequential Monte Carlo methods). All these would be huge changes requiring a lot of work. Even if someone would offer to do most of the work, the changes would have effects all over Stan. Making Stan to allow plug-in algorithms for these would be a very big project.

See also Modularity for making Stan more Pythonic and Rthonic - #30 by bgoodri and Algorithm Inclusion Policy

4 Likes
12

Strong agree with everything that Aki says here. The language and model concept have been designed to expose the target density and the gradient of that density to support algorithms that operation on the target density as a whole. The problem with supporting general algorithm development is that this encapsulation isn’t sufficient and would require redesigning the language and the model concept; it’s not just a matter of changing how objects are exposed in the interfaces.

Anything that falls into the “user specifies a joint density and algorithms create output from that joint density” UX will require comprehensive changes to the core code.

13

First I want to say to Bob - thanks for engaging in this in a pretty level-headed and unsnarky way. And thanks for giving me the benefit of the doubt in some of what I said and assuming I was engaging in good faith - glad you didn’t take offense.

I’m open minded to this being the right way the more that I read about it, but I’m still not very sure what you specifically have in mind and I think there are a lot of potential pit falls we should think about at least. Can you lead us through a possible scenario here for a recent issue, maybe taking the parallelism RFC or the TLS AD change as an example?

It’s not the speed that’s the issue, it’s the lack of real goalposts or engagement from the reviewer. I think it’s just too big a chunk of work for someone with not enough large chunks of time to really engage on it. Plus there’s at least the appearance of a conflict of interest in that Generable was secretly working on their own private parallelism at the same time (see Daniel’s StanCon talk).

We can work on these things - it doesn’t necessarily mean the model is flawed.

I’m pretty sympathetic to this particular issue and stance. Let’s use this as an example for how I think such a change would need to be made. I think you need two things here: you need a leader to begin the work of listening to all the folks involved to curate a good minimal set of policies is for all stan-dev projects, and you need a vote to ratify that set of policies. You just brought up adding a chair position as Apache has - I think this is crucial. I guess the delta between a chair and a director is that the director is ultimately making the decisions, and a chair is putting forward all decisions to be ratified? The switch to the latter seems to make a lot of sense to me.

I think your voting / apache proposal still has some holes in it from your presentation of it - places where you’ve sort of explained that issues are tricky but haven’t provided concrete proposals to solve them. For example, who exactly should vote (at least decide between tech leads, all devs, or all electorate), who can trigger a vote and how often, and how to deal with things that don’t cause an impasse but are good to get cohesion on.

❤️

I wanted new ideas or arguments, but also for folks to know that most of the stakeholders had already signed off on the stuff that was present. Basically setting a somewhat high bar for serious arguments on especially bike-sheddy topics like the names, similar to rough consensus - looking mostly for concerns that the listed proposals won’t work for some reason.

I think there’s a lot of dev organizing / project planning / management-y stuff to do that is not technical in nature. Google separates tech leads from managers, for example. I realize we’re not truly managing people but I think there could possibly be some separation there. I listed some of the stuff before: organizing and facilitating dev meetings and/or hackathons, collating a roadmap, centralizing requests for help and triaging new contributors, hiring and managing contractors.

I actually do think we should insist on regular video meetings with most tech leads for the sake of cohesion and alignment. Perhaps the existing meeting is not the right schedule or format but I will say it’s gotten a lot better after Breck and Andrew started cracking the whip on the status update part. I think we can improve it further or start a new meeting, but that face time is important for a project.

I think that was probably just oversight. But also we figured the people who were seriously concerned about governance were already in the room.

I read something since I asked you that original question that has affected my view on this a bit. I wish the solutions applied more to our situation, but I’ll post anyway because I think there’s a ton of interesting thoughts about organizations in here: What do executives do, anyway? - apenwarr.

14

Also, don’t lead developers on. Be clear. If the lead won’t ever accept the central tennent of the patch that makes it work, they should not be telling the developers that the tennent “makes sense”. Central to that is that a lead should be exceptional at coding (e.g. C++) in order to understand patches. It’s a sure sign that things have gone wrong from a technical understanding viewpoint when a lead tries and then fails to re-implement their own version of a PR after having stymied a contributing dev’s working and tests-passing version of it.

I suggest that leads be subject to periodic elections by the devs at defined intervals, and possibly term limits as well. The aim is to alleviate the SGB from the unpleasant task of firing leads, and introduce more accountability into the PR process.

2 Likes
15

Just my personal experience, but I think what you are saying here actually flows into what Bob and Breck are saying above. This week I’m going to try to work on something for Stan math that set’s expectations better for developers given the amount/complexity of code they want to contribute. I think it may also be nice to have a “sample template” for a reviewer so they can have a form that asks a few Qs to make sure they have asked all the regular questions for a PR.

I’d actually flip this around, we have some pretty smart cookies leading and working on our libraries, but no one can know everything. C and C++ in particular let people do very weird and clever things. If a contribution has those weird and/or clever things then I think the onus is on the contributor to make sure everyone is on the same page about what’s happening. I like the weird and esoteric parts of C++, but it can be weird and gross! More than once, explaining how I was going about something in detail gave a much better perspective on tradeoffs of the clever version of the code or a much nicer way to go about the same thing.

Keep in mind leads are maintainers who devote their free time to maintaining Stan and I think it’s pretty reasonable to assume positive intent from them. But that limited free time is a two way street that frustrates developers. I think clear contributor guidelines would relieve a lot of pressure from both sides. EOD, relationships are a two way street! Transparency, awareness, listening, and assuming positive intent should come on both sides!

16

I think it’s pretty much what you mean by “rough consensus” but with an agreement threshold determined in advance rather than issue-by-issue. I think that’s pretty much the Apache process.

Exactly. Rather than tech leads, we’d have chairs. That change in terminology reflects the role as being more of a moderator than leader.

Nevertheless, I think the roles will require a lot of leadership in the sense of being proactive, gathering opinions, consolidating and presenting it, etc… And as @seantalts pointed out earlier, there’s a lot of power concentrated simply in deciding when to call a vote.

I think the decision to include the Apache license went this way. We collected issues, got answers from a lawyer, discussed it online, got some more help from a lawyer, then when everyone had pretty much said what they were going to say we asked if anyone objected and went ahead with it. If someone had objected, we’d have taken a vote at that point.

For the sake of argument, let’s say whoever’s involved in your rough consensus or whoever the SGB legislates is in the upcoming (?) vote. I don’t see the set of voters here being any different.

My own preference is to have committers for projects vote on things with in projects and all committers vote on project-wide decisions. Again, like Apache does.

That leaves open who’s a committer on a project. @seantalts has suggested the hurdle be a “substantial PR”, which is what most projects do, what we’ve been doing informally, and what I’d like to do going forward.

I’m really hoping there won’t need to be many votes. I think a bunch of the stuff around interfaces is exactly where we need to coordinate globally across interfaces and thus there’s a lot of disagreement.

Please bring specific complaints up with the board. I don’t think the forums are a good place to discuss this topic other than in general terms about who’s responsible for it (and I think everyone’s going to agree that has to be the board).

What schedule do you imagine? Have you asked the current tech leads?

I think that’s the critical wrong assumption in all this. I really don’t have the patience to stay in that room, but I’m seriously concerned about governance. I realize that not being in the room will give me less of say, and in fact that amount of say may be nothing depending on how decision-making gets legislated.

+1 to that. And thanks for getting the ball rolling.

17

Thanks, all, for keeping the discussion moving. At this point, I feel very much like Bob feels on so many of his points, especially the part that’s quoted by everyone above (lack of…).

@ChrisChiasson, I’m truly sorry about how this ended up. I really appreciate the effort and time you put into it. (There’s more complexity here than is presented, but that’s ok. I’ll chalk it up as my L.)

+1. I don’t know about term limits just because it is a volunteer thing and we may not have volunteers available.

I think part of this is directed towards me. I’ll address this as a reviewer (not as a tech lead).

As a reviewer on Math, I’m asked to determine if a patch meets the standards of the Math library, the code is documented, the code is tested, and the code is idiomatic C++. I walk through the code and mark down where that’s failed. Often times, the PRs that I am asked to review are not easy. In particular, the ones that have been slow have side-effects that were not documented, tests that were missing, or non-idiomatic C++. So at that point, I ask for a revision.

Unfortunately, a lot of these PRs have gone through that process multiple times. When there are large side-effects, as a reviewer, I ask the Math dev community to weigh in.

If that seems like it’s moving goalposts, I’m not communicating clearly enough. I can pinpoint the last two PRs this happened where I spent over a hundred hours helping @wds15. An alternative to what I’m doing, I could just reject them and not provide additional assistance, but I don’t find that so useful. When I say testing or side-effects, I mean things like: a PR is written going against what is the current policy in order to gain speed. That seems like that’s the main effect and it should be validated prior to the merge. Or… one PR may increase the speed under one condition by 10% but slow down another condition by 10%. It’s not clear as a reviewer whether I’m supposed to accept that PR, so we discuss it.

+1. I don’t think the existing meeting is the right forum for the TWG.

No. @seantalts hasn’t recognized tech leads. I think he’s been waiting on the SGB to do this. Is that right? And at some point, I didn’t think @seantalts believed that the TWG had a purpose, although that’s changed with this thread.

@seantalts, after reading all of this, it seems like you have an idea now on what a TWG would be. You still have the power to create the TWG. Would you like to take steps towards that now or would you rather wait on it? One concrete weakness I see in the proposal is that it’s focused on tie-breaking rather than encouraging consesus-seeking. I don’t know what decisions ultimately need to be forced past a 50/50 vote. Or even something close. I think it should be on the group to understand the pros and cons and come to an understanding that it’s better for the development of the team and so the group is on-board with the decision.

Thanks for posting this. I feel like a lot of these concepts apply and should be considered.

+1. I’m worn down from reviewing a lot of PRs that are super-complicated, need a lot of attention, and limit my time to work on other things that I could be working on.

I’d love to have a more concrete policy that I can point to in order to accept quickly or reject quickly.

18

I’m not opposed but it’s interesting that you were at one point TWG Chair and had pretty strong opinions that it needed to become a Director position with actual power when you resigned. Do you think we’re oscillating back to exactly that situation? Is it a sort of “well, we tried both, and a chairship is the lesser of two evils” kinda thing?

Yeah, I liked how this went. I don’t think the Director should be unilaterally deciding anything close to major issues but it’s good for forward progress that minor things can be resolved quickly and reverted later if need be.

This is also where I perceive the biggest need for someone to essentially be the villain and make tough decisions (that they then get ratified). Which is the plan for the current roadmap doc, so I guess that it’s been more of a chair-style tenure so far.

I haven’t started work on this yet, no. I’d be happy with as infrequently as once a month.

re: TWG - I still read the original post as giving TWG-creation authority to the SGB, which is one reason I haven’t just done it myself. I’m pretty happy with this thread in terms of getting community engagement on a plan. @syclik is it your opinion that that phase is basically over and someone just needs to enact it? If so I’d propose we run it through the SGB officially.

I also would like to see like a concrete spec for what @Bob_Carpenter is proposing - “rough consensus” is not specific enough for my very concrete mind. Would you say that the diff between your ideas and the original post in this thread would be getting rid of “CTO” and renaming TWG Director back to TWG Chair (and thus showing that all of its decisions must be ratified by the TWG)?

19

That is an accurate report of my opinion at the time. Since then, I have changed my mind, flip-flopped, reversed course, and now hold the opposite opinion. Specifically, I think we should move to the tech lead role being a “chair” in the Apache sense and viewed as a facilitator for consensus and as a last resort, someone to call for a vote if things can’t be decided by consensus.

The idea is absolutely to seek consensus, and only when that fails seek a vote. And I don’t mean someone saying “I would do it differently,” but more “I am willing to put the work in to do it differently and here’s my counterproposal.” There’s a lot written about this in all that material floating around on open-source decision making.

To take one example, when I objected to the way the stanfit objects are designed and presented a counterproposal, I was not willing to put in the work on PyStan and RStan to code them the way I want. So I have to let that go and let other people build them in ways that to me seem to violate fundamental principles of software design in languages like C++ or Java. Nevertheless, I wanted the chance to express my objection rather than facing a fait accompli.

How can there be a chair if there’s no committee?

I also think this whole plan is complicated with an entire roadmap at varying levels of granularity going up. I’d much rather see this legislated in small, independent pieces. I have to vote on it all or nothing as is, which puts me in a difficult bind if I want to vote my actual opinion and I want the project to not be stalled.

This was the thing that annoyed me the most about being “TWG director”—that there was no notion of what the group was supposed to be. I didn’t want to be running a large committee.

Sure. Could you send me two things so I can get a sense of the level of detail required:

  1. the rules for how the TWG and its director currently operate—the only thing I’ve seen is this post by Breck.

  2. Your proposal for how things should work going forward.

So what did you mean by “rough consensus” when you used it in the roadmap doc (my emphasis)?

Anyway, let me be clear now and say that I do not want to sit in a room and hum and let the chair play decibel meter and decide if we have rough consensus. I would strongly prefer a fixed threshold (1/2 or 2/3 seem to be reasonable thresholds).

20

I think this would take even longer than the current approach! I guess if you think there are pieces of it that wouldn’t pass majority vote alone and want to pull those out into separate votes I’d be okay with that, just let me know which pieces.

That’s the only thing I know about too. I don’t actually have a proposal here, but Breck has one as the top post. I’m not saying we should use that but it has some detail around how the tech leads get their jobs and what their job description is and who is responsible to whom for what ( in some cases).

I’d be willing to sit down with you and make a proposal or just take one that you write to the SGB but I may ask questions about it and make suggestions first :P

Sure, no humming required. I like the way you expressed it above, which is that if you have an objection you have to also have a counter-proposal and be willing to take over the effort to take over the project to achieve its goals. That could end up being a pretty high bar though, and I’ve definitely seen people say they would take over and do XYZ instead but then fail to implement it, as has @ChrisChiasson. But I guess it can always get sent to a vote if it’s still contentious.

1 Like