I think I agree with what @syclik has been saying and I disagree with
what @wds15 has been saying.
Writing, discussing, and finalizing a design doc can be useful even if
the design doc does not get adopted. For example, I can easily imagine
scenarios where it would be very useful to have someone propose a big
change and have the proposal refined via discussion — even if the
proposal ends up not being adopted.
I think this is what the IETF RFCs do. So perhaps they could be our
model instead of the Rust RFCs or the Python PEPs.
This is the only part I worry about. In rust that’s a committee, maybe an easy solution here is for each repo to have tagged “know it alls” who have some level of ownership to make sure the right people are aware of a design doc. imo I don’t think these folks need to formally vote on a design doc, but be aware that it exists
There’s no formal voting structure so the opposite happens, no one accepts it! I think the informal rules right now are that no code associated with a design doc is accepted till the design doc is accepted. Adding a formal rule for this then requires answering when code needs a design doc.
This is actually what I like about the design docs, there are no answers to these questions. We rely on good social mores. One thing I also want to make clear, I’m not sure you are assuming this or not, but I think the current process actually makes it harder to get in a design doc and then implement it.
That is what we do right now. There is a lot of very good discussions going on. It’s just that when a design doc gets accepted it should be done along these lines. Though there can always be refinement…as it happened for reduce sum wrt to arguments order, for example.
I like the current process and it’s evolving which is all great. We never had anything before, so this is big progress.
Should we get to larger conflicts we would need moderation processes, but we will figure it out when needed.
What does “accepted” mean in this case? Who does the accepting? Is it
consensus? Consensus among who?
I have two alternatives to the consensus rule:
Something gets “published” (i.e., merged into the repo) if at least
one other (non-author) Stan developer endorses the design doc (and the
original authors declare it finished).
I can also imagine having an appointed editor who decides when a
design doc is finished.
In my view accepted means anyone that cared enough on a topic to review/comment/request changes agrees with the change. I have not seen anyone rushing anyone to respond. Quite the opposite. Enough time must be given, I agree there.
I also think no one would object if someone came posted “Give me a week to process this. Or two.”. We are all reasonable enough to tag anyone that may have missed it.
One of my design doc PR (JSON output) was closed because the discussion showed that its a bad idea. So I would count that as rejected design doc.
And I am glad I closed it, as it may have been a slight improvement to the status quo but I would now spend my days trying to find which package supports which encoding in each language. And if someone comes in 2 years and says: “Hey why dont you have JSON output” we can point them to that discussion.
Sebastian also had 2 attempts for the parallel stuff that eventually led to reduce_sum which is awesome and I think the users like it also. So that is a huge win for design docs and whatever we call the process we have.
we should update the template to include things we want to see (@bbbales2)
and let’s not make it so prescriptive that it sounds like it’s a bunch of rules (@mitzimorris)
I’m just lost… I doesn’t seem like it’s working because things don’t get merged in? I also don’t know how things are supposed to go from start to finish. Maybe it’s working ok, but even writing that down would make it a lot better and give a chance for non-developers to know how to participate here.
Just on the code front, open-source is a little tricky since it’s so distributed. One of the ways to make open-source development sustainable is to have the invariant on every merge into the centralized repository be “working code” (that’s in quotes because it’s really a best effort thing). “Move fast and break things” doesn’t quite work because if the code doesn’t work after a merge, this essentially makes it harder for everyone that’s collaborating because they have to check whether they’ve broken it or it’s someone else. And if this continues and there are a few changes that are broken then it pushes the problem down to the community.
It’s not that we want a crazy meta process… we just want the community of developers, whether it’s a couple, a dozen, or even a few hundred, to be able to pick up at any point, build on top of it, without every single dev needing to check every time they merge that the central repository is “working.”
I like the format for the current design docs and the discussions have been very productive. We’ve leaned toward inclusiveness and toward letting the person who writes the spec drive the process. I think that’s a good way to lean for open-source projects. As much as I’d like to see some things done differently, we can only do them one way and I can’t do everything.
For all of the designs up there, I don’t think there’s much of an issue with whether we want the general feature. Of course we want a sparse matrix type. So it’s really just a matter of finding a platform to collaboratively figure out how to do it.
From my perspective, it has been ad hoc (though the Rust format really helps with that). But it hasn’t felt unfair.
It hasn’t been slow because of lack of process, it’s been slow because everyone’s busy and there’s a lot to digest and incorporate in these bigger designs. Also, Stan’s a big-ish project and design’s hard.
That discussion seemed to be going productively. Was there a speciic issue there? I thought we’d broadly decided to move to a mono-repo ages ago and just failed to ever execute.
I think the real problem is that it’s unlikely we’re going to find solutions to everything that makes everyone happy.
At some point, we’ll want to update beyond C++11. Some devs will favor backward compatibility over new features and we’ll have to work out a compromise. We started thinking about the move to C++11 years before we did it and had the same issue, but I think that went off OK.
Is there disagreement as to wanting to have static (what I’d call “immutable”) matrices or was it about specifics of the design?
The assumption with all of Stan is that everyone’s welcome to jump into the discussion.
Thanks. Most of the time it’s fine, but there have been times when it’s felt unfair to me. Of course, fairness isn’t necessarily the metric to optimize here. I think I’m just expressing that it’d be good to reflect in the documentation what we want and intend.
No, no particular issue now. But thinking about the monorepo, the question is how we can be inclusive and at the same time make a decision. This particular one will affect all developers in one way or another. I’d rather have a procedure where we can move forward rather than being stuck with what we have due to conflict of opinions.
No problem with the topic. Following that thread, it went from having a language component to not. Just trying to think if there could have been some guidance on what to provide that would have prevented that detour. Or… maybe there should be certain things that are limited to repos?
That isn’t what’s written in the README.md, or at least how I interpret it.
Write a design document roughly following the template. Designs that do not present convincing motivation, demonstrate understanding of the impact of the design, or are disingenuous about the drawbacks or alternatives tend to be poorly received. Share it with the Stan core developers, advertising it on discourse and ideally the weekly meeting. Especially share it with one or more developers with design approval privileges. Designs rarely go through this process unchanged, especially as alternatives and drawbacks are illuminated. Edit the existing Google doc and allow it to preserve the history. Submit the design to a reviewer and work with them until it is accepted or postponed.
Notice our doc states that it’s to be shared with developers that have “design approval privileges.” Also, earlier in the document, it says “one or more developers turning these ideas into a design proposal” and “one proposal [introduced] by the relevant managing stakeholder.” When reading that, it’s not clear that everyone is invited to participate.
everything on stan-dev is open to discussion from everyone, and
everyone is welcome to become a developer.
I also agree we need some kind of approval process. Right now, we’re just assuming we’ll keep discussing until we get a consensus. [Edit: I mean de facto that’s how it’s working—clearly it’s not documented that way. I don’t think anyone’s felt empowered to just override everyone else and approve a PR like this.] At least for the language stuff, it’s working well in my opinion. We don’t really have an approval process for anything at the moment, so hopefully we can get one in place before the next decisionmaking crisis arises.
hi all, I finally was able to update the README! It’s in a PR:
I’ve tried to take the relevant parts of this conversation while leaving as much of the original document intact. Please review, comment, edit as you see fit.
Just a heads up, I read through the IETF RFC (the posted link and the one on IETF) and then read through the IETF Internet-Drafts and thought that was closer to what we were discussing. Mind taking a look and seeing if that is indeed the right interpretation of your comment? If I’m mistaken, please let me know.
I-Ds are draft versions of things that (potentially) become RFCs, I think.
I’m not sure the analogy is precise though, the I-Ds are published but
there is often a series of them, one published I-D replacing the
previous version. I’m not sure that’s what we intend.
@ariddell, thanks! That’s pretty spot on with how I read it. Specifically, the analogy isn’t precise but it is a little closer to what we had in mind. I also don’t think we’re intending for a series of design docs meant to replace the previous ones, but at the same time, I don’t think we’re intentionally blocking that from happening.
I think there’s an open question about what we want the process to be and what we understand an accepted design doc to mean. Reading the thread above, it sounds like most of the devs here are looking for something a bit fluid and using this repository to talk about designs and iterate to come up with a solution. And that an accepted design document could be changed.
It seems different from the purpose of Rust RFCs. Once accepted as a Rust RFC, the stakeholders have, in principle, accepted the major change. The procedures lay it out so that all of the relevant stakeholders not only are contacted, they explicitly have to sign off for it to be accepted. For someone to propose an RFC, there’s a list of stakeholders to contact, the proposal has to be advertised in certain venues, and there’s a path from a proposed RFC -> discussion by the relevant sub-team(s) -> motion for final comments (with who is allowed to bring this forward, where this motion needs to be made, how long it must last at a minimum… and a condition that " all members of the subteam must sign off") -> which concludes with acceptance of the motion (either accept, reject, or postpone) if the final comment period is quiet. There are some really nice properties of that doc which includes wording that an accepted RFC is not a “rubber stamp,” but the process they’ve asked for makes it such that everyone in the subteam must sign off on the RFC, so not sure how in practice it’s not. Anyway, I’m just pointing out that this doesn’t seem like it’s what we want.
(Looking at Python PEP also doesn’t look too close.)