Hey all, can someone point me to our process for the design docs?
I honestly don’t know what things are supposed to be in there, who is supposed to “approve” them, what it means when it’s been approved, etc. The repo has been there a while, but the processes for it were never set / discussed here.
There is a short “The process” section. But it’s lacking the direction that would be useful for this to be actionable. Are we striving for an approval by consensus? What’s the timing of these things… should we have a focused meeting once a quarter (release cycle) to approve ready proposals? Who is “we”?
There are 11 open PRs there and I’m not sure who has the authority to either approve or close PRs. (Enough people have permissions.)
larger issues that need more detailed explanation thats a bit much for a repo issue (OpenCL kernel generator)
proposals for changes to the language (ragged, interpolation)
new algorithms or new interfaces (laplace, cmdstanpy)
I also see it as a good place to hash out ideas, get “consensus” on larger things that will affect anyone (for example: latest Stan CI PR). Discourse topics quickly vanish and it’s then hard to find them even after a week. Much less a month or a year.
Not sure on who can approve, but the way I see it the final approval will always come as approval to specific PRs on separate repositories. This can be used to set up the plan before you dig too deep.
Yes this. When something is a pull request, it’s never clear to me as a reviewer if I should just be checking correctness/C++ery, or weighing in on the design itself.
Design docs is a place to talk about alternatives/design inputs/etc. And like @rok_cesnovar said to have conversations that span repos.
I guess one question I’ve had is to what extent do we need to be maintaining the docs in design docs? It’s certainly a process to me, but once the implementation is in, I would go to the implementation/implementation docs before the design docs for specifics.
Interesting thought makes sense, of course. Just implies the burden to keep it up to date, but it’s probably worth it. If so, then we should maybe start to link source code with respective design doc? The reduce_sum doxygen tags should then probably include a link to the design doc?
I like the idea, I just wonder if that creates quickly a lot of outdated links, etc.
I don’t see a big reason for this. Once it’s documented and implemented elsewhere, that’s fine. We’re not doing the C++ thing where we write a specification and everyone goes and implements it – there’s just one implementation. Ofc. the idea was to be copying Rust. I dunno what they do with their design-docs long term.
We did have some pretty big mistakes in the Laplace approximation design docs. So that was handy getting those corrected.
I can see a case for distinguishing between “finished/published” and “draft” documents. I’m not sure it’s essential to designate a document as being approved or not.
One middle ground might be to have reviewers who endorse a finished design document attach their name to it.
I think another goal of the design docs is to prevent the misunderstandings and hard feelings that arise when someone goes the the time and effort of implementing a feature and submits a PR and is told no, or told to change everything and do it differently.
the process and documentation were borrowed from the Rust project - either Sean agreed with all of it, or else he found it expedient to borrow someone else’s well-written policy docs. either way, a good decision.
Thanks all for writing. It’s confirmed my own confusion. Based on the responses here, it looks like we don’t have the same expectation for what PRs mean on the repo.
Could I suggest we move away from modeling it after Rust’s RFC and maybe documenting it closer to what we believe? It seems like it’s more useful for us to align cross-repo thoughts or even things that need more technical detail. But it’s not really meant for Request For Comments for a proposal that will make a lasting change on Stan (in the way it’s designed for Rust).
Just thinking through the use case… suppose we wanted to do something drastic in Stan. Hypothetically, let’s say we wanted to stop supporting computers and only supporting phones. This is ridiculous, but if it was a technically sound proposal, who rejects it? If it’s rejected and there’s a PR that implements it, does it automatically get rejected? Alternatively, what if it’s accepted? Who is allowed to accept it? Does that acceptance mean Stan is allowed to change in that direction? Hearing the comments above, it sounds like none of this is what this is intended to be. So… it’d help if we were clearer that our expectation is that these are ideas that we want to collaborate on. And we should have some lighter way of approving these and merging into the repo.
Could we call it something like “brainstorm” or “draft designs”? Would that help? (Naming is hard.)
In my understanding the design doc repo is far more binding than “brainstorm” or “draft”.
I mean, everything what is approved in the design doc repo should really go into the code. Possibly scattered over multiple repos.
So if something comes to a PR and it is implemented along the lines of the design doc, then it should be accepted (design wise); though there is always the option to come to the conclusion that the design needs revision given details found during implementation.
The concept is build around consensus, but to me it’s still a lot more than mere “brainstorm”.
(I don’t have a good overview of open-source processes I have to say)
Thanks, @wds15. If that’s the understanding, then we (the community) should agree on the scope and we should write out the process of approval for openness, transparency, setting the proper expectations for everyone involved.
So far, it didn’t seem like that. I think the design doc repo needs a design doc.
Thanks, @mitzi. Yes, agree that it’s better to have something discussed in a design and approve / rejected at that point before a PR. But this just pushes the issue away from the PR. Now, if a design doc was approved, a PR could still be rejected for the same reasons. It just doesn’t seem like it binds. (Even without explicit rules, I don’t think we’re all treating the approved design docs as how the language will or is allowed to change.)
It’d actually be nice if we had a proposal mechanism where we approved large changes that happen to the ecosystem. This is something we’re lacking. It’d be great if we could gather broad consensus using this mechanism.
Should we (all of us) collaboratively draft something that would enable this?
Lol, possibly. Iterating makes sense. Design docs have found a purpose through use, and maybe that is different than the intended purpose (I’m not sure what that was now).
Also I’m happy for someone to use it differently than however we document it. Indeed I hope the process evolves without us passing any decrees to evolve it. If we document the process in a way that makes it less intimidating to use, great.
That’s true, but I don’t think we can ever do hard-and-fast rules cause that means we need to take the threat of phone-Stan seriously.
I guess, where does this question come from? Just reviewing repos? Or do you have a phone-Stan idea :D?
Yes, it could still be rejected. But the chances of that happening become slim or slimmer if all involved parties are on the same page in the design phase.
Now if someone agrees with a design on the design docs but then starts disagreeing with the design once an actual PR is open on a repo, then we have a problem (I am not talking objecting implementation things, that is fair).
I completely agree! It’s not that it’s not useful. It’s more that without an agreement on what it is and what the process is, it’ll be ad hoc and feel unfair. And since it’s slow moving, anyone new without a lot of tenacity will bounce.
Yup. We just need a process that allows for process change. And for us to keep it in synch with our current use.
It’s something that I was wondering early on, but didn’t feel comfortable raising concerns at the time. These thoughts came up recently with the mono repo design doc. And also thinking about dropping C++11 / moving to a new toolchain. And static matrices. And reviews from PRs on Math where it’d help to have a design doc. And looking at double digit open PRs on that repo that don’t have movement, don’t have a path towards acceptance or rejection.
And… damn right I want to run on a phone. :)
EDIT: (Steve) added newline for quote to show up nicely
Thanks, @rok_cesnovar. The one assumption here is that this includes enough of the involved parties. That’s where I think we need to be clear about the expectations from the parties that need to be involved. And we also need to keep the queue managed so the involved parties know they’re supposed to go there, check it, accept / reject / postpone it. If not, we’re going to end up with more and more open ones and it’ll be hard for us to track what’s even active.
Makes sense. I assume the place to start would be thinking about the template and comparing what’s in the template to what gets written up (and comparing it to what is written here).
And then editing the readme around what it looks like we’re currently doing.
tl;dr: I think current process is working and evolving, and that’s good enough.
I’m always looking for a good compromise between “move fast and break things” and the level of process required to engineer manned spacecraft. the problem with trying to come up meta documentation on process is that every situation is just that much different from the intended ideal, thus leading to ever more meta discussions about the process and how should it be applied in this case. in short, all rules lawyering, all the time.
I am, however, appreciative of the question of “is everybody who should be participating in the loop?” given that we are a decentralized, co-operative, open-source organization where all participation is voluntary (i.e. it’s always possible to just fork the repo and go your own way), I think it’s on participants to participate.