Questions for candidates for Stan Governing Body

General
1

As the elections for the next Stan Governing Body come closer, the community likely has specific questions to ask the candidates (who can be found in the nomination Discourse thread). We encourage anyone from the community to ask questions, voice concerns, or make suggestions in this thread so we can gather them in one place.

Some suggestions to keep this thread productive (these are definitely not binding):

  • Remember that SGB candidates can come from varying backgrounds. When asking specific questions about development, applications, education, etc., try to provide some context that would let candidates without strong background in the topic understand the gist of the question.
  • I kindly ask the candidates to delay posting replies for a while, so that the discussion does not run too far before most nominations are announced and people who have questions have had time to voice them. I will contact all candidates and link them to this thread once I will think it is suitable to move the discussion forward, which will be on November 3rd at the latest.
  • If you post a question in this thread please wait before most/all candidates reply to their questions before posting followup questions to avoid fracturing the thread. If any topic would seem to require longer discussion, it should be moved to a new thread.

If you disagree with the process around asking questions as outlined above, please discuss it in the Election thread and keep this thread for the actual questions.

6 Likes
3

There is a thread, that is just one of many, where people attempt to work through the problem of stalled PRs. I’m one of the commenters in the thread, but I’ll just quote other people here.

Governing means actually making decisions and taking responsibility. Therefore, I’d like to know how** you’re going to fix this.

**Note, we all understand that the SGB is not a singular person. Answers saying you’ll listen to everyone respectfully and attempt to achieve consensus are, IMO, just stating the obvious minimum requirements. During the election, I’m assuming voters are interested in the predispositions of the candidates, because the desired composition of the board can then be more effectively set by the electorate based on those identified predispositions.

4 Likes
4

Just to add context to @ChrisChiasson’s question as some people have been confused by the question (and please correct me, if I am wrong):

  • The current governing structure for Stan development is described at Your new governing body some further refinement can be found at Sean Talts is Technical Working Group Director
  • The core point of contention AFAIK is how the Technical working group (TWG) should operate. The specific example @ChrisChiasson is talking about is what happens when someone implements a new feature in a pull request (PR), the PR is not merged and the author of the PR thinks the decision, or the process surrounding the decision is unfair or takes too long. Current state is that merge decisions are made by the lead for the given subproject/domain at their discretion and there is no clear method of appeal. Please correct me, if I am misrepresenting this.
  • In a still broader context, discussion has been held whether the TWG/leads should be more “directors” that make decisions and get things done or whether TWG should act like “chairs” that facilitate discussion but don’t make a lot of decisions themselves. The current system is closer to the “director” role of leads and AFAIK the discussion has not provided a clear consensus what is the ideal state.

Hope this makes things clearer.

6 Likes
5

Thanks @martinmodrak. Actually, while @seantalts’s position was officially ratified by the SGB, the other “leads” were never officially approved by the SGB. So we’ve been operating using informal tech leads based on what made sense due to people’s past contributions in certain areas. How to formalize the individual tech lead positions (or whether to take a different approach) is something that the next SGB will need to decide (ideally in collaboration with developers and anyone interested who has relevant experience/ideas). And given the confusion around this, clearer communication from the SGB is needed on these matters.

To get to the particulars of @ChrisChiasson’s question: regardless of how the next SGB proposes to organize the TWG (or a replacement for the TWG idea), I think we definitely do need some sort of formal appeals procedure. I’m not sure exactly what that procedure should be (and I’d also like to take a look at how some other projects handle this), but here a few things I’m thinking about:

  • We could have an official mechanism for requesting a change in reviewer if a PR submitter feels that they are being treated unfairly by a particular reviewer.
  • Regardless of how the tech “lead” issue is handled by the next SGB, if there is any sort of leadership position for particular tech areas then there should be a transparent process for changing who holds that leadership position based on whether they have or have not fulfilled their responsibilities. The responsibilities quoted above:

seem reasonable and these (and any other responsibilities) should be laid out transparently if these leadership positions are created, so that anyone who has a complaint can point to the particular responsibilities they think are not being fulfilled.

My predisposition is that if there is someone who wants to contribute to Stan but feels they are being treated unfairly then that is a serious problem for the project. I don’t claim to have a perfect solution to this in mind, but I don’t take it lightly at all.

Totally fair, although I really would like to hear concrete suggestions from you or anyone else who has felt unfairly treated by the PR process in the past!

6 Likes
6

I think that one solution would be to have some sort of quick review with more than just the lead doing the review. ropensci seems to have a successful approach to it:

Even though it’s for reviewing R packages, a similar approach could be used: specially the use of reviewer volunteers and clear documentation of what is expected (in a gitbook). But notice that the scope of what the reviewers are doing is much larger, and the time as well (3 months). Another thing that might be driving the conflict is that maybe “approving PRs, requesting changes to the PRs, marking working in progress on PRs or closing ideally within one week of submission” is too optimistic and leads to pressure to reject complicated PRs when there is not enough time. But I’m just speculating here without knowing anyone who was unfairly (or fairly) treated by the PR process.

Disclaimer: I’ve never submitted anything to Ropensci (but I’m planning to), so I’m just assuming that it might work better than current Stan’s approach.

4 Likes
7

great suggestion Bruno!

I met Noam Ross of rOpenSci last night at PyData NYC and had a great conversation.

rOpenSci’s focus is the individual researcher in a lab who’s good enough at R to write their own R package, but has no exposure to software engineering and the principles of good design, efficient coding, and unit and end-to-end testing. rOpenSci wants to improve both the packages available to the community and to make the package authors into better programmers. rOpenSci has a network of about 100 reviewers and it sounds like a fantastic organization providing a much needed service.

the Stan C++ code base is fairly mature, and the team has tried to put in place good coding and testing practices. like the R community, we face the problem that new contributors are talented and enthusiastic hackers and we need to turn them into good software engineers. I could go on, but don’t want to further hijack this discussion.

8 Likes
8

Thank you for your question. My view is that PR difficulties are a manifestation of issues around whose priorities take precedent, who gets to contribute, and where the language is headed. These are normal growing pains for a successful open-source project, but we do need to address them more directly.

Here is my assessment of the landscape right now and how we got to where we are. Stan has outgrown the restrictive identity that served it well as it navigated its early days. With limited bandwidth, the Stan team focused on what it thought it could accomplish better than anyone else. The quality of the software developers and the statistical sophistication of its sampling algorithm (and statistical advisors) set it apart and defined its niche. Compared to its early competitors (e.g., JAGS), Stan was much more efficient and provided reliable estimation across a wider array of models. As a result, the language has a reputation for excellence. If you perform your diagnostic checks and Stan does not raise any flags for you, you can feel more confident about your inference than you could with any other Bayesian approach out there. That excellence is an achievement worth preserving.

What this model does not facilitate is a broad, diverse base of contributors or organic, user-directed feature prioritization, both of which are vital to the continued success of the project. We now have many more people (and organizations) willing to devote resources to the growth of Stan, but we are not using this to full advantage.

In my view, Stan needs:

  • A community-approved vision that encourages more contribution from outsiders
  • Key personnel to commit to both the vision and the idea that the community sets it

I propose some guardrails, which the SGB will need to make more concrete, with natural next questions in italics:

  • Little to no effort from the core team is spent on algorithms that have not yet had important theoretical properties established in peer reviewed statistical literature (What counts as an important property? How much effort is too much effort?)
  • Rejection of pull requests which compromise the integrity of sampler performance (i.e., no substantial performance regressions for models identified as “core”) (What are the core models? What constitutes a substantial performance regression? How do we communicate to users which features might be considered “in beta” and/or have less theoretical grounding?)

Within these guardrails, I lean towards a more liberal approach to PR acceptance.

Stan would expect potential contributors to have:

  • Early and repeated engagement with Stan core team
  • Submissions which include:
    • Standardized technical documentation, including descriptions of scope and analysis of side effects
    • Sufficient context to explain why a feature is needed and how it contributes to the community-approved vision
  • Grace and understanding that the Stan team are either volunteers or effectively volunteers (in that they could earn much more elsewhere)

In return, contributors should be able to expect from Stan:

  • Consistent communication regarding documentation requirements for a PR to be accepted, including examples of successful PRs ranging in size from small changes to larger feature expansions
  • A transparent review process, potentially including appeals
  • Initial responses on PRs within a reasonable period of time
  • Respectful, considered explanations if the PR is rejected due to lack of alignment with the agreed-upon vision for Stan

In summary, my feeling is that we need to step back, re-orient around our shared goals, and establish processes that are more welcoming of contributions from outsiders.

7 Likes
9
  • In what way do you plan to contribute to fundraising in support of Stan?
  • “SGB is to advance diversity”, and the way I see it includes the diversity of applications. Considering the nominees are dominantly in bio/pharma fields, which under-represented application area(s) do you see potential for increased adoption of Stan/Bayesian methods? how do you plan to address it?
7 Likes
10

I’m not going to pretend to know all of the issues that contributors are facing with the PR process, so it would be good to be able to have a more formal way to source all of this information. With that I’m sure the SGB will be able to make a more informed decision about how to deal with project lead issues. At a high level it seems like,

  1. the project leads need to have a clearly defined role and scope
  2. there should be a process in place that allows contributors to appeal to have a project lead removed.

My predisposition is towards transparency. As an on-again-off-again contributor it’s difficult to keep up with changes in the code base, who the leads are for the various projects, and what the process is for contributing. It would be good to have a clearly documented process for contributing to projects and a way for developers to easily keep up-to-date with changes. My hope is that if these issues are resolved then not only does the project become more welcoming to new developers, but it will also resolve some of the issues that existing developers are facing. As the projects stand, it’s quite intimidating for new developers to join the project, despite the fact that they have the skills and time commitment to contribute.

4 Likes
11

One of this issues I’ve had with Stan it that it’s been so bio/pharma focused. That’s not a bad thing and I definitely don’t want to take away from Stan’s presence in that industry. However, Stan’s lack of presence in a diverse range of industries hurts its adoption. In my opinion two industries that we could target are marketing and entertainment (the industry that I work in).

I’ve gone out of my way to try to get people to adopt Stan in other industries (by teaching or producing industry-specific case-studies). Here are some of the issues that I’ve come across,

  1. Stan is not marketed in a way that makes it easy to onboard new users who know nothing about Stan.

    • This could be solved by improving the user experience of our website. In particular, including a quick start guide and/or a “wow factor” example that most software have (e.g. Keras). It sounds cheesy, but it does work.

  2. There is limited discoverability of how Stan can be applied to various industries.

    • This could be improved by extending the presence of Stan at not only PPL specific conference (e.g. ProbProg) but also conferences are attended by a majority of industry data scientists (e.g. R Conference, Sloan Sport Analytics Conference).
    • This could also be solved by improving the discoverability of our case studies and encouraging users to contribute industry-specific case studies.

  3. There is a barrier to entry requiring people to learn Stan in addition to the language they’re interfacing with.

    • The interface agnostic feature of Stan is incredible but most data scientists see it as a shortcoming since they have to learn Stan. It would be good to publicize why being interface agnostic is useful. A lot of teams in industry are split between languages (e.g. R/python) and will find Stan to be a useful interface point between data scientists. (This is particularly useful for teams that like to develop in R but have to productionalize in python.)
    • This issue can also be solved by expanding the functionality of our existing high-level interfaces (e.g. rstanarm) and developing high-level interfaces for popular languages (e.g. the python version of rstanarm, possibly by emulating the scikit-learn python ML package).

In my opinion there needs to be a huge push towards using Stan in marketing. Particularly, inference done through A/B testing is much more interpretable to the business when it’s done through Stan rather than through statistical significance testing. Broadly speaking, people in marketing are not fully aware of this, and Bayesian statistics in marketing is still niche rather than the industry standard.

I can’t speak much to the fundraising aspect, but I hope that if Stan permeates into a diverse set of industries then those industries will be willing to provide support to the project. I know that if I was in the right position then I would give back to the Stan project (if not financially then by providing support to employees to contribute to the project).

7 Likes
12

1. In what way do you plan to contribute to fundraising in support of Stan?

As I mentioned in my nomination, I currently work at a Foundation and everyday I am exposed to different business plans and I advice organizations about how to improve the quality of their pitches for different potential funders. More concretely, there are three items I am interested to work on:

a) “Business-like” document that can easily introduce the Stan project to funders and tailor those pitches depending on the strategic interest of these funders. For instance, many ventures and philanthropic institutions set strategic goals for the medium to long-term and understanding how we can seek funding for Stan will require identifying overlaps between funders and the Stan community. That said, something I explicitly want to avoid is seeking funding that comes attached with conditions that are unfair for the Stan team and/or the Stan community in general (i.e. prioritize a feature just because funding was provided).

b) Identifying legal structures that facilitate funding. Often, for complex legal reasons, funding can be provided in the form of gifts, donations, grants or whatnot. Each of these items has its own legal complexity level and something that often prevents funding from happening is the excessive amount of steps between when an offer for funds is made and when the money arrives. This is something I am fairly familiar with and I’d like to identify what are the most efficient avenues for funding depending on the funders legal overview/requirements.

2. Which under-represented application area(s) do you see potential for increased adoption of Stan/Bayesian methods? how do you plan to address it?

I completely understand your concern about the predominance of the current candidates mostly in bio/pharma. Speaking for myself, I am so glad to see Stan being used in these fields (bio in particular) because many existing tools (or research methods) in these sectors are not particularly aiming for excellence given safety is at stake. That said, this might create a misconception that Stan is mostly a software for particular fields and as a consequence we may have less contributions from other fields, which is in general a loss for the project as a whole.

I wholeheartedly agree with Imad on marketing and entertainment as low-hanging fruits for several reasons:

  1. As a person already working in marketing put it to me the other day “I suspect the developers may be mis-calibrated in the distribution of skills among users”. The fact is that Stan is already widely used in the marketing community and I know this because I worked in marketing too.
  2. There are already several professors teaching marketing who incorporate Stan in their curriculum because the ease of interpretation for A/B testing for students.
  3. The other area where Stan is also used is in entertainment, more specifically in media and content distribution. I used to work at BuzzFeed and during my time there I met several folks in different media and entertainment businesses already using (or wanting to use) Stan in their products. To my knowledge, there are at least three to four of the largest media companies in the US with staff already using Stan for some of their internal analysis.

However, I don’t want to limit to these two areas but given my exposure to marketing and media, I see these two industries already have the buy-in which is the harder part and it’s just a matter of being more inclusive of their needs, which I think Imad accurately summarize regarding the challenges about interfaces.

A few points I’d like to add that can potentially help:

  1. Survey a few folks in these industries and understand what are the challenges they have faced in terms of infrastructure to have Stan in production code and assess whether that is something that can already be solvable or if existing work is addressing those challenges.
  2. Talk to a couple of instructors already using Stan who can be a reference person for people considering using Stan in those fields and who perhaps are willing to summarize the needs of their communities.
  3. Often, these communities might not be entirely comfortable attending Stan conferences initially because little time is devoted to use cases in their fields. Instead they would prefer to have other options (i.e. Stan sessions in conferences where most of their industry already attends) that might have the seal of approval/support of the Stan team/community.
8 Likes