There has recently been some confusion regarding how external algorithms are considered for inclusion into Stan. Over the past few years the core algorithm developers have adopted consistent criteria for consideration that we have individually shared with those who have proposed new algorithms and methods. Unfortunately these criteria were not made publicly available and consequently not everyone was aware of those criteria. This ultimately led to varying expectations and disagreements.
In order to avoid such confusion in the future, I have formalized a policy as the current Algorithm Tech Lead, with input from the active developers who have most closely worked with new algorithms and methodologies. The policy is available on the Stan Wiki, https://github.com/stan-dev/stan/wiki/Proposing-Algorithms-for-Inclusion-Into-Stan, and I encourage you to send anyone inquiring about new methods there so that they are aware of what will be required of any new method.
Thanks.
9 Likes
I like a lot of this, but I think it would be nice to have a TL;DR somewhere which seems to be
- Get your algorithm working with the stat_comp_benchmarks (which should probably have it’s own doc)
- Make a design proposal that also discusses the conceptual questions
- If those are both approved then you are good to go for a new algorithm in Stan
The verbage tends to come off a bit aggressive. I think we should try to be helpful as possible. When I’m good at something I like helping others be good at it since
- That usually makes me better
- I like crushing my opponents when they are at their best :-)
Our algorithms are quite good, if we can make it easy for people to benchmark against it then all the better
4 Likes
Good idea. I added one to the doc.
I don’t disagree with the sentiment but you have to understand just how often we are presented with new algorithms, whether in person, online, or via the literature. Most of these presentations replicate existing work and many of them are hampered by their authors misunderstanding of Bayesian computation.
I think that this demonstrates a strong difference between stats and computer science: in stats people often don’t know what they’re computing, which obfuscates fair comparisons and benchmarks. That means before even addressing the algorithm itself one has to address all of the stats first, and that’s only if the presenting party is even interested in having that conversation.
Anyways, I tried to make it clear that people are welcome to start conversations on Discourse while also making it clear that we cannot be responsible for evaluating incomplete proposals. I know it can sound harsh but in my experience anything less explicit ends up encouraging too many people to believe that they can get their algorithm into Stan. Really the goal is to set clear expectations so that people aren’t let astray about what is required.
That was the intent of the stat_comp_benchmark suite, of which no one has yet taken advantage, and Aki’s bayesbench project.