Some time ago, we discussed what to put in the warning Stan issues about divergent transitions at Text for warning message
Since then, we have a public-facing, but not really used link http://discourse.mc-stan.org/divergences that currently redirects to Divergent transitions - a primer but we can change the target at any time, should we have a better treatment of the topic. I think having an “official” link in the warning message is useful and I believe the current target is a good overview of resources one can use to address divergences.
So the question is:
- Do we want an “official” link for divergent transitions (and possible treedepth/BFMI)?
- If so, is the current #howto a good target? Should something change about it? Or should we point to a completely different resource?
Tagging the participants from the previous discussion: @andrewgelman, @bgoodri, @betanalpha, @bbbales2, @jonah, @mitzimorris
I very much like the idea of having a link to a webpage rather than trying to cram advice into a warning message.
As I said in the previous discussion the current “how to” is problematic and misleading. Investigating and resolving to diagnostics is an open ended process that cannot be reduced to a simple checklist no matter how convenient that would be, and trying to force that convenience only misleads users unnecessarily. That said given the unproductive history of the conversations on this particular topic I will not be commenting further and just hope that whatever goes up respects the users enough to warn them of its incomplete nature.
I don’t think I understand what exactly you consider unproductive. I tried to implement many of your suggestions and tried to give reasons for not following others in my response Divergent transitions - a primer - #6 by martinmodrak . You didn’t follow-up on my response (which you have every right to do), but I honestly don’t know how to make a conversation much more productive than that. What was the reason you decided to end the conversation at that point?
In any case I believe that among all possible combinations of words we can display to a user who tries to understand divergences there is a hypothetical global optimum that would help the most users make a useful next step in their analysis and/or understanding of Bayesian methods. I also think it is clear that either showing nothing or just “read a statistics textbook and then come back” we are far from this optimum. Similarly we would IMHO clearly be far from the optimum if the resource became a statistics textbook of its own. There are also bounds on how much time and energy can be invested in writing. So I tried to strike a middle ground. It is quite possible that I failed at it. If you have any suggestions how to move towards a text that would be closer to such global optimum (including replacing the text with something completely different), I’d be glad to hear them and will try incorporate them (or, you can directly edit the post yourself).
The howto says in the introduction (this was included in August 2020 as a part of response towards your feedback):
We should note that resolving modelling issues is generally hard and requires some understanding of probabilistic theory and Hamiltonian Monte-Carlo, see Understanding basics of Bayesian statistics and modelling for more general resources.
What follows is a list of brief hints that could help you diagnose the source of degeneracies in your model - or at least let you get faster help here on forums. Where they exist, we link to additional resources for deeper understanding. The aim is to provide a birds-eye view of approaches we’ve had success with in the past, point you to additional resources and give you keywords to search for :-) This is not, and can’t be a definitive guide - each degenerate posterior is problematic in its own way and there is no single approach that would always work.
If you fail to diagnose/resolve the problem yourself or if you have trouble understanding or applying some of the hints, don’t worry, you are welcome to ask here on Discourse, we’ll try to help!
Do you believe we should give even more warnings of the incomplete nature of the resource?
I do hope @betanalpha reconsiders and responds. I am fascinated by the fact that divergences actually get to the epistemology of Bayesian models. What does it mean for a model to be “good” or give “good” results?
In a sense, what Michael says,
and, more importantly, the first paragraph of Identity Crisis (betanalpha.github.io) is the natural preface that a user should confront on this epistemological quest.