# Automagically increase \`adapt\_delta\` until all divergences are eliminated, what could go wrong?

**URL:** <https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601>\
**Category:** Modeling\
**Created:** [May 20, 2021, 7:13am UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601 "2021-05-20T07:13:10Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![Funko\_Unko](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/funko_unko/32/10786_2.png) [@Funko\_Unko](https://discourse.mc-stan.org/u/Funko_Unko)\
**Post date:** [May 20, 2021, 7:13am UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/1 "2021-05-20T07:13:10Z")

</div>

So, I thought my above question was stupid, but then I found this:

> [@Adapt\_delta](https://discourse.mc-stan.org/t/adapt-delta/7947/1):
>
> If our first recommendation is to increase adapt\_delta, why not do this automatically? As a user, I’d find that convenient.

So, in pseudo-pseudo code but actually actual code, I do this:

```python
    while fit.no_divergences:
        print(f'We have {fit.no_divergences} divergences. Increasing `adapt_delta` from {fit.adapt_delta}.')
        fit = fit.resample(
            adapt_delta=np.sqrt(fit.adapt_delta)
        )

```

where `resample` takes the original fit, takes the last draws as initialization, recomputes the metric based on the previous samples, and then just restarts with exactly the same arguments except a higher `adapt_delta`.

My question is, what could go wrong? (Except the loop never terminating).

Edit: Forgot to mention something: Here, resample only runs the [final adaptation window](https://mc-stan.org/docs/2_19/reference-manual/hmc-algorithm-parameters.html) to find a step size and then retries to sample.

---

<div class="post-metadata">

**Author:** ![be\_green](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/be_green/32/9342_2.png) [@be\_green](https://discourse.mc-stan.org/u/be_green)\
**Post date:** [May 20, 2021, 12:36pm UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/2 "2021-05-20T12:36:36Z")

</div>

I think the concern is that usually if adapt\_delta is set very high and still isn’t working then there is probably a large issue with the model… Generally I try to figure out where the degeneracies are happening and reparameterize the model if possible. If you just keep increasing adapt\_delta, you might end up ignoring important parts of your model which aren’t working as expected. Just my 2 cents though.

---

<div class="post-metadata">

**Author:** ![mike-lawrence](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mike-lawrence/32/59_2.png) [@mike-lawrence](https://discourse.mc-stan.org/u/mike-lawrence)\
**Post date:** [May 20, 2021, 12:57pm UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/3 "2021-05-20T12:57:21Z")

</div>

Just chiming in to support @be_green’s response.

Current opinion from folks I respect here seems to be that increasing adapt\_delta is a last resort for fixing divergences. Divergences usually signal that the model has fundamental structural pathology, and addressing those will yield not only elimination of divergences but faster and more accurate inference.

Note that after Andrew’s comment you linked, [there was discussion](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/) to change the wording of the diagnostics output to something more aligned with my statement above.

---

<div class="post-metadata">

**Author:** ![jsocolar](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/jsocolar/32/2486_2.png) [@jsocolar](https://discourse.mc-stan.org/u/jsocolar)\
**Post date:** [May 20, 2021, 1:56pm UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/4 "2021-05-20T13:56:46Z")

</div>

I think this is a great question, and it’s also related to the zeitgeist here that models requiring extremely deep treedepths ought to be reparameterized, even if the computational resources are available to fit the model as-is.

My understanding is that as models venture into difficult territory that requires high `adapt_delta` and long integration times, we should begin to get concerned that the complicated posterior geometry hides important features that we miss in our exploration. Thus we worry about passing diagnostics without having properly explored the posterior.

Note however, that it seems to be generally accepted that we can (and often must) safely use high `treedepth` and `adapt_delta` for some important classes of model, including those with horseshoe priors.

---

<div class="post-metadata">

**Author:** ![mike-lawrence](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mike-lawrence/32/59_2.png) [@mike-lawrence](https://discourse.mc-stan.org/u/mike-lawrence)\
**Post date:** [May 20, 2021, 2:28pm UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/5 "2021-05-20T14:28:06Z")

</div>

> [@jsocolar](#):
>
> models requiring extremely deep treedepths ought to be reparameterized

Ah, I wasn’t aware that high treedepths also signal necessity of reparameterization, thanks!

> [@jsocolar](#):
>
> some important classes of model, including those with horseshoe priors.

Thinking about automated workflows: The new stanc3 --info output lists the functions the model uses; an idea would be to use this to discern if we’re dealing with that class before automatically modifying those sampler parameters. Is the horseshoe a standard `_lpdf` function these days, or are people defining something by hand still?

---

<div class="post-metadata">

**Author:** ![Funko\_Unko](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/funko_unko/32/10786_2.png) [@Funko\_Unko](https://discourse.mc-stan.org/u/Funko_Unko)\
**Post date:** [May 20, 2021, 2:44pm UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/6 "2021-05-20T14:44:25Z")

</div>

Thank you all for your contributions :)

> [@be\_green](#):
>
> I think the concern is that usually if adapt\_delta is set very high and still isn’t working then there is probably a large issue with the model

Yes, that’s what I think as well, I guess at some threshold of `adapt_delta` one will have to give up hope D:

> [@mike-lawrence](#):
>
> Note that after Andrew’s comment you linked, [there was discussion](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/) to change the wording of the diagnostics output to something more aligned with my statement above.

Right, the warning is no longer

> There were 23 divergent transitions after warmup. Increasing adapt\_delta above 0.8 may help.

but apparently

> Examine the pairs() plot to diagnose sampling problems

I didn’t actually notice. I think this makes sense.

> [@jsocolar](#):
>
> Note however, that it seems to be generally accepted that we can (and often must) safely use high `treedepth` and `adapt_delta` for some important classes of model, including those with horseshoe priors.

This is also good to know, thanks!

> [@mike-lawrence](#):
>
> Ah, I wasn’t aware that high treedepths also signal necessity of reparameterization, thanks!

(I think) It may also just signal insufficient adaptation, but this may in turn signal the need for a reparameterization D:

---

<div class="post-metadata">

**Author:** ![mike-lawrence](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mike-lawrence/32/59_2.png) [@mike-lawrence](https://discourse.mc-stan.org/u/mike-lawrence)\
**Post date:** [May 20, 2021, 3:13pm UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/7 "2021-05-20T15:13:11Z")

</div>

> [@Funko\_Unko](#):
>
> It may also just signal insufficient adaptation, but this may in turn signal the need for a reparameterization D:

Do you know if this would be checkable during adaptation itself? I’m working on a non-standard workflow with checks-during-sampling that I could easily make checks-during-warmup-and-sampling and make things continue warmup until this check passes.

---

<div class="post-metadata">

**Author:** ![Funko\_Unko](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/funko_unko/32/10786_2.png) [@Funko\_Unko](https://discourse.mc-stan.org/u/Funko_Unko)\
**Post date:** [May 20, 2021, 3:14pm UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/8 "2021-05-20T15:14:45Z")

</div>

Hmm, In my experience the best indicator that your previous adaptation was insufficient is that further adaptation reduces the average `n_leapfrog__`. So not sure how helpful this is D:

---

<div class="post-metadata">

**Author:** ![mike-lawrence](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mike-lawrence/32/59_2.png) [@mike-lawrence](https://discourse.mc-stan.org/u/mike-lawrence)\
**Post date:** [May 20, 2021, 5:00pm UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/9 "2021-05-20T17:00:27Z")

</div>

> [@Funko\_Unko](#):
>
> In my experience the best indicator that your previous adaptation was insufficient is that further adaptation reduces the average `n_leapfrog__` .

Does this suggest an adaptation termination condition whereby the history of n\_leapfrog is checked for going flat?

---

<div class="post-metadata">

**Author:** ![Funko\_Unko](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/funko_unko/32/10786_2.png) [@Funko\_Unko](https://discourse.mc-stan.org/u/Funko_Unko)\
**Post date:** [May 20, 2021, 5:02pm UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/10 "2021-05-20T17:02:13Z")

</div>

> [@mike-lawrence](#):
>
> Does this suggest an adaptation termination condition whereby the history of n\_leapfrog is checked for going flat?

Hmm, I’m not really the person with the expertise to ask, but to first approximation I would say yes. But probably depends on a thousand other factors.

---

<div class="post-metadata">

**Author:** ![jsocolar](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/jsocolar/32/2486_2.png) [@jsocolar](https://discourse.mc-stan.org/u/jsocolar)\
**Post date:** [May 20, 2021, 5:45pm UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/11 "2021-05-20T17:45:45Z")

</div>

> [@mike-lawrence](#):
>
> Does this suggest an adaptation termination condition whereby the history of n\_leapfrog is checked for going flat?

The problem is that the metric adaptation is windowed, so if you intend to check the distribution of treedepths for stationarity, you end up doing an entire extra window, and since the window size keeps doubling you end up with a huge amount of unnecessary warmup computation (though to be fair, if it goes flat fast enough you might end up saving a huge amount of computation).

If you are going to try to check this adaptively, it seems like it would be more direct to check the metric itself for going flat, rather than indirectly assessing this by looking at the distribution of treedepths.

You also need to be vigilant about the possibility of false flatness due to bad updates to the metric, which should tend to be more severe earlier in warmup.

---

<div class="post-metadata">

**Author:** ![mike-lawrence](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mike-lawrence/32/59_2.png) [@mike-lawrence](https://discourse.mc-stan.org/u/mike-lawrence)\
**Post date:** [May 20, 2021, 5:48pm UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/12 "2021-05-20T17:48:03Z")

</div>

> [@jsocolar](#):
>
> You also need to be vigilant about the possibility of false flatness due to bad updates to the metric, which should tend to be more severe earlier in warmup.

Yes, my thought would be to have a minimum of some sort (possibly consisting of the current standard 1e3 for the entire warmup) before checking

---

<div class="post-metadata">

**Author:** ![andrewgelman](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/andrewgelman/32/1577_2.png) [@andrewgelman](https://discourse.mc-stan.org/u/andrewgelman)\
**Post date:** [May 20, 2021, 8:25pm UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/13 "2021-05-20T20:25:09Z")

</div>

Hi all. Just to clarify: Much (if not most) of the time there are these HMC problems, the solution is to just change the model, perhaps by adding stronger priors or by fixing a bug or a misunderstanding somewhere. Reparameterization can be fine too, but I don’t think “reparameterization” is a great general recommendation because that implies that the model remains unchanged and is just reparameterized. I’d say the first step is to figure out what’s happening and the second step is typically to fix a problem with the model or to add some soft constraints (prior information) to get both computational and inferential stability.

We definitely don’t want to recommend that people automatically increase these tuning parameters, as this can have the effect of making bad models take longer to run. We want to fail fast where possible.

---

<div class="post-metadata">

**Author:** ![Funko\_Unko](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/funko_unko/32/10786_2.png) [@Funko\_Unko](https://discourse.mc-stan.org/u/Funko_Unko)\
**Post date:** [May 21, 2021, 7:11am UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/14 "2021-05-21T07:11:56Z")

</div>

> [@jsocolar](#):
>
> You also need to be vigilant about the possibility of false flatness due to bad updates to the metric, which should tend to be more severe earlier in warmup.

Wait, I just accepted this at first, but why would the metric (spuriously) stay flat?

> [@andrewgelman](#):
>
> I’d say the first step is to figure out what’s happening

Hm, this is what I feared. But in all seriousness, I _do_ wonder how to decide when it is appropriate to increase `adapt_delta`… BRB, I’m just quickly gonna find out.

For some context:

I had a model, which fit perfectly with the default `adapt_delta=0.8`. Then, I added the modelling of some effect in the data, and suddenly I had some divergences during sampling, but with Rhat\<1.01 and EBFMI\>.99, which was new terrain for me.

Then, I introduced the above loop, and the model fit fine, with an `adapt_delta` around .9.

Then however I added an additional term, and suddenly the divergences did not disappear any longer.

The good thing is, I do usually fail as fast as possible, so the feedback that something was wrong was quite immediate. _But_, still I wonder, was the above loop even appropriate in the first place?

Thank you all!

---

<div class="post-metadata">

**Author:** ![jsocolar](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/jsocolar/32/2486_2.png) [@jsocolar](https://discourse.mc-stan.org/u/jsocolar)\
**Post date:** [May 21, 2021, 2:15pm UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/15 "2021-05-21T14:15:06Z")

</div>

> [@Funko\_Unko](#):
>
> Wait, I just accepted this at first, but why would the metric (spuriously) stay flat?

In the long run it won’t but the updates are stochastic, and more-so in the early phases when windows are short.

---

<div class="post-metadata">

**Author:** ![betanalpha](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/betanalpha/32/4_2.png) [@betanalpha](https://discourse.mc-stan.org/u/betanalpha)\
**Post date:** [June 14, 2021, 7:00pm UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/16 "2021-06-14T19:00:57Z")

</div>

> [@Funko\_Unko](#):
>
> My question is, what could go wrong? (Except the loop never terminating).

The loop never terminating is an immediate obstruction for me.

There are _zero_ guarantees that using a less aggressive adaptation will reduce the number of divergences. In many cases using a smaller step size can actually increase the number of divergences as the finer resolution of the exploration can better see just how nasty some part of the target distribution is.

On a more practical note once you push past adaption targets of 0.99 or so there usually isn’t enough information in the early phases of exploration for the adaptation to be able to reach very different values anyways. In other words the adaptation is dominated by the variation in the Markov chain realizations unless _really_ long windows are used. To push past that one would have to configure the step size directly, but that still wouldn’t avoid the lack of guarantees mentioned above.

Even in the best cases forcing a much less aggressive adaptation will typically result in much longer, and more expensive, trajectories. Quickly enough the max treedepth will be saturated and you’ll have to intervene to change that anyways, limiting the automation possibilities a bit.

---

<div class="post-metadata">

**Author:** ![Funko\_Unko](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/funko_unko/32/10786_2.png) [@Funko\_Unko](https://discourse.mc-stan.org/u/Funko_Unko)\
**Post date:** [June 15, 2021, 7:04am UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/17 "2021-06-15T07:04:25Z")

</div>

> [@betanalpha](#):
>
> In many cases using a smaller step size can actually increase the number of divergences as the finer resolution of the exploration can better see just how nasty some part of the target distribution is.

Which is a good thing though, isn’t it? Not for exiting the loop, but for the information it provides.

---

<div class="post-metadata">

**Author:** ![betanalpha](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/betanalpha/32/4_2.png) [@betanalpha](https://discourse.mc-stan.org/u/betanalpha)\
**Post date:** [June 28, 2021, 2:59pm UTC](https://discourse.mc-stan.org/t/automagically-increase-adapt-delta-until-all-divergences-are-eliminated-what-could-go-wrong/22601/18 "2021-06-28T14:59:22Z")

</div>

It’s very a much a good thing _if you can access that information_. Tuning `adapt_delta` by hand and investigating the results (not just the change in the number of divergent transitions but also how the geometry of those divergent transitions changes) is a powerful strategy.

But if you’re attempting to automatically increasing `adapt_delta` until the divergences are eliminated and then end up stuck in a non-terminating loop because the divergences don’t actually decrease with increasing `adapt_delta` then you’ll never have access to that information. Even if you’re dumping the fit info after each iteration to a place where it can be recovered while the loop is still running, you’ll end up wasting a ton of compute on the latter iterations where the step size isn’t changing much and you’re just recovering equivalently pathological fits.

On the other hand working through the iterations by hand – increasing `adapt_delta` and then thoroughly investigating before trying to increase `adapt_delta` again – is straightforward to implement and avoids any computational waste.