# Sampler parameters which the typical user might need to set (hmc\_nuts\_diag\_e\_adapt only)

**URL:** <https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738>\
**Category:** Developers\
**Created:** [November 4, 2019, 7:18pm UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738 "2019-11-04T19:18:05Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![ariddell](https://avatars.discourse-cdn.com/v4/letter/a/57b2e6/32.png) [@ariddell](https://discourse.mc-stan.org/u/ariddell)\
**Post date:** [November 4, 2019, 7:18pm UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/1 "2019-11-04T19:18:05Z")

</div>

In NYC this summer we talked at one point about simplifying the default sampling function (`hmc_nuts_diag_e_adapt`) in PyStan/httpstan/others(?) by fixing some of the sampler parameters at their default values (i.e., not allowing the typical user to set them). (Advanced users would have lots of alternatives.)

I recall there being a consensus that the typical user should not manipulate most of the parameters. I’m having trouble recalling which ones the typical user should potentially be allowed to change. Can someone help? I think @betanalpha had opinions about this.

For reference, here are the sampler parameters I’m thinking about:

- mass matrix
- stepsize
- stepsize\_jitter
- max\_depth
- delta
- gamma
- kappa
- t0
- init\_buffer
- term\_buffer
- window
- init\_radius

The thinking was that some of these sampler parameters depend on each other – manipulating them correctly is generally not going to yield good results for the typical PyStan/Rstan/\*Stan user.

Added 2019-11-05: Added `init_radius` to discussion.

---

<div class="post-metadata">

**Author:** ![betanalpha](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/betanalpha/32/4_2.png) [@betanalpha](https://discourse.mc-stan.org/u/betanalpha)\
**Post date:** [November 5, 2019, 3:50am UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/2 "2019-11-05T03:50:13Z")

</div>

`stepwise_jitter` should be deprecated regardless.

`delta`, `gamma`, `kappa`, and `t0` configure the dual averaging of the integrator step size. The last three probably shouldn’t be touched by anyone not heavily versed in Nesterov optimization theory, but `delta` defines the target adaptation stat and would nominally be exposed in case of divergences. That said, it might be easier to avoid `delta` entirely and have users set the step size themselves (without the adaptation being able to change it).

~~`mass matrix`~~ `inv_metric`, `init_buffer`, `term_buffer`, and `window` configure the adaptation of the inverse metric elements. It’s not that these shouldn’t be exposed to the user but rather that they should be exposed only in certain patterns. For example if a user wants to specify their own inverse metric components and not allow them to be changed then they might call something like `adaptation=static_inv_metric` or something where `inv_metric` is set and `window` is forced to zero at the same time.

---

<div class="post-metadata">

**Author:** ![ariddell](https://avatars.discourse-cdn.com/v4/letter/a/57b2e6/32.png) [@ariddell](https://discourse.mc-stan.org/u/ariddell)\
**Post date:** [November 5, 2019, 12:25pm UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/3 "2019-11-05T12:25:48Z")

</div>

Perfect. So just `stepsize` for beginners – and `max_depth`?

While I have everyone’s attention. What about `init_radius`? Does anyone use this? I always specify initial values directly. I might have used `init_radius` once in my life.

---

<div class="post-metadata">

**Author:** ![avehtari](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/avehtari/32/5935_2.png) [@avehtari](https://discourse.mc-stan.org/u/avehtari)\
**Post date:** [November 5, 2019, 2:31pm UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/4 "2019-11-05T14:31:53Z")

</div>

> [@ariddell](#):
>
> What about `init_radius` ? Does anyone use this?

Sometimes

---

<div class="post-metadata">

**Author:** ![stevebronder](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/stevebronder/32/17242_2.png) [@stevebronder](https://discourse.mc-stan.org/u/stevebronder)\
**Post date:** [November 5, 2019, 4:45pm UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/5 "2019-11-05T16:45:09Z")

</div>

Should we get rid of them totally? Which params would a user need from a Stan model in order to restart a run?

---

<div class="post-metadata">

**Author:** ![ariddell](https://avatars.discourse-cdn.com/v4/letter/a/57b2e6/32.png) [@ariddell](https://discourse.mc-stan.org/u/ariddell)\
**Post date:** [November 5, 2019, 10:03pm UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/6 "2019-11-05T22:03:48Z")

</div>

@betanalpha I’m only thinking about `hmc_nuts_diag_e_adapt` at the moment (to use the Stan services function name). This is going to be what users use when they call the default `sample` function in PyStan 3.

Are you saying that the user should ignore delta and manually set `stepsize` with no stepsize adaptation? (This is `hmc_nuts_diag_e` right?)

---

<div class="post-metadata">

**Author:** ![betanalpha](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/betanalpha/32/4_2.png) [@betanalpha](https://discourse.mc-stan.org/u/betanalpha)\
**Post date:** [November 7, 2019, 1:12am UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/7 "2019-11-07T01:12:14Z")

</div>

> [@ariddell](#):
>
> Perfect. So just `stepsize` for beginners – and `max_depth` ?

Stepsize adaptation should largely be limited to defaults, with `adapt_delta` or another method to lower the initial fitted step size in the presence of divergences. `max_depth` should be accessible for those with particularly long trajectories.

> [@ariddell](#):
>
> While I have everyone’s attention. What about `init_radius` ?

Everytime I’ve used it I was dealing with what became clear in hindsight was a bad model, and I think that’s the circumstance _most_ of the time its employees. That said I don’t see any reason to remove it as there are principled ways to use it based on prior choice and general scales in the problem.

> [@ariddell](#):
>
> I’m only thinking about `hmc_nuts_diag_e_adapt` at the moment (to use the Stan services function name)

As we talked about at the developer retreat this function is probably too overloaded for most users. Instead of having a suite of adaptation parameters that configure the adaptation routes we talked about breaking the main service route into different functions, each designed for a specified kind of adaptation. For example `nominal`, `restart`, `init_metric`, `reduced_stepsize` or the like.

> [@ariddell](#):
>
> Are you saying that the user should ignore delta and manually set `stepsize` with no stepsize adaptation? (This is `hmc_nuts_diag_e` right?)

No. By default they shouldn’t set anything; the question is rather what happens when divergences pop up and they have to modify the step size.

---

<div class="post-metadata">

**Author:** ![ariddell](https://avatars.discourse-cdn.com/v4/letter/a/57b2e6/32.png) [@ariddell](https://discourse.mc-stan.org/u/ariddell)\
**Post date:** [November 7, 2019, 4:22pm UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/8 "2019-11-07T16:22:25Z")

</div>

In the interest of simplicity (fewer possible arguments), one could specify initial values directly instead of using `init_radius`, right?

---

<div class="post-metadata">

**Author:** ![ariddell](https://avatars.discourse-cdn.com/v4/letter/a/57b2e6/32.png) [@ariddell](https://discourse.mc-stan.org/u/ariddell)\
**Post date:** [November 7, 2019, 4:25pm UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/9 "2019-11-07T16:25:06Z")

</div>

> [@betanalpha](#):
>
> As we talked about at the developer retreat this function is probably too overloaded for most users. Instead of having a suite of adaptation parameters that configure the adaptation routes we talked about breaking the main service route into different functions, each designed for a specified kind of adaptation. For example `nominal` , `restart` , `init_metric` , `reduced_stepsize` or the like.

Ok. This sounds really good. In the interest of having a minimum viable PyStan 3, which of these shall I implement? Or should I put in the docs “if you encounter a problem with divergences, you need to use cmdstan or pystan 2”?

---

<div class="post-metadata">

**Author:** ![stevebronder](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/stevebronder/32/17242_2.png) [@stevebronder](https://discourse.mc-stan.org/u/stevebronder)\
**Post date:** [November 7, 2019, 4:56pm UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/10 "2019-11-07T16:56:27Z")

</div>

Instead of having them all as function arguments what if we allowed the user to specify a config file for the sample? So it would be like

```python
my_mod.sample(blah, blah1, sampler_config="./my_settings.conf")

```

So then we reduce the args in the function while also letting people fiddle with things if they want

---

<div class="post-metadata">

**Author:** ![ariddell](https://avatars.discourse-cdn.com/v4/letter/a/57b2e6/32.png) [@ariddell](https://discourse.mc-stan.org/u/ariddell)\
**Post date:** [November 7, 2019, 6:47pm UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/11 "2019-11-07T18:47:25Z")

</div>

This isn’t a common technique—in Python libraries at least. A key  
disadvantage is that it requires working with the file system, which can  
vary dramatically depending on the OS. We also want to allow people  
without file systems (say online Jupyter users) to be allowed to  
configure things.

---

<div class="post-metadata">

**Author:** ![mitzimorris](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/mitzimorris/32/29_2.png) [@mitzimorris](https://discourse.mc-stan.org/u/mitzimorris)\
**Post date:** [November 7, 2019, 7:06pm UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/12 "2019-11-07T19:06:39Z")

</div>

fwiw, here’s the current set of params for CmdStanPy’s `sample` method:

[https://cmdstanpy.readthedocs.io/en/latest/api.html#cmdstanpy.CmdStanModel.sample](https://cmdstanpy.readthedocs.io/en/latest/api.html#cmdstanpy.CmdStanModel.sample)

---

<div class="post-metadata">

**Author:** ![stevebronder](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/stevebronder/32/17242_2.png) [@stevebronder](https://discourse.mc-stan.org/u/stevebronder)\
**Post date:** [November 7, 2019, 7:07pm UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/13 "2019-11-07T19:07:53Z")

</div>

That’s fine was mostly throwing it out there, if the goal is just to shrink the signature some sort of dictionary config would probs do that as well

---

<div class="post-metadata">

**Author:** ![betanalpha](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/betanalpha/32/4_2.png) [@betanalpha](https://discourse.mc-stan.org/u/betanalpha)\
**Post date:** [November 12, 2019, 4:30am UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/14 "2019-11-12T04:30:48Z")

</div>

> [@ariddell](#):
>
> This sounds really good. In the interest of having a minimum viable PyStan 3, which of these shall I implement?

In my opinion a minimal viable interface would have nominal and a way to adjust the step size, through modifying `adapt_delta` or whatever. Modifying the metric by hand is definitely a more advanced feature.

> [@stevebronder](#):
>
> That’s fine was mostly throwing it out there, if the goal is just to shrink the signature some sort of dictionary config would probs do that as well

One possibility that has been discussed is having a stateful object that initializes to the default configuration which can then be modified through mutator methods. This was the intention of the argument configuration validator class used in CmdStan, but it’s not clear how that would work with a C++ API being accessed by both Python and R clients.

---

<div class="post-metadata">

**Author:** ![stevebronder](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/stevebronder/32/17242_2.png) [@stevebronder](https://discourse.mc-stan.org/u/stevebronder)\
**Post date:** [November 12, 2019, 5:26am UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/15 "2019-11-12T05:26:36Z")

</div>

> [@betanalpha](#):
>
> One possibility that has been discussed is having a stateful object that initializes to the default configuration which can then be modified through mutator methods.

I think we are both talking about a input config and output config with defaults in which case yes I agree

> [@betanalpha](#):
>
> but it’s not clear how that would work with a C++ API being accessed by both Python and R clients.

In my mindspace this feels like an R tmpfile and python pickle we point cmdstan to right?

---

<div class="post-metadata">

**Author:** ![Charles\_Driver](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/charles_driver/32/6524_2.png) [@Charles\_Driver](https://discourse.mc-stan.org/u/Charles_Driver)\
**Post date:** [November 12, 2019, 11:33am UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/16 "2019-11-12T11:33:03Z")

</div>

Specifying initial values is seriously annoying. The radius (or r equiv) is very handy. I’m happy for information, but I prefer software that doesn’t take every opportunity to beat me over the head with a stick in case my model is not somebody’s idea of optimal.

---

<div class="post-metadata">

**Author:** ![betanalpha](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/betanalpha/32/4_2.png) [@betanalpha](https://discourse.mc-stan.org/u/betanalpha)\
**Post date:** [November 14, 2019, 2:29pm UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/17 "2019-11-14T14:29:26Z")

</div>

> [@stevebronder](#):
>
> In my mindspace this feels like an R tmpfile and python pickle we point cmdstan to right?

The problem is that this introduces a file system intermediary which breaks the up the current flow of the interfaces, where everything is local to the interface environment.

---

<div class="post-metadata">

**Author:** ![Bob\_Carpenter](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bob_carpenter/32/9230_2.png) [@Bob\_Carpenter](https://discourse.mc-stan.org/u/Bob_Carpenter)\
**Post date:** [November 27, 2019, 7:43pm UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/18 "2019-11-27T19:43:15Z")

</div>

> [@ariddell](#):
>
> (i.e., not allowing the typical user to set them). (Advanced users would have lots of alternatives.)

How do you prevent typical users from setting a feature that an advanced user could set?

> [@betanalpha](#):
>
> > What about `init_radius` ?
> 
> Everytime I’ve used it I was dealing with what became clear in hindsight was a bad model,

I find a need to lower this in 100-coefficient logistic regression simulations. Those can be really numerically unstable at wide initial values.

> [@ariddell](#):
>
> I always specify initial values directly.

What kind of models do you have where that’s necessary? We almost never have to do that with the more recent versions of Stan.

> [@ariddell](#):
>
> one could specify initial values directly instead of using `init_radius` , right?

That’s a huge burden for large models.

> [@betanalpha](#):
>
> a minimal viable interface would have nominal

Does “nominal” just mean default here?

---

<div class="post-metadata">

**Author:** ![betanalpha](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/betanalpha/32/4_2.png) [@betanalpha](https://discourse.mc-stan.org/u/betanalpha)\
**Post date:** [December 3, 2019, 3:04pm UTC](https://discourse.mc-stan.org/t/sampler-parameters-which-the-typical-user-might-need-to-set-hmc-nuts-diag-e-adapt-only/11738/19 "2019-12-03T15:04:21Z")

</div>

> [@Bob\_Carpenter](#):
>
> How do you prevent typical users from setting a feature that an advanced user could set?

Obfuscation through documentation – they would just not be shown in the introductory treatments and examples. Many of the existing arguments have this flavor already.

> [@Bob\_Carpenter](#):
>
> Does “nominal” just mean default here?

Yes, “nominal” in the physics/engineering sense of default.