# Why doesn't sampling fail when optimizing does?

**URL:** <https://discourse.mc-stan.org/t/why-doesnt-sampling-fail-when-optimizing-does/13094>\
**Category:** Algorithms\
**Tags:** optimization, mcmc\
**Created:** [February 12, 2020, 10:57am UTC](https://discourse.mc-stan.org/t/why-doesnt-sampling-fail-when-optimizing-does/13094 "2020-02-12T10:57:24Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jeffpollock9](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/jeffpollock9/32/3826_2.png) [@jeffpollock9](https://discourse.mc-stan.org/u/jeffpollock9)\
**Post date:** [February 12, 2020, 10:57am UTC](https://discourse.mc-stan.org/t/why-doesnt-sampling-fail-when-optimizing-does/13094/1 "2020-02-12T10:57:24Z")

</div>

Hi,

As far as I understand, finding the posterior mode via optimization usually doesn’t work for hierarchical models since the objective function is unbounded due to the contribution from the prior, e.g.:

```R
# parameters
effects <- c(0.0, 0.0)
effects_sd <- 0.0

# prior lpdf
effects_prior_lpdf <- sum(dnorm(effects, 0.0, effects_sd, log = TRUE))

> print(effects_prior_lpdf)
[1] Inf

```

so the maximum value of the objective function could be found at infinity via setting the all the hierarchical effects and scale to 0. In this case the solution effectively ignores any contribution to the objective function from the likelihood - which doesn’t sound particularly useful.

What I am struggling to get my head round is, why does sampling work fine? Why doesn’t the sampler eventually propose some value sufficiently close this unbounded mode and get stuck?

Any help would be awesome - thanks!

---

<div class="post-metadata">

**Author:** ![betanalpha](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/betanalpha/32/4_2.png) [@betanalpha](https://discourse.mc-stan.org/u/betanalpha)\
**Post date:** [February 12, 2020, 2:51pm UTC](https://discourse.mc-stan.org/t/why-doesnt-sampling-fail-when-optimizing-does/13094/2 "2020-02-12T14:51:12Z")

</div>

The posterior mode is a consequence of an arbitrary parameterization and is not “typical” of the posterior _distribution_ itself. Algorithms like Markov chain Monte Carlo try to quantify an entire probability distribution and hence rarely spend any time around the uncharacteristic mode.

For a much more detailed explanation see [https://betanalpha.github.io/assets/case\_studies/probabilistic\_computation.html](https://betanalpha.github.io/assets/case_studies/probabilistic_computation.html).

---

<div class="post-metadata">

**Author:** ![jeffpollock9](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/jeffpollock9/32/3826_2.png) [@jeffpollock9](https://discourse.mc-stan.org/u/jeffpollock9)\
**Post date:** [February 13, 2020, 8:28am UTC](https://discourse.mc-stan.org/t/why-doesnt-sampling-fail-when-optimizing-does/13094/3 "2020-02-13T08:28:22Z")

</div>

Thanks for this reply, @betanalpha. The case study is super helpful!

---

<div class="post-metadata">

**Author:** ![Bob\_Carpenter](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bob_carpenter/32/9230_2.png) [@Bob\_Carpenter](https://discourse.mc-stan.org/u/Bob_Carpenter)\
**Post date:** [February 13, 2020, 9:26pm UTC](https://discourse.mc-stan.org/t/why-doesnt-sampling-fail-when-optimizing-does/13094/4 "2020-02-13T21:26:29Z")

</div>

> [@jeffpollock9](#):
>
> Why doesn’t the sampler eventually propose some value sufficiently close this unbounded mode and get stuck?

A slightly more technical answer is that the sampler will stay in the typical set, which is the set of elements with log density within epsilon of the entropy (expected log density). The typical set usually doesn’t contain the mode in even moderately high dimensions.

I find it useful to work through simple examples by simulation, which I do in this case study: [Typical Sets and the Curse of Dimensionality](https://mc-stan.org/users/documentation/case-studies/curse-dims.html)