CmdStan Guide now online

Interfaces
#3

Yes, the instructions should be:

> git clone https://github.com/stan-dev/cmdstan.git --recursive
> cd cmdstan
> make build

Posted just now: https://github.com/stan-dev/docs/issues/203#issuecomment-653397630

5 Likes
#4

Hvala Rok,

Richard

3 Likes
#5

Looking for feedback on the chapter on the sampler configuration arguments: https://mc-stan.org/docs/2_23/cmdstan-guide/mcmc-config.html

looking for input from power users like @wds15, @bbbales2, @mike-lawrence as well as folks who’ve tried to use CmdStan and run into problems - (I know @lauren has mentioned this to me - I’m sure she’s not alone).

4 Likes
#6

I read through the mcmc-config stuff. It seems okay to me.

  • In the list with the metric options, there’s an unpaired tick after ‘dense_e’: “metric=dense_e`”

  • “By default, all parameters are initialized to random draws from a uniform distribution” -> “By default, all parameters are initialized on an unconstrained scale to random draws from a uniform distribution”. We’re not saying what the unconstrained scale is but it might be confusing to not say there is one.

suggestions for more examples where folks often find themselves wrestling with the command line argument parser?

I think the manual has enough examples, so Discourse is probably the place.

3 Likes
#7

Just finished a read-through, here are some notes:

  • 1.3: stan-update is mentioned once but never again; possibly an example would help?
  • 3.2: typo: “willis”
  • 4.4: what’s going on with the ridiculously high Rhat for stepsize__?
  • 7: generated quantities: is this section incomplete? For example, it’s unclear if the bernoulli_yrep.stan should literally only contain a generated quantities section, or should it have all the content from bernoulli.stan as well?
  • 9.1: “An EFF of at least 100 is required to make a viable estimate. The precision of your estimate is √N; therefore every additional decimal place of accuracy increases this by a factor of 10.” Ah, this is something I hadn’t come across before (I only had a vague related intuition). Very useful.
  • 9.1 On thinning: should it say something like “Some users familiar with older approaches to MCMC sampling might be used to thinning to eliminate an expected autocorrelation in the samples. HMC is not nearly as susceptible to this autocorrelation problem and thus thinning is generally not required nor advised except in circumstances where storage of the samples is limited and/or RAM for later processing the samples is limited.” ?
  • 9.2.1 “Raising the value of delta will also allow some models that would otherwise get stuck to overcome their blockages” & also this from 16.3.1: “If the divergent transitions cannot be eliminated by increasing the adapt_delta parameter, we have to find a different way to write the model that is logically equivalent but simplifies the geometry of the posterior distribution.” --> This seems to imply that the first step in troubleshooting divergences is to change adapt_delta, but I thought it was becoming more strongly recommended that folks not do this, and instead first look to changing the model geometry, which could yield both more efficient and faster sampling.
  • 9.5.1: what happens if you specify the random seed and id as ${i} ? Would this thwart the in-built mechanisms for avoiding overlap in seeds?
7 Likes
#8

good question. I don’t think the Rhat statistic makes sense for stepsize, but I’m curious as to why it has the value that it does. pinging @avehtari - what is the correct thing to report? don’t analyze sample state columns in Stan csv output file?

yes. working on it now.

excellent suggestion - will add - many thanks!

I could move or copy the discussion from 9.5.6 (https://mc-stan.org/docs/2_23/cmdstan-guide/mcmc-config.html#changing-the-nuts-hmc-adaptation-parameters) to 9.2.1.
note that not long ago, this was disussed elsewhere - Improve warnings for low ESS - at which point Bob said:

short answer - no - each chain will be running off an RNG which has its own seed.
the samplers use both the seed and the id together to create the RNG used by the sampler - https://github.com/stan-dev/stan/blob/develop/src/stan/services/util/create_rng.hpp

many thanks for the close read and good suggestions!

#9

Great stuff @mike-lawrence

Slightly related question:

Could we fix that https://mc-stan.org/docs/ works and is not a 404 error. Either redirect to the latest docs or a link to a landing page with just links of all the docs (2.23, 2.22, …)

2 Likes
#10

Correct, there is no sense computing or reporting Rhat or ESS for sampler diagnostic columns.

Also it would be useful give an example value to try (e.g. 0.9 or 0.95), that logically it has to be <1, and it’s rarely sensible to try adapt_delta>0.99 ) as it is already strong indication of bad geometry and the sampling tends to take long time with larger adapt_delta values.

#11

Ok, but just double-checking: if one were to non-sensically attempt to compute an Rhat on the stepsize__, a value as extreme as 2.5e+13 is a reasonable expectation and not a signal that something is concerningly awry in the Rhat computation generally?

#12

Can you provide the sequence and the exact Rhat code you call? The latest implementation in rstan monitor.R and in posterior package should not provide anything like that unless you have similar order of magnitude of iterations.

#13

This was observed in @mitzimorris’s new cmdstan guide, section 4.4, where stansummary is used on a simple binomial model.

#14

The link you provide for for “section 4.4” is https://discourse.mc-stan.org/t/brms-constrain-group-level-effects-to-positive-values/16160

#15

Oops! I wonder how that happened; I didn’t even look at that thread (since I don’t know brms at all). Edited the above to have the correct url.

#16

This is a guess, but I think the difference is that the rhat C++ code in https://github.com/stan-dev/stan/blob/d8c34d315f92892a9d19b96e06b196bd7640b7e5/src/stan/analyze/mcmc/compute_potential_scale_reduction.hpp should be updated to what posterior is using, which is based on this paper https://arxiv.org/abs/1903.08008 right?

This C++ is what cmdstan uses if you use stansummary. (cmdstanr uses posteriors implementation if you dont specifically want the cmdstan one).

1 Like
#17

I just realized that for Rhat, no matter which version, this doesn’t make sense. Step size should be constant after warm-up. Thus all within variances should be zero. But different chains may have different constant step size, so that between variance should be positive. So there is some floating point accuracy mishap in the current C++ so that var_between / var_within ends up being really large number instead of Inf. The correct results would be Inf, but then it doesn’t make sense to report Rhat and ESS for
accept_stat__, stepsize__, treedepth__, n_leapfrog__, and divergent__

2 Likes
#18

So it seems that boost::accumulators::variance(acc_draw) is not returning 0 variance for the constant step size for at least one of the chains (). It seems boost::accumulators::variance uses equation M_n^{(2)} - \mu_n^2, which may get numerical floating point error.

3 Likes
#19

just filed an issue to omit sampler diagnostic columns from CmdStan stansummary: https://github.com/stan-dev/cmdstan/issues/903

question - report Rhat and ESS for lp__ ?

1 Like
#20

@mitzimorris In section 9.2.1 (“Step size optimization configuration”), should the second gamma be kappa?

2 Likes
#21

yes!! I’ve fixed that in the Rmd, haven’t republished.

#22

@mitzimorris Will the online CmdStanR guide eventually have a URL that links to the latest version rather than having to specify a particular version number? I was going to link to it from the CmdStanR doc but instead of the current link with a version number

we’d need something like

https://mc-stan.org/docs/cmdstan-guide/

to point to the latest version so that we don’t need to keep updating URLs everywhere we link to it. Does that make sense?