2 problems with cmdstanr: one model gives the wrong answer, the other gives an error in compilation

Interfaces CmdStan
1

Thanks to Mitzi, I was able to get cmdstanr running on my mac. But when I tried to run it, I had two problems.

  1. The example model on the cmdstanr Getting Started page compiled and ran but gave the wrong answer.

  2. The 8 schools model gave an error in compilation.

I don’t know if these two problems are related. I’ll describe them both below. I’m on Rstudio version 1.2.5033, R version 3.6.1, Mac Catalina 10.15.4, rstan 2.19.2 (not using rstan for this, but it’s in my environment so I thought I’d let you know), cmdstanr version 0.0.0.9000

–

Problem 1. The example model on the cmdstanr Getting Started page compiled and ran but gave the wrong answer.

I went to the example on the cmdstanr Getting Started page:

file <- file.path(cmdstan_path(), "examples", "bernoulli", "bernoulli.stan")
mod <- cmdstan_model(file)

and this seemed to work fine. I checked by printing the model:

mod$print()

and it was just as on the Getting Started page. But then I ran it:

data_list <- list(N = 10, y = c(0,1,0,0,0,0,0,0,0,1))
fit <- mod$sample(
  data = data_list, 
  seed = 123, 
  num_chains = 2, 
  num_cores = 2
)

and I got this:

1159 of 2000 (58.0%) transitions hit the maximum treedepth limit of 10 or 2^10-1 leapfrog steps.
Trajectories that are prematurely terminated due to this limit will result in slow exploration.
Increasing the max_depth limit can avoid this at the expense of more computation.
If increasing max_depth does not remove warnings, try to reparameterize the model.

and this:

> fit$summary()
# A tibble: 3 x 10
  variable      mean    median       sd      mad        q5       q95  rhat ess_bulk
  <chr>        <dbl>     <dbl>    <dbl>    <dbl>     <dbl>     <dbl> <dbl>    <dbl>
1 lp__     -7.31e+ 0 -7.01e+ 0 8.09e- 1 3.58e- 1 -8.87e+ 0 -6.75e+ 0  1.00   770.  
2 theta     2.54e- 1  2.35e- 1 1.25e- 1 1.28e- 1  8.04e- 2  4.77e- 1  1.00  1597.  
3 sigma     1.13e+20  9.99e+19 1.03e+20 1.03e+20 -2.43e+19  3.14e+20  1.65     3.43
# … with 1 more variable: ess_tail <dbl>

Uh oh!

–

Problem 2. The 8 schools model gave an error in compilation.

Next I thought I’d try the 8 schools model. I used the following file, “8schools.stan”:

data {
  int<lower=0> J;         // number of schools 
  real y[J];              // estimated treatment effects
  real<lower=0> sigma[J]; // standard error of effect estimates 
}
parameters {
  real mu;                // population treatment effect
  real<lower=0> tau;      // standard deviation in treatment effects
  vector[J] eta;          // unscaled deviation from mu by school
}
transformed parameters {
  vector[J] theta = mu + tau * eta;        // school treatment effects
}
model {
  target += normal_lpdf(eta | 0, 1);       // prior log-density
  target += normal_lpdf(y | theta, sigma); // log-likelihood
}

and then I did this:

mod <- cmdstan_model("8schools.stan")

And this is what appeared in the Rstudio console:

Compiling Stan program...
Warning:
  The parameter mu has 0 priors.
Warning:
  The parameter tau has 0 priors.
Warning:
  The parameter theta has 0 priors.
/var/folders/j8/kg_4ryhs4d78nts87zqw73wh0000gn/T/RtmphsxHbn/model-38f583623e3.hpp:4:11: error: expected identifier or '{'
namespace 8schools_model_namespace {
          ^
/var/folders/j8/kg_4ryhs4d78nts87zqw73wh0000gn/T/RtmphsxHbn/model-38f583623e3.hpp:4:11: error: expected unqualified-id
/var/folders/j8/kg_4ryhs4d78nts87zqw73wh0000gn/T/RtmphsxHbn/model-38f583623e3.hpp:494:9: error: expected unqualified-id
typedef 8schools_model_namespace::8schools_model stan_model;
        ^
/var/folders/j8/kg_4ryhs4d78nts87zqw73wh0000gn/T/RtmphsxHbn/model-38f583623e3.hpp:503:3: error: unknown type name 'stan_model'
  stan_model* m = new stan_model(data_context, seed, msg_stream);
  ^
/var/folders/j8/kg_4ryhs4d78nts87zqw73wh0000gn/T/RtmphsxHbn/model-38f583623e3.hpp:503:23: error: unknown type name 'stan_model'
  stan_model* m = new stan_model(data_context, seed, msg_stream);
                      ^
5 errors generated.
make: *** [/var/folders/j8/kg_4ryhs4d78nts87zqw73wh0000gn/T/RtmphsxHbn/model-38f583623e3] Error 1
Error in processx::run(command = make_cmd(), args = c(tmp_exe, include_paths,  : 
  System command 'make' failed, exit status: 2, stderr (last 10 lines):
E> typedef 8schools_model_namespace::8schools_model stan_model;
E>         ^
E> /var/folders/j8/kg_4ryhs4d78nts87zqw73wh0000gn/T/RtmphsxHbn/model-38f583623e3.hpp:503:3: error: unknown type name 'stan_model'
E>   stan_model* m = new stan_model(data_context, seed, msg_stream);
E>   ^
E> /var/folders/j8/kg_4ryhs4d78nts87zqw73wh0000gn/T/RtmphsxHbn/model-38f583623e3.hpp:503:23: error: unknown type name 'stan_model'
E>   stan_model* m = new stan_model(data_context, seed, msg_stream);
E>                       ^
E> 5 errors generated.
E> make: *** [/var/folders/j8/kg_4ryhs4d78nts87zqw73wh0000gn/T/RtmphsxHbn/model-38f583623e3] Error 1
Type .Last.error.trace to see where the error occured

–

A few problems here:

First, it said, “Warning: The parameter theta has 0 priors.” But theta is a transformed parameter. I don’t think you’d expect to have a prior on a transformed parameter. I mean, sure, you can, but it wouldn’t be the usual thing. So I don’t think this warning is appropriate.

Second, it said “occured” rather than “occurred” in the error message. No big deal but we might as well fix typos.

Third, and most importantly, it gave a compilation error, and I can’t figure out what the error is. I followed the instructions and typed .Last.error.trace in the Rstudio console, and this is what came out:

Stack trace:

 1. cmdstanr:::cmdstan_model("8schools.stan")
 2. CmdStanModel$new(stan_file = stan_file, compile = compile, ...)
 3. .subset2(public_bind_env, "initialize")(...)
 4. self$compile(...)
 5. processx::run(command = make_cmd(), args = c(tmp_exe, include_paths,  ...
 6. throw(new_process_error(res, call = sys.call(), echo = echo,  ...

 x System command 'make' failed, exit status: 2, stderr (last 10 lines):
E> typedef 8schools_model_namespace::8schools_model stan_model;
E>         ^
E> /var/folders/j8/kg_4ryhs4d78nts87zqw73wh0000gn/T/RtmphsxHbn/model-38f583623e3.hpp:503:3: error: unknown type name 'stan_model'
E>   stan_model* m = new stan_model(data_context, seed, msg_stream);
E>   ^
E> /var/folders/j8/kg_4ryhs4d78nts87zqw73wh0000gn/T/RtmphsxHbn/model-38f583623e3.hpp:503:23: error: unknown type name 'stan_model'
E>   stan_model* m = new stan_model(data_context, seed, msg_stream);
E>                       ^
E> 5 errors generated.
E> make: *** [/var/folders/j8/kg_4ryhs4d78nts87zqw73wh0000gn/T/RtmphsxHbn/model-38f583623e3] Error 1

–

Thanks in advance for your help!
Andrew

2

Hi @andrewgelman, I’m just editing your post for formatting to make it more readable. Discourse is being a bit weird, but I’ll get it done.

EDIT: done.

1 Like
3

Thanks.

4

is this warning from the pedantic mode version of the stanc3 compiler?

5

I assume so. Awhile ago I followed your insructions to set up pedantic mode so I guess that’s what was running?

6

The bernoulli example on this pedantic mode branch was modified by adding a non used parameter. Just so we test that pedantic mode is running and working.

This is the model code:

data {
  int<lower=0> N;
  int<lower=0,upper=1> y[N];
}

parameters {
  real<lower=0,upper=1> theta;
  real sigma;
}

model {
  theta ~ beta(1,1); // uniform prior on interval 0,1
  y ~ bernoulli(theta);
}

If you remove sigma, all is well. So the pedantic mode warning here is very helpful.

These are related to the pedantic mode branch you are using. Not directly with cmdstanr. I am going to post it to the pedantic mode thread so Ryan doesnt miss them.

The compile error on 8schools occurs because the filename starts with a number. This doesnt play well with cmdstan, which has always been the case and I think is also listed in the cmdstan guide. But the way it errors currently is bad. Will fix that on cmdstan/stanc3 for this release.

4 Likes
7

Interesting. If cmdstan doesn’t like files beginning with a number, then maybe we should remove filenames beginning with a number in other documentation.

I got that 8schools.stan name from the rstan Getting Started page. Perhaps we should rename it to schools.stan and make sure that we don’t have other filenames beginning with numbers in any of our documentation, to avoid problems in conversion if we do start recommending that people use cmdstanr and cmdstanpy.

Also, I guess things will improve if cmdstan returns an error message, something like, “You can’t have a file name that begins with a number.” People won’t always read through the entire manual to find this!

2 Likes
8

They mostly already are. For instance 8 schools is eight_schools in the example models repository: https://github.com/stan-dev/example-models/blob/48dcab3e1b63c57948389602e9be469cb6702484/misc/eight_schools/eight_schools.stan

Its probably the docs that are associated with rstan that use 8schools. Because it works there.

There was an error message there before, but this never got put in stanc3. Fixing now: Error on model name that starts with a number or non-underscore symbol by rok-cesnovar · Pull Request #507 · stan-dev/stanc3 · GitHub

1 Like