File ending of .R is not supported by cmdstan

Rob_J_Goedman · January 27, 2022, 10:57pm

Hi,

Recently I’ve been getting:

file ending of .R is not supported by cmdstan
ERROR: LoadError: failed process: Process(`/var/folders/l7/pr04h0650q5dvqttnvs8s2c00000gn/T/jl_47OF5E/bernoulli num_threads=4 sample num_chains=4 num_samples=1000 num_warmup=1000 save_warmup=0 thin=1 adapt engaged=1 gamma=0.05 delta=0.8 kappa=0.75 t0=10 init_buffer=75 term_buffer=75 window=25 algorithm=hmc engine=nuts max_depth=10 metric=diag_e stepsize=1.0 stepsize_jitter=0.0 random seed=-1 init=/var/folders/l7/pr04h0650q5dvqttnvs8s2c00000gn/T/jl_47OF5E/bernoulli_init_1.R id=1 data file=/var/folders/l7/pr04h0650q5dvqttnvs8s2c00000gn/T/jl_47OF5E/bernoulli_data_1.R output file=/var/folders/l7/pr04h0650q5dvqttnvs8s2c00000gn/T/jl_47OF5E/bernoulli_chain_1.csv refresh=100`, ProcessExited(1)) [1]

when trying to provide an init .R file (bernoulli_init_1.R above). Is that no longer supported?

Rob_J_Goedman · January 28, 2022, 2:58am

Just tried it using the cmdstan installation test (bernoulli). Same error with an .R init file, works fine with a .json init file. This is with cmdstan-2.28.2

rob@Rob-M1 cmdstan % ./examples/bernoulli/bernoulli num_threads=6 sample num_chains=4 data file=examples/bernoulli/bernoulli.data.json init=examples/bernoulli/bernoulli.init.R

method = sample (Default)
  sample
    num_samples = 1000 (Default)
    num_warmup = 1000 (Default)
    save_warmup = 0 (Default)
    thin = 1 (Default)
    adapt
      engaged = 1 (Default)
      gamma = 0.050000000000000003 (Default)
      delta = 0.80000000000000004 (Default)
      kappa = 0.75 (Default)
      t0 = 10 (Default)
      init_buffer = 75 (Default)
      term_buffer = 50 (Default)
      window = 25 (Default)
    algorithm = hmc (Default)
      hmc
        engine = nuts (Default)
          nuts
            max_depth = 10 (Default)
        metric = diag_e (Default)
        metric_file =  (Default)
        stepsize = 1 (Default)
        stepsize_jitter = 0 (Default)
    num_chains = 4
id = 1 (Default)
data
  file = examples/bernoulli/bernoulli.data.json
init = examples/bernoulli/bernoulli.init.R
random
  seed = 2771963563 (Default)
output
  file = output.csv (Default)
  diagnostic_file =  (Default)
  refresh = 100 (Default)
  sig_figs = -1 (Default)
  profile_file = profile.csv (Default)
num_threads = 6

file ending of .R is not supported by cmdstan

rob@Rob-M1 cmdstan % ./examples/bernoulli/bernoulli num_threads=6 sample num_chains=4 data file=examples/bernoulli/bernoulli.data.json init=examples/bernoulli/bernoulli.init.json

method = sample (Default)
  sample
    num_samples = 1000 (Default)
    num_warmup = 1000 (Default)
    save_warmup = 0 (Default)
    thin = 1 (Default)
    adapt
      engaged = 1 (Default)
      gamma = 0.050000000000000003 (Default)
      delta = 0.80000000000000004 (Default)
      kappa = 0.75 (Default)
      t0 = 10 (Default)
      init_buffer = 75 (Default)
      term_buffer = 50 (Default)
      window = 25 (Default)
    algorithm = hmc (Default)
      hmc
        engine = nuts (Default)
          nuts
            max_depth = 10 (Default)
        metric = diag_e (Default)
        metric_file =  (Default)
        stepsize = 1 (Default)
        stepsize_jitter = 0 (Default)
    num_chains = 4
id = 1 (Default)
data
  file = examples/bernoulli/bernoulli.data.json
init = examples/bernoulli/bernoulli.init.json
random
  seed = 2780940255 (Default)
output
  file = output.csv (Default)
  diagnostic_file =  (Default)
  refresh = 100 (Default)
  sig_figs = -1 (Default)
  profile_file = profile.csv (Default)
num_threads = 6


Gradient evaluation took 2e-06 seconds
1000 transitions using 10 leapfrog steps per transition would take 0.02 seconds.
Adjust your expectations accordingly!



Gradient evaluation took 0 seconds
1000 transitions using 10 leapfrog steps per transition would take 0 seconds.
Adjust your expectations accordingly!



Gradient evaluation took 0 seconds
1000 transitions using 10 leapfrog steps per transition would take 0 seconds.
Adjust your expectations accordingly!



Gradient evaluation took 0 seconds
1000 transitions using 10 leapfrog steps per transition would take 0 seconds.
Adjust your expectations accordingly!


Chain [1] Iteration:    1 / 2000 [  0%]  (Warmup)
Chain [3] Iteration:    1 / 2000 [  0%]  (Warmup)
Chain [2] Iteration:    1 / 2000 [  0%]  (Warmup)
Chain [4] Iteration:    1 / 2000 [  0%]  (Warmup)
Chain [1] Iteration:  100 / 2000 [  5%]  (Warmup)
Chain [3] Iteration:  100 / 2000 [  5%]  (Warmup)
Chain [1] Iteration:  200 / 2000 [ 10%]  (Warmup)
Chain [4] Iteration:  100 / 2000 [  5%]  (Warmup)
Chain [2] Iteration:  100 / 2000 [  5%]  (Warmup)
Chain [3] Iteration:  200 / 2000 [ 10%]  (Warmup)
Chain [1] Iteration:  300 / 2000 [ 15%]  (Warmup)
Chain [2] Iteration:  200 / 2000 [ 10%]  (Warmup)
Chain [4] Iteration:  200 / 2000 [ 10%]  (Warmup)
Chain [3] Iteration:  300 / 2000 [ 15%]  (Warmup)
Chain [1] Iteration:  400 / 2000 [ 20%]  (Warmup)
Chain [3] Iteration:  400 / 2000 [ 20%]  (Warmup)
Chain [2] Iteration:  300 / 2000 [ 15%]  (Warmup)
Chain [4] Iteration:  300 / 2000 [ 15%]  (Warmup)
Chain [1] Iteration:  500 / 2000 [ 25%]  (Warmup)
Chain [3] Iteration:  500 / 2000 [ 25%]  (Warmup)
Chain [2] Iteration:  400 / 2000 [ 20%]  (Warmup)
Chain [1] Iteration:  600 / 2000 [ 30%]  (Warmup)
Chain [4] Iteration:  400 / 2000 [ 20%]  (Warmup)
Chain [1] Iteration:  700 / 2000 [ 35%]  (Warmup)
Chain [3] Iteration:  600 / 2000 [ 30%]  (Warmup)
Chain [2] Iteration:  500 / 2000 [ 25%]  (Warmup)
Chain [4] Iteration:  500 / 2000 [ 25%]  (Warmup)
Chain [1] Iteration:  800 / 2000 [ 40%]  (Warmup)
Chain [3] Iteration:  700 / 2000 [ 35%]  (Warmup)
Chain [2] Iteration:  600 / 2000 [ 30%]  (Warmup)
Chain [4] Iteration:  600 / 2000 [ 30%]  (Warmup)
Chain [1] Iteration:  900 / 2000 [ 45%]  (Warmup)
Chain [3] Iteration:  800 / 2000 [ 40%]  (Warmup)
Chain [2] Iteration:  700 / 2000 [ 35%]  (Warmup)
Chain [4] Iteration:  700 / 2000 [ 35%]  (Warmup)
Chain [1] Iteration: 1000 / 2000 [ 50%]  (Warmup)
Chain [1] Iteration: 1001 / 2000 [ 50%]  (Sampling)
Chain [3] Iteration:  900 / 2000 [ 45%]  (Warmup)
Chain [2] Iteration:  800 / 2000 [ 40%]  (Warmup)
Chain [4] Iteration:  800 / 2000 [ 40%]  (Warmup)
Chain [3] Iteration: 1000 / 2000 [ 50%]  (Warmup)
Chain [2] Iteration:  900 / 2000 [ 45%]  (Warmup)
Chain [3] Iteration: 1001 / 2000 [ 50%]  (Sampling)
Chain [4] Iteration:  900 / 2000 [ 45%]  (Warmup)
Chain [2] Iteration: 1000 / 2000 [ 50%]  (Warmup)
Chain [2] Iteration: 1001 / 2000 [ 50%]  (Sampling)
Chain [4] Iteration: 1000 / 2000 [ 50%]  (Warmup)
Chain [4] Iteration: 1001 / 2000 [ 50%]  (Sampling)
Chain [1] Iteration: 1100 / 2000 [ 55%]  (Sampling)
Chain [3] Iteration: 1100 / 2000 [ 55%]  (Sampling)
Chain [2] Iteration: 1100 / 2000 [ 55%]  (Sampling)
Chain [4] Iteration: 1100 / 2000 [ 55%]  (Sampling)
Chain [1] Iteration: 1200 / 2000 [ 60%]  (Sampling)
Chain [3] Iteration: 1200 / 2000 [ 60%]  (Sampling)
Chain [4] Iteration: 1200 / 2000 [ 60%]  (Sampling)
Chain [1] Iteration: 1300 / 2000 [ 65%]  (Sampling)
Chain [2] Iteration: 1200 / 2000 [ 60%]  (Sampling)
Chain [3] Iteration: 1300 / 2000 [ 65%]  (Sampling)
Chain [4] Iteration: 1300 / 2000 [ 65%]  (Sampling)
Chain [1] Iteration: 1400 / 2000 [ 70%]  (Sampling)
Chain [2] Iteration: 1300 / 2000 [ 65%]  (Sampling)
Chain [3] Iteration: 1400 / 2000 [ 70%]  (Sampling)
Chain [4] Iteration: 1400 / 2000 [ 70%]  (Sampling)
Chain [1] Iteration: 1500 / 2000 [ 75%]  (Sampling)
Chain [2] Iteration: 1400 / 2000 [ 70%]  (Sampling)
Chain [3] Iteration: 1500 / 2000 [ 75%]  (Sampling)
Chain [4] Iteration: 1500 / 2000 [ 75%]  (Sampling)
Chain [1] Iteration: 1600 / 2000 [ 80%]  (Sampling)
Chain [2] Iteration: 1500 / 2000 [ 75%]  (Sampling)
Chain [3] Iteration: 1600 / 2000 [ 80%]  (Sampling)
Chain [1] Iteration: 1700 / 2000 [ 85%]  (Sampling)
Chain [4] Iteration: 1600 / 2000 [ 80%]  (Sampling)
Chain [2] Iteration: 1600 / 2000 [ 80%]  (Sampling)
Chain [3] Iteration: 1700 / 2000 [ 85%]  (Sampling)
Chain [1] Iteration: 1800 / 2000 [ 90%]  (Sampling)
Chain [4] Iteration: 1700 / 2000 [ 85%]  (Sampling)
Chain [2] Iteration: 1700 / 2000 [ 85%]  (Sampling)
Chain [3] Iteration: 1800 / 2000 [ 90%]  (Sampling)
Chain [1] Iteration: 1900 / 2000 [ 95%]  (Sampling)
Chain [4] Iteration: 1800 / 2000 [ 90%]  (Sampling)
Chain [2] Iteration: 1800 / 2000 [ 90%]  (Sampling)
Chain [3] Iteration: 1900 / 2000 [ 95%]  (Sampling)
Chain [1] Iteration: 2000 / 2000 [100%]  (Sampling)

 Elapsed Time: 0.008 seconds (Warm-up)
               0.018 seconds (Sampling)
               0.026 seconds (Total)

Chain [4] Iteration: 1900 / 2000 [ 95%]  (Sampling)
Chain [2] Iteration: 1900 / 2000 [ 95%]  (Sampling)
Chain [3] Iteration: 2000 / 2000 [100%]  (Sampling)

 Elapsed Time: 0.009 seconds (Warm-up)
               0.018 seconds (Sampling)
               0.027 seconds (Total)

Chain [4] Iteration: 2000 / 2000 [100%]  (Sampling)

 Elapsed Time: 0.01 seconds (Warm-up)
               0.017 seconds (Sampling)
               0.027 seconds (Total)

Chain [2] Iteration: 2000 / 2000 [100%]  (Sampling)

 Elapsed Time: 0.009 seconds (Warm-up)
               0.018 seconds (Sampling)
               0.027 seconds (Total)

rob@Rob-M1 cmdstan %

Bob_Carpenter · February 1, 2022, 10:06pm

Hi, @Rob_J_Goedman. I believe the R extensions should still work. I’m pinging @mitzimorris who should know what’s going on.

Rob_J_Goedman · February 1, 2022, 10:28pm

Thanks Bob. It certainly still works for data.R files but no longer for init.R files.

For StanJulia I have over the weekend enabled json in Stan.jl v9.

Just for completeness (for Mitzi?), I’m running this in native (ARM) mode, not Rosetta, on an M1, but I don’t really think that is the problem. If that would help I can try it on Intel.

mitzimorris · February 1, 2022, 11:01pm

both CmdStan and CmdStanPy allow rdump format for data, inits, and initial metric values.

here’s file bernoulli_init_3.R:

theta <-  0.5

from command line:

./bernoulli sample  init=bernoulli_init_3.R  data file=bernoulli.data.json

from cmdstanpy:

stan_file = os.path.join( 'bernoulli.stan')
data_file = os.path.join('bernoulli.data.json')
model = CmdStanModel(stan_file=stan_file)
model.sample(data=data_file,
             inits=os.path.join(ex_path, 'bernoulli_init_3.R'))

are you using CmdStanR?

Rob_J_Goedman · February 1, 2022, 11:34pm

I use vanilla cmdstan.

In the 2nd post post I completed the cmdstan build proces and did:

rob@Rob-M1 cmdstan % ./examples/bernoulli/bernoulli num_threads=6 sample num_chains=4 data file=examples/bernoulli/bernoulli.data.json init=examples/bernoulli/bernoulli.init.R

It works fine with:
rob@Rob-M1 cmdstan % ./examples/bernoulli/bernoulli sample data file=examples/bernoulli/bernoulli.data.json init=examples/bernoulli/bernoulli.init.R

or with

rob@Rob-M1 cmdstan % ./examples/bernoulli/bernoulli num_threads=6 sample num_chains=4 data file=examples/bernoulli/bernoulli.data.json init=examples/bernoulli/bernoulli.init.json

mitzimorris · February 2, 2022, 12:02am

sorry, don’t understand - is this working?

Rob_J_Goedman · February 2, 2022, 12:10am

Yes, the last 2 commands are working fine, it’s just if the cmdline activates threads and multiple chains by inserting e.g. “… num_threads=6 sample num_chains=4 …” that it fails (e.g. as in the first example command above) and produces the error message…

mitzimorris · February 2, 2022, 12:46am

This error is triggered by use of num_chains - when this was added the logic was changed, and the change was not intended - I think what happened is that for some reason a check for suffix .csv was added when the check should be for suffix .R.

github.com

stan-dev/cmdstan/blob/166712876de05c8548af05bb7acbedd0410735e0/src/cmdstan/command.hpp#L129

      
        
            context_vector get_vec_var_context(const std::string &file, size_t num_chains) {
              using stan::io::var_context;
              if (num_chains == 1) {
                return context_vector(1, get_var_context(file));
              }
              auto make_context = [](auto &&file, auto &&stream,
                                     auto &&file_ending) -> shared_context_ptr {
                if (file_ending == ".json") {
                  using cmdstan::json::json_data;
                  return std::make_shared<json_data>(json_data(stream));
                } else if (file_ending == ".csv") {
                  using stan::io::dump;
                  return std::make_shared<stan::io::dump>(dump(stream));
                } else {
                  std::stringstream msg;
                  msg << "file ending of " << file_ending << " is not supported by cmdstan";
                  throw std::invalid_argument(msg.str());
                  using stan::io::dump;
                  return std::make_shared<dump>(dump(stream));
                }
              };

filed issue CmdStan checks for file suffix `.csv` but should be checking for suffix `.R` · Issue #1076 · stan-dev/cmdstan · GitHub
sorry this wasn’t caught sooner, else we could have gotten it into upcoming release.

Rob_J_Goedman · February 2, 2022, 12:54am

Thanks for your help Mitzi, I do have a work around (using json) so if it is not in the upcoming release that is ok. But I’m glad you found what caused it!

Topic		Replies	Views
Bug with init argument, json file and num_chains>1, cmdstan 2.28.1 CmdStan	5	443	November 15, 2021
Chains finish unexpectedly in new install of CmdStanR CmdStan cmdstanr	8	1561	April 25, 2023
Cmdstanr unable to sample - chains are always finishing unexpectedly in Linux Interfaces cmdstanr	0	401	April 25, 2023
Cmdstanr backend ends R process in Emacs but not terminal brms cmdstanr	3	402	September 21, 2023
CmdStan fails reading .R data on Linux General cmdstan , linux	4	401	November 26, 2020

File ending of .R is not supported by cmdstan

Related Topics