Error during diagnose with CmdStanpy: Assertion `index >= 0 && index < size()' failed

General
1

I met a problem when I tried to conduct diagnose after sampling using CmdStanpy. The error is:

RuntimeError: ERROR
diagnose: stan/lib/stan_math/lib/eigen_3.3.9/Eigen/src/Core/DenseCoeffsBase.h:425: Eigen::DenseCoeffsBase<Derived, 1>::Scalar& Eigen::DenseCoeffsBase<Derived, 1>::operator()(Eigen::Index) [with Derived = Eigen::Matrix<double, -1, 1>; Eigen::DenseCoeffsBase<Derived, 1>::Scalar = double; Eigen::Index = long int]: Assertion `index >= 0 && index < size()’ failed.

import os

import time

start_time = time.time()
fit = model.sample(data=lg_data_stan, inits=0.1, chains=4, max_treedepth=10, adapt_delta=0.9,
        iter_warmup=1000, save_warmup=True, iter_sampling=1000, thin=2, show_progress=True, output_dir= path + '/output/', 
        save_diagnostics=True)
elapsed_time = time.time() - start_time
fit.diagnose()

Does anyone have any idea about what the error is? Thanks ahead!

2

Does that work with thin=1

3

It’s an out-of-bounds indexing error. Somewhere in your code there is vec[i] such that i is zero or greater than the size of vector vec. Double-check all the vector indexes in your model and remember that while Python indexes start from 0 Stan indexing starts from 1 so they differ by one.

4

It looks like this is a failure with CmdStan’s diagnose method; not necessarily a problem with anything in your model.
did the call model.sample succeed or were there errors thrown by the sampler?
what is the result of adding print(fit) to the above code, before calling fit.diagnose?

5

Yes! model.sample works well and finished successfully.
When I run print(fit), it is normal output I think:

CmdStanMCMC: model=itc_gradient_policy chains=4[‘method=sample’, ‘num_samples=1000’, ‘num_warmup=1000’, ‘save_warmup=1’, ‘thin=2’, ‘algorithm=hmc’, ‘engine=nuts’, ‘max_depth=10’, ‘adapt’, ‘engaged=1’, ‘delta=0.9’]
csv_files:
202103032210-1.csv
202103032210-2.csv
202103032210-3.csv
202103032210-4.csv
output_files:
stdout.txt
202103032210-2-stdout.txt
202103032210-3-stdout.txt
202103032210-4-stdout.txt

6

I am not sure where such error may occur. The model is compiled and sampled successfully. But the error only occurred when I ran fit.diagnose()

7

I am getting the same error. Did anyone find a solution?

8

which version of CmdStanPy?
does fit.summary return a similar error?

how about trying this:

sampler = fit.draws().  # if this fails, the output CSV are corrupt
sampler_diagnostics = fit.method_variables().
divergences = sampler_diagnostics['divergent__']
9

Version 0.9.77. Yes, fit.summary returns a similar error. What I did try was opening the CSV file with from_csv and it worked:

from cmdstanpy import from_csv
fit = from_csv("model_output.csv")
print(fit.summary()) # This works

But if fit is the return value of the sample function, it doesn’t work, same error as OP.

10

a few more questions:

  • what is your OS?

  • what version of CmdStan?

  • did you run the sampler and then call the fit.draws() method? i.e.:

drawset = fit.draws()
drawset.shape

do you see the expected number of sampling draws, chains, parameters?

  • the file “model_output.csv” - is that the exact same CSV file that was produced by the sampler?
  • does from_csv work if you give it the entire set of CSV files produced by the sampler?
  • do all of the CSV files have the same number of lines?

finally:

  • what is you favorite color. ;-)
11

I somehow made the error go away and I can’t bring it back. I even tried to restore an old snapshot of my system from Friday, which was when I posted my first message above, but the error didn’t come back. I don’t even know how that’s possible.

For the record, I’m running Fedora 34, the output file was the CSV file produced by the sampler, I was running only one chain, so there was only one CSV file, and my favorite color is blue.

1 Like
12

gremlins. 'tis the season. glad it’s all good now.