Cmdstan run error

Hi,

I have encountered the following error:
Unrecoverable error evaluating the log probability at the initial value.
ran beyond end of program in trace()
ran beyond end of program in trace()

I have cmdstan 2.20.0,
code is in cliff.stan (3.3 KB)
data is cliff.data.r (281.0 KB)

Can somebody help me to figure out what is wrong? The stan model is to fit time series data (how much canopy is left on the tree each year) but instead ar(1) model I am using neural network with 1 hidden layer with 5 nodes.

Few notes. When I decrease the dimensionality of design matrix X1… from 13 columns to 4 columns and decrease hidden layer to 1 node I get the following but it keeps sampling. BTW, this message occurs only once.

Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
ran beyond end of program in trace()
If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.

1 Like

I’m not familiar with this part of the codebase, but that error comes from here where we try to return the error associated with the line in the stan program from the C++. So it’s trying to throw an error but is not able to figure out where that error came from (I think?)

It might be an indexing error, can you try to replicate this with a simpler model?

I am not sure that I can make model simpler and replicate the error. However, if I use the model with this or smaller data set I get the error. Sometimes it kills the sampler sometimes it simply continues by providing informational message about running beyond… In fact rstan simply died so I switched to cmdstan.

Also I have encountered this error when using MPI with Bayesian Neural Nets. The code was always working but for one dataset it gave this error. I always use one node (28 shards) for MPI and when distribute data between shards add padding and never encountered problems. For this dataset I tried to switch to one shard and it started working. This particular dataset had uneven # of reservations while all previous (and I am running sucessfuly now) have even. Just a guess. It is very possible that it is indexing error but without your help I don’t think I will be able to do much. Unless there is some debugging…

I was able to find max # of columns of design matrix where sampler still runs but the message run beyond… is shown only once. It seems that this message is somewhat related to data.

Still I am getting an error below (at the beginning of sampling) and sampler doesn’t die!!!
Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
ran beyond end of program in trace()
If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.

Something is going on. I have simplified model quite a bit cliff1l.stan (1.2 KB) . Data in cliff1l.data.r (51.3 KB)

Still I receive the error - could you please help:

Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
ran beyond end of program in trace()
If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.