A hunting cmdstan message "ran beyond end of program" - is it a bug

General
1

When I am using stan to model Bayesian Neural Networks and always get a message in each chain at the start of sampling.

Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
ran beyond end of program in trace()
If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.

I just ignored it as suggested. However, lately with cmdstan2.20 I started to get

ran beyond end of program in trace()

twice and it kills the sampler. After extensive experimentation I found that adding data for one more explanatory variable kills the sampler.

I wonder how to fit the full model. My guess would be that both problems are linked. Can somebody help? Many thanks.

2

Weird. I think the error is thrown here but to me it looks like that should be called at compile time, not run time. Can you share your model code?

3

Thank you. This one is rather benign - sampler just gave this only at the start of sampling:

Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
ran beyond end of program in trace()
If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.

cliff.stan (4.1 KB)
cliff.data.r (274.3 KB)

4

Okay, I was able to get this error out of it

Exception: multiply: A[1] is -nan, but must not be nan!line 8

The underlying problem seems to be that sometimes numerical errors corrupt layer_values. This is probably harmless but you should check if you can make the calculation more stable. The trace() message happens when Stan tries to report the line number where the problem was discovered but the compiler somehow didn’t include the required information in the program. Strangely the new stanc3 compiler is also unable to compile the model.

1 Like
5

Thanks. However, this data (the same data + data for one more explanatory variable) kills the sampler with:

Unrecoverable error evaluating the log probability at the initial value.
ran beyond end of program in trace()
ran beyond end of program in trace()

cliff.data.r (279.6 KB)

Can you please help me with this full data set? I already standardized X and Y. Maybe there are more tricks to improve numerical stability?

6

This time the error is

Exception: normal_lpdf: Location parameter[1] is nan, but must be finite!line 28
7

Awesome!!! What should I do to get those messages instead uninformative run beyond…?

8

Well, I just edited rethrow_located so that instead of that if-else there’s just

 o << "line " << line << std::endl;

Sorry, probably not very helpful.
The compiler generates garbled C++ so I don’t see any way of fixing it without touching Stan’s internals.

1 Like
9

Hey, uh, @seantalts Can you look into this? Stan3 compiler won’t compile this model for me, says Nyears is out of scope (even though it’s data?) and trying to reduce the code gives other nonsensical errors.

10

Thanks. It seems there was a bug on my side when generating data. I relied on stan_rdump but evidently passes nan. So the second mystery is resolved - sampler doesn’t die. Is it possible to make stan_rdump complain if data contains nan?

I still get the message that proposal is rejected because of run beyond… but it might be related to initial parameter values. I need to see if supplying inits removes the message.

Thanks for helping to find the bug in my code.

11

The error around N_years that you’re seeing has to do with the weird line endings in the file that linas provided, possibly related to working in Windows, though I had to go in and manually replace ^M with newlines and dos2unix didn’t work for me.

There is another error there though! Thanks for tagging me, adding it to the list.

12

I hope the parser works with Windows newlines as well as unix newlines. Otherwise, we’re in for a world of hurt as people use all kinds of windows-based editors.

1 Like
13

In fact stanc breaks if there are include files. I am working with Rstudio in win but compiling/running on Linux. When I copy files from win to Linux the ^M shows up and messes everything up.

14

Hey linas, are you compiling and running with CmdStan? Would you mind trying with CmdStan develop and the new stanc3 nightly? You should be able to rm bin/stanc to have it download the new one. I believe the newline stuff was sorted by https://github.com/stan-dev/stanc3/pull/341

15

No problem. Can you please give me instructions how to update local git repo (I am using cmdstan from git)? I added make/local to use openCL. Instead of bin/stanc I should call bin/stanc3?

16

You should be able to

git checkout develop
git pull origin develop
rm bin/stanc

and then use make as before to build models.

17

Do I need to do make clean-all and then make build?

18

Yes, that is advised.