I analyzed a model using cmdstan and the output csv file (~250 Mb) has some NUL characters in it. Is this a recognized failure mode for cmdstan, i.e., an indication that cmdstan could not generate samples from the posterior distributions on that particular iteration? Or, is it telling me something else?
That seems strange to me. I don’t know of a reason an unusual character would be there (the csv is supposed to be human readable).
Do you have a model that is easy to share that reproduces this behavior?
The code takes ~35 days to run on the cluster at school, which may make duplication of the problem difficult. A cut down version of the problem with the code running half the data works fine. Two out of four chains have completed running and both have NUL characters in the csv output. Could this have been a glitch with the cluster?
Not sure. Let me get a 2nd opinion. @mitzimorris do you know if Stan ever emits nulls in the output csvs?
Stan doesn’t emit NUL.
something that runs ~35 days might have run up against either disk space, memory, or processing time limits on the cluster - if the process is terminated, no way to record the error, other than to check the process return code.
the two chains that return OK - how long did they take to run?
Sounds like this could be the case then. I haven’t (knowingly) hit this before myself with long running models and sounds like Stan doesn’t emit these things normally.
I spoke to the HPC folks about this and they said that the job did not hit any memory/time/space limits. They think it was a “noisy neighbor” problem because I was using only one core on the 25-core cpu and the job ran for a long time.
Tom
1 Like