I am running some long parallel chains on a cluster where instead of printing the screen output, it is saved to a file. I have a limited space for these files, and I am getting repeated warnings:
Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
Exception: cholesky_decompose: Matrix m is not positive definite (in 'unknown file name' at line 122)
If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine, but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.
That is the case, so the warnings are expected, the only problem being that they are creating large files for my quota instead of the brief percentage progress.
I would like to suppress these warnings, is that possible?
Does your cluster have a temporary directory with a higher quota (e.g. something called $WORKDIR)? If so, I suggest that you redirect those messages to a file in such a directory. That way, you don’t overrun your quota, and you can still examine the warnings to make sure that something hasn’t gone wrong.
Thanks for the quick reply, but it looks like that is the opposite of what I need, with refresh=0 I get only the warnings and no progress report. The PyStan description of this option makes it sound like this controls the frequency (in terms of total iterations) of the “progress bar”, so it would only be able to eliminate the progress, not the warnings. Am I missing something about it?
The cluster does have a temporary directory, but it seems that anything redirected there is lost after the job is finished. That wouldn’t be a problem, I’m not interested in the error messsages themselves, I know it’s happening, it’s a routine message, and it’s expected to some extent, but then I wouldn’t be able to monitor progress as that output would also be redirected there.
The warnings are from Stan, not Linux, so I’d think I cannot treat them as stderr and the progress statement as stdout – it’s all screen output as far as the OS is concerned – and I can’t see how to do what you are suggesting.
I was thinking the simplest way would be an option in Stan not to show any warnings, if that exists, but if there was a more elaborate way I’d be happy to try it. Thanks.
Stdout and stderr are where command-line applications write their regular output and their error messages. Stan uses them just like any other Unix application does. Redirecting stdout and stderr to files is fairly straightforward, though the way you’d do it depends on what shell you use for your job script. Do you use C shell (/bin/csh or /bin/tcsh) or some variant of Bourne shell (such as /bin/sh or /bin/bash)?
Yes, I’m usually using bash and I understand generally how input and output work on unix-based systems, but being able to suppress warnings and errors would require them to be sent to stderr, in which case it may be possible to just append 2>/dev/null to the command running stan and just discard them of redirect them to some file elsewhere. I didn’t know if that was the case or not, if mathematical/statistical issues qualified as stderr or was just something printed to screen like other details of the run.
I thought there could be an option like verbose=0,1,2 that controlled the level of output in case that gets in the way somehow. That would also be useful for interactive jobs to avoid excessive scrolling, for instance, but right now I’m just trying to solve the issue with large text output files, and this might just do. Thanks.
This way, you should be able to save your error/warning messages without using up too much of your quota.
That said, if a text file of error/warning messages is actually taking up a lot of space, then that may be a sign of problems that you should check out. Be very leery of suppressing warning.