I have been struggling with convergence challenges in a pretty advanced model which I have mentioned on another thread. The interest in this topic is to bring up real time tracking of a Cmdstan model from r.
I have, in many cases, noted that for a new model almost all the transitions are divergent, for one reason or another. My models can run for over an hour, even at there most optimized, and result in a failed model overall. It would be great to be able to observe the performance (ShinyStan style I suggest) in real time, or updated every few iterations. I could then kill models early, and correct them, speeding up development time.
I had thought that maybe Tensorboard might be able to be utilised, but it is beyond me to understand how at this stage.
I have been thinking about this for a few days now as I don’t think I am the only one with this issue. I thought there might be some example code out there for loading the Cmdstan log file intermittently and processing it into an object to be plotted that has a refresh button, or refreshes at a specified rate. The advantage of this would be increased productivity and learning about what models work, and which don’t, with difficult data sets.
The model development phase can be very difficult for real world data in many cases, and a tool like this, in retrospect, would save significant time in my opinion.
I assume that there is already code out there that performs this task to some extent, but I just can’t find it. Maybe someone can point me in the right direction?