Wasn’t quite sure where to post / ask about this, but a while ago I wrote some code so I could diagnose troublesome stan models more quickly. A colleague wanted it today so I put it into a coherent function that works just like a regular call to stan, but generates simple on the fly plots with shiny. Maybe it’s handy for others to have - if so then just tell me where to put it :) stanWplot.R (3.5 KB)
Ha, coincidentally enough I coded something similar over the past couple days. Mine is aimed at running big models on lots of cpus in parallel (I have been running several-day models on 32-core GCE machines lately). It actually re-invents the parallelism that rstan::stan will achieve automatically with its “cores” argument, but with the benefit of not locking up the R console during sampling. I also re-invented the progress reports to something a bit more sensible in my opinion. I was just about to add functions to plot diagnostics as the samples come in, so will probably be cribbing from your code!
Hey Charles, sounds cool. I’m getting some errors or warnings when I run the example though. Probably very minor but I haven’t had the time to look into it.
When it starts I get these messages:
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `Rscript.exe --slave --no-restore -e source('/var/folders/h6/14xy_35x4wd2tz542dn0qhtc0000gn/T//Rtmp6s4Q2X/stanplottemp.R') &'
then it runs the model ok but when it finishes I get
rm: /var/folders/h6/14xy_35x4wd2tz542dn0qhtc0000gn/T//Rtmp6s4Q2X/1491585202samples.csv1: No such file or directory
rm: /var/folders/h6/14xy_35x4wd2tz542dn0qhtc0000gn/T//Rtmp6s4Q2X/1491585202samples.csv2: No such file or directory
The other thing is that it doesn’t open shiny at any point when I run it.
I only tested it on windows so far, but didn’t see anything obvious that would stop it working elsewhere - which OS are you on? Shiny won’t open because it runs in the rscript command that gave a syntax error… maybe someone can see the obvious error in that line?
We’ve wanted something that does on-the-fly traceplots. Jonah Gabry’s maintaining ShinyStan, but I’m not sure that’s the right place to put this. Jonah will know what to do, but I don’t know hot ping him via this Discourse thing, so hopefull he’ll see this. If not, you could also include it as an issue for a feature request in RStan.
+1 would be neat to wrap this up for users and convince them to actually diagnose models based on the first 10-50 iterations. I recently saw a speaker who claimed he had to write his own Gibbs sampler and didn’t use Stan because he ran his model in Stan for a week (month?) and it didn’t mix well…
Not sure if this is of use to anyone else, but I thought I’d share that I worked out how to more efficiently check for updates in sample files from still-running chains (I think both Charles & I were simply re-reading the entire sample file). Here’s a gist with a basic demo, and here’s the principle applied to watching stan files (lines 24-35).