ShinyStan 3.0 Alpha test

[ducoveen]

Hello Everyone,

@jonah and I have been working on an update for shinystan and we would be very happy if you could try it out and provide feedback. We replaced the entire back end for the shiny application and used modules to rebuild it so it will become easier to add / remove features in the future.

Updates shinystan application

• All plots are now based on the bayesplot package
• Warnings are displayed upon opening of the application
• All plots come with captions to guide users through the diagnostics
• Plots open on certain variables based on the worst performing parameters in terms of n_eff to help users troubleshoot their models
• Plots can be customized and exported as reports (markdown generated pdf), plots (pdf) or .rds files
• Version 2.5.0 can still be called by adding old_version = TRUE to any launch function
• Shinystan now works with objects of class stanjm, stanmvreg and blavaan

Report generation

We added a functionality to autogenerate reports for your models. This can be done for the diagnostics, the parameter estimates or both. The report can print information concerning the n worst performing parameters in terms of n_eff, specific parameters via the pars argument or all parameters in your model. The function is called generate_report and can output pdf, html and word files. One of the aims is to make it as easy as possible to autogenerate an online supplementary file for your work. Please let us know if you have any ideas on how to further improve this.

Installing

We would be very happy if you could try it out and provide us with feedback. To install the alpha test version 3.0 you can use the following r code:

if (!require("devtools")) {
install.packages("devtools")
}
devtools::install_github("stan-dev/shinystan", ref = "v3-alpha", build_vignettes = TRUE)

Providing feedback and reporting bugs

There are several ways in which you can provide feedback, report bugs or request features:

• Github Issues, please name the issue V3 - issue, report issues at: https://github.com/stan-dev/shinystan/issues
• Reply on this discourse post
• Email to d.veen@uu.nl

Hoping to get your feedback,

Best,

Johan and Duco

[stevebronder]

Nice! Some general comments

• It would be nice to be able to have some hotkey to cycle through the variables (or have some way to quickly zip through the parameters)
• I’ve personally found the mcmc_parcoord plots very useful for looking at where divergences are happening. Some additional stuff from that vignette may be nice to automate
• It would be cool if shinystan utilized the structure of brms and rstanarm a bit more. For instance a brmsfit object has the data for posterior checks already built into the model. Maybe if it’s a brmsfit object you could also have a tab for looking at conditional effects. brms can also return back draws from the priors and it would be nice to have a quick way to check priors for each variable against the computed posterior

[lauren]

I don’t think I have any new comments, but I’ll download the new version and let you know (I’ve been using the alpha for months and love it!). I shared this on Twitter so hopefully you’ll get some useful comments from there too. :)

[ducoveen]

Hey Steve,

Thanks so much for the feedback! These are some great suggestions.

1. I’ll see if this can be done with an actual hotkey or if adding another button to cycle through the variables will need to do the trick.

2. Would you like the parcoord stuff to be added to the reports? They can take quite some time to generate, so I could add an option to ignore or run them.

3. This is a great idea and a very good use case to see if adding specific modules for specific objects works out well. Do you have an example of the type of plots you would like to look at?

Best, Duco

[ducoveen]

Thanks so much Lauren!

[avehtari]

Looks great!

Some comments

• How is the xlim selected for autocorrelation plot? It seems to show too many lags. You could use Geyer’s rule (available soon in posterior https://github.com/jgabry/posterior/issues/33)
• There is check box with text “Show/Hide Caption”. If I click that, does it Show or Hide, or Show and Hide? Maybe it could have just text “Show Caption”?
• I would show captions by default, as they are quite useful for someone learning Shinystan and diagnostics
• Could change “Trace plots” to be the last in the list (on the left sidebar under “MCMC”) as they are often the least informative
• “No parameters have an Rhat value above 1.1.” the new recommendation is 1.01
• Add Bulk-ESS and Tail-ESS?
• Add local and quantile efficiency plots?
• “No parameters have an effective sample size less than 10% of the total sample size.” This reminds about the issue that not all variables are parameters. Also it would be good to write something like “…have an effective sample size for estimating the posterior mean less than…” to remind that effective sample size depends on the function, too (identity for mean).
• In area plots, add option for vertical scaling (part of bayesplot already)

[ducoveen]

Thank you for the helpful suggestions @avehtari . I have worked out a few and a few questions for others. The questions are responses in bold face.

• How is the xlim selected for autocorrelation plot? It seems to show too many lags. You could use Geyer’s rule (available soon in posterior https://github.com/jgabry/posterior/issues/33 )

  - Currently there is no selection, just a default of 20. I’m not familiar with Geyer’s rule, do you have a paper or something I could read for this?

• There is check box with text “Show/Hide Caption”. If I click that, does it Show or Hide, or Show and Hide? Maybe it could have just text “Show Caption”?

  • I changed this.

• I would show captions by default, as they are quite useful for someone learning Shinystan and diagnostics

  • I changed this.

• Could change “Trace plots” to be the last in the list (on the left sidebar under “MCMC”) as they are often the least informative

  • I changed this.

• “No parameters have an Rhat value above 1.1.” the new recommendation is 1.01

  • I changed this.

• Add Bulk-ESS and Tail-ESS?

  • Added in both the estimates and diagnose sections.

• Add local and quantile efficiency plots?
  - Are these in bayesplot package?

• “No parameters have an effective sample size less than 10% of the total sample size.” This reminds about the issue that not all variables are parameters. Also it would be good to write something like “…have an effective sample size for estimating the posterior mean less than…” to remind that effective sample size depends on the function, too (identity for mean).

  • How about: No variables have an effective sample size for estimating the posterior mean less than 10% of the total sample size., would that be better?

• In area plots, add option for vertical scaling (part of bayesplot already)

  • I’ve added the options

Hopefully you can help me out with these questions too. Thanks so much.

Best, Duco

[jtimonen]

Hi Duco! I think we met in last StanCon. Nice package. Here are my comments

• In the \widehat{R} - n_{eff} - se page the warning text sometimes freezes when I move the warning threshold slider. I cannot figure out why sometimes it updates the warning text as it should and sometimes not. Also some typos in the caption (paramters, vales)

• The “Launching ShinyStan faster” page advises to use the drop_parameters function to reduce the size of a shinystan object by removing parameters. Is this a place where one should also consider that not all variables are parameters like Aki mentioned in his comments? I would more often want to drop all the generated quantities (and I have much more of them than actual parameters).

• In the caption of the divergence information and divergent scatter plots pattern is typoed patern.

• In the “Report” page the Download plots, Download report texts are hard to read because of low contrast between text and background (if I don’t hower mouse over them).

[ducoveen]

Hello Juho,

We met at the last StanCon indeed. Thank you so much for your comments.

You were right about the \hat{R}-n_{eff}-se page, this was a coding error which I’ve fixed now.

I’ve also worked out your other comments although I’m not sure about the last one with the contrast of the download buttons. I could not replicate this error but I have edited some of parts of the css file which could have been the cause. Hopefully the contrast will be white on black and black on white now for you too.

Thanks again for your feedback,

Best, Duco

[avehtari]

@ducoveen

I have worked out a few

Cool

@ducoveen

• - Currently there is no selection, just a default of 20. I’m not familiar with Geyer’s rule, do you have a paper or something I could read for this?

Geyer (1992) Practical Markov Chain Monte Carlo, is the original reference. Additional discussion for antithetic chains and discussion of multi-chain autocorrelation is Section 3.2 of Rhatv5 and ESS paper [1903.08008] Rank-normalization, folding, and localization: An improved $\widehat{R}$ for assessing convergence of MCMC. tl;dr autocorrelaton estimates after the Geyer’s truncation lag are likely to be just noise (except in very strong antithetic case we need to use more specific statement, that is, the sum of consequent even and odd lag estimates is likely to be just noise).

@ducoveen

• Add local and quantile efficiency plots?
  - Are these in bayesplot package?

Not yet, but they are on wishlist (if only we would have more Jonahs)
Code for the plots are now at https://github.com/avehtari/rhat_ess/blob/master/code/monitorplot.R, but would require a little bit of work to make the bayesplot versions.

@ducoveen

How about: No variables have an effective sample size for estimating the posterior mean less than 10% of the total sample size. , would that be better?

To make it easier to read maybe: No variables have an effective sample size, for estimating the posterior mean, less than 10% of the total sample size.

[jtimonen]

@ducoveen

Hopefully the contrast will be white on black and black on white now for you too.

For me it is now blue on dark gray when not hovering and black on white when hovering. This both with safari and rstudio browser. But it is now much better than previously.

[ducoveen]

I’ve adapted the warning text.

@avehtari

You could use Geyer’s rule

I’ve added this by default now. The maximum is taken based on which parameters and chains are selected to inspect, with a cap at 20. It should be nicely visable in the following example:

library(shinystan)
library(rstanarm)
example("example_model")
launch_shinystan(example_model)

@avehtari

Not yet, but they are on wishlist

I’ll take a look later this week and see if I can build something for in the bayesplot package. Once they are in there I can add them to shinystan, that seems the right order.

[bgoodri]

@ducoveen

Hoping to get your feedback,

When it says on the first screen that “13 of 4000 iterations saturated the maximum tree depth” and you click on the hyperlink, it takes you to the Divergent Scatter thing rather than the Treedepth Information thing.

IMHO, having the autocorrelation plot display the zero-order autocorrelation (which is always 1) is harmful because people do not necessarily know it is 1 by construction and it messes up the scale of the vertical axis.

The Stats tab under Diagnose does not seem to actually produce any statistics. The same is true for the Stats tab under Estimate.

The pairs plot does not have a caption although it has a Show Caption checkbox.

Where did the three-dimensional plot go? That was good, although it could have been better if there were an option to show divergent transitions etc. in a different color.

Thanks!

[bgoodri]

Also, it seems that when I go to download the report, there is no option to get an RMarkdown file that can be edited. Only PDFs and .rds files.

[ducoveen]

Hello @bgoodri,

Thanks so much for your comments.

@bgoodri

The Stats tab under Diagnose does not seem to actually produce any statistics. The same is true for the Stats tab under Estimate.

What do you mean with this? @jonah and I couldn’t quite figure that out. Is it a bug and is actually nothing showing up?

@bgoodri

When it says on the first screen that “13 of 4000 iterations saturated the maximum tree depth” and you click on the hyperlink, it takes you to the Divergent Scatter thing rather than the Treedepth Information thing.

This is an unfortunate side-effect of using the modules structure. Before the Diagnose page is visited, the specific modules such as Treedepth Information arn’t actually loaded yet. Therefore the link only works up to the main diagnose tab. If you would go back to the warnings thereafter, the links would take you to the correct module. I will see if there is a work around but currently I do not see it yet.

@bgoodri

IMHO, having the autocorrelation plot display the zero-order autocorrelation (which is always 1) is harmful because people do not necessarily know it is 1 by construction and it messes up the scale of the vertical axis.

I’ve talked to @jonah and we will make a list of some of some of the things we would like to implement in bayesplot as options or as new plots so we can use them in shinystan thereafter. We will include this on the list.

@bgoodri

The pairs plot does not have a caption although it has a Show Caption checkbox.

This was a coding error which was also true for the parallel coordinates plot. Both are fixed now.

@bgoodri

Where did the three-dimensional plot go? That was good, although it could have been better if there were an option to show divergent transitions etc. in a different color.

I will bring this back as an additional option to look at in the divergent scatter tab.

@bgoodri

Also, it seems that when I go to download the report, there is no option to get an RMarkdown file that can be edited. Only PDFs and .rds files.

This is an excellent idea. I will add this!

Best, Duco

[bgoodri]

@ducoveen

@bgoodri

The Stats tab under Diagnose does not seem to actually produce any statistics. The same is true for the Stats tab under Estimate.

What do you mean with this? @jonah and I couldn’t quite figure that out. Is it a bug and is actually nothing showing up?

It looks like

Also, where did the posterior predictive checks go for objects produced by rstanarm?

[ducoveen]

Sorry for the late reply, I’ve had some other deadlines I had to meet. This does not look good! I’ll try to figure out what is going on. I’ll work on it this week. For the posterior predictive checks we are trying to figure out a nice way to get them back in. They can still be viewed using the old version but are not yet in the new version.