Looking for examples of exact loo calculation and recombination with approximate loo for non-rstanarm stan models

I ran loo on my models and there are 5-7 observations out of ~3000 per model that have pareto k values > 0.7. I want to do an exact loo calculation for these observations and recombine with approximate loo, but my model is with rstan, not rstanarm.

In the loo documentation, it says that when you have just a few k values that are problematic, you can just do an exact loo calculation for that observation and combine them with the approximate loo calculations.

If there are a small number of problematic k values then we can use a feature in rstanarm that lets us refit the model once for each of these problematic observations. Each time the model is refit, one of the observations with a high k value is omitted and the LOO calculations are performed exactly for that observation. The results are then recombined with the approximate LOO calculations already carried out for the observations without problematic k values

There’s an example given for rstanarm models using the loo function, but it doesn’t work with a general stanfit object.

I know how to rerun my stan model while leaving out an observation, but I don’t understand how to do the exact calculation and recombination steps. Are there examples somewhere you can point me to?


I think you’ll probably want to reproduce the code from rstanarm/loo.R at master · stan-dev/rstanarm · GitHub using the information from the refits done manually for your model.

It may also be helpful to take a look at ArviZ documentation on its experimental reloo capabilities.

ArviZ is a python package that aims to provide tools for exploratory analysis of bayesian models, plots and stats and so on for pystan and cmdstanpy models but also for other libraries like pymc3 or numpyro. We are working on supporting reloo and other functions that require refits using custom classes that homogenize the sampling apis of the inference libraries. It is therefore more heavy on the user and does not work automatically for anybody (at least for now) unlike with brms or rstanarm, but aims to support all kinds of models with minimal effort on the user side. This is still completely untested and experimental but if you feel like playing, we would appreciate the feedback, ,especially about the extensibility of the current implementation

1 Like