This would be a great project for someone to develop in Python. In R, the loo package uses the output of RStan, and has special hooks for RStan, but is otherwise pretty general I believe given the right form of MCMC output.
Right—it’s doing the equivalent of a posterior predictive calculation using full Bayes from the sample rather than a point estimate. That’s also why you’ll find