Developing Services

I’m adding some functionality to services, that is, K-hat and ESS for the VI diagnostics.

Is there anything else that we’re currently implementing in one of our interfaces that we’d like to add to services?

1 Like

@roualdes is this: https://github.com/stan-dev/stan/blob/a820fb06108c0edac8606e4362ff808441691141/src/stan/analyze/mcmc/compute_effective_sample_size.hpp#L2

The same ESS/Neff we want in services? If so I’m going to skip implementing this, or just write a wrapper that calls your function.

Andre

See PR https://github.com/stan-dev/stan/pull/2794 and issues mentioned there

@drezap, I don’t feel versed well enough on VI to make a definitive judgment on this. This thread seems to have a proposal for ESS applied to ADVI as a diagnostic…

Updating the VI diagnostic began a discussion that would make diagnostics for other algorithms Stan supports available in services. I should have updated the thread.

So if we have ESS available in services, then Pystan, Rstan, Cmdstan, etc can just call stan-dev/stan’s service. This will help with maintainability, replicability, consistency in the future.

I’m looking at it as a learning opportunity. I’m familiar with simple importance sampling, but this is great because it shows one example about how it’s used in practice.

Re that same thread, isn’t this what the Analysis API was/is meant to do?

Looks like it from my skim:

But from the PR we’re implementing stuff only applicable to MCMC, where this is more general, for other algorithms.

I’m hoping someone more senior will say something?

May be we should move this ESS into services. Because we could use an ESS for MCMC and VI.

I haven’t checked Aki’s reference though, could be a different ESS.

I’ll do some reading then leave a more thoughtful comment.

Yes, different equations to compute ESS for MCMC and for IS (and vb and optimizing can be used to form the proposal for IS)