For rstan used in other package, perhaps optionally remove or soften "This procedure has not been thoroughly tested and may be unstable or buggy."

Interfaces
1

When using rstan to build other R packages, the variational inference can have been tested for specific use cases in various papers, but sill the user (may not be a statistician or expert on stan ) reads

EXPERIMENTAL ALGORITHM:
## Chain 1:   This procedure has not been thoroughly tested and may be unstable
## Chain 1:   or buggy. The interface is subject to change.

and may get alarmed no matter how much the variational inference has been tested independently.

Could we make this message optional, or at least soften it out?

2

I think the current version has a known bug, so the buggy part is true. It’s been rare to see people actually using it, so there hasn’t been incentive to fix it (or remove it). Even if that bug would be fixed, it would be better to replace most of the algorithm with more recent proposals that have been shown to be more stable and include diagnostics. Can you tell more about your use case and can you list those papers?

3

I agree that the current algorithm is suboptimal but applicable in some cases. With this strong message, that cannot be moderated, I think we do a disservice for legitimate use cases.

A paper where I tested it for a specific application is here Probabilistic outlier identification for RNA sequencing generalized linear models - PMC

When more modern techniques such as Pathfinder will be available to rstan I will happily offer it in my packages. But for now, I think more flexibility could be given to developers of third-party apps.

4

Oh, right!

Since the submission of that paper, I have changed my mind to be more against the use of the current ADVI implementation, based on our and other’s additional experiments. Right now I think the warning is not strong enough, and it would be useful to more pressure for people to use something else, even if it would mean using something else than Stan. I think allowing hiding the problems would make things worse. This is only my personal opinion.

I don’t think there is currently anyone actively working on improving or replacing ADVI in Stan. The message comes from Stan core, and it is not controlled by any of the interfaces. I hope that someday there will be somebody that has time and skills to implement a better algorithm in Stan core, but there are so many other things that would be cool, too!

5

Noted.

Is pathfinder an alternative to ADVI, or will HMC-like be the only real option?

Users prefer to use models with wrong assumptions but faster, than better models with full Bayes but much slower. That’s the sad reality. Having a good compromise would practically do a lot of good.

6

Based on the Pathfinder paper experiments, Pathfinder often works better than the current ADVI implementation. It is not an alternative in that sense that it is not directly optimizing the same variational objective, but it is fast compared to also other variational methods. As far as we know, there are no bugs in Pathfinder.

I assume you meant “use inference with wrong assumptions but faster, than better inference with full Bayes but much slower”? I’m not objecting use of approximate inference or variational inference. I’m objecting to remove the warning about possible bugs when we do actually know there is a bug. The current code doesn’t do what the ADVI paper states. Approximate inference is fine if the implementation matches the claimed properties (and preferably there is a practical diagnostic to check how good the approximation is).

1 Like