FYI: Pyro: a new probabilistic programming language


#1

http://pyro.ai/

Pyro is a universal probabilistic programming language (PPL) written in Python and supported by PyTorch on the backend. Pyro enables flexible and expressive deep probabilistic modeling, unifying the best of modern deep learning and Bayesian modeling. It was designed with these key principles:

Blog post about it: https://eng.uber.com/pyro/

Looks like it aims to do for PyTorch what Edward does for TensorFlow.


#2

Thanks for posting. As with Edward, I don’t think Pyro is targeting the same audience as we are—it all looks much more like Python programming and much less like statistical modeling.

I had heard rumors Uber was going to open-source something—I hadn’t realized Noah Goodman was involved. He was a participant in the DARPA PPAML program, and the developer of PPL and WebPPL, which are Church-like (he was at MIT before Stanford).

Sounds like they’re doing variational inference with autodiff, but plan to do HMC. I am glad to see they went with arbitrary control structures within program (like us, but not like PyMC3 due to being tied to Theano). I can’t quite see how the statement-driven randomness of Church plays out here, but I’ve always found that confusing.

Like most of the other probabilistic programming systems (but unlike Stan), it mixes inference and modeling.

Here are some examples of programs:

http://pyro.ai/examples/

Here’s a regression example:

http://pyro.ai/examples/bayesian_regression.html

It starts with “Now let’s define our regression model in the form of a neural network.” and goes on to build something that’s a mix of an off-the-shelf named model and some priors and some computation.


#3

FYI the HMC in pyro is recently merged into dev branch, I made a small notebook comparing the HMC with the one we have in PyMC3. https://github.com/pymc-devs/pymc4_prototypes/blob/master/pytorch/pymc4_torch/HMC_in_pyro.ipynb

Currently it is still very limited, as it doesn’t sample from bounded variables since automatic transformation is not implemented yet.

Reading from the doc i would say they make an effort to separate model with inference: http://pyro.ai/examples/svi_part_i.html#Guide
where you need to build an approximation function (a Variational approximation) to do inference.


#4

Thanks for the heads up on their HMC and the clarification about separation. It’s not so much HMC that people need, but one that at least adapts scale by parameter and overall step size, as well as one that does NUTS. By itself, HMC isn’t as efficient and is very difficult to tune.