Bayesian neural networks

Modeling
1

I recall that EP is not very good for BNN with high number of features (high dimensionality). Can somebody provide a reference?
Also BNN may provide biased posterior as compared to Gaussian process. Can somebody provide a reference?
Maybe @avehtari could help?

Thank you

2

This is a bit off-topic as Stan doesn’t have EP or special support for neural network computation (which would be required for any efficient inference for NNs). HMC can be used for neural networks, but it requires some special software and a lot of computation time (see, .e.g, Approximate Inference in Bayesian Deep Learning).

Some of the EP issues for BNNs are discussed in

  • Pasi Jylänki, Aapo Nummenmaa and Aki Vehtari (2014). Expectation propagation for neural networks with sparsity-promoting priors. Journal of Machine Learning Research, 15(May):1849-1901. Online.

but there are later papers with some improved solutions.

BNN tends to have more challenging posterior geometry than GPs. Both deep neural networks and deep GPs have challenging posterior geometries. There is continuous improvement for integration in both cases.

There will probably be interesting results published later this year in the Approximate Inference in Bayesian Deep Learning workshop.

2 Likes