I think we could write a flowchart:
Are you really confident the posteriors/priors will be a well-known multivariate distribution?
Yes - then plug in the estimated joint (not marginals) distribution as the next prior
No - then you need some kind of non-parametric density estimation. Are there not too many parameters, i.e. dimensions in parameter space, maybe fewer than 8?
Yes - try kernel density estimation or sampling importance resampling but beware of degeneracy in the posteriors. Tuning the bandwidths is crucial and, as far as I’ve found, needs human oversight at least.
No - There is ongoing work on high-dimensional density estimation, e.g. via diffusions, normalising flows or neural networks but a common problem is that they don’t often supply smooth (to the 2nd derivative) probability densities that we can pass to Stan. You also want something scalable because you are already working in a big data problem (at least in the sense that you don’t want to fit the model to ALL the data).
I have a paper under review now and some more stuff coming that tests out these approaches. In classic hubristic fashion, I think my latest approach solves the high-dimensional problem, but it still needs more tests. Happy to share by email (journal forbids preprints) and as soon as I can, I’ll post about it here. You can take a look at GitHub - robertgrant/non-parametric-bayes-updating: Non-parametric kernel methods for updating a Bayesian model's parameters with new batches of data but it’s not terribly clear where to begin. I’ll sort that out later too.
And another note that looks relevant to @JAQuent – if you have some parameters that only appear in some batches of the data, then you should prime the pump first with a sample of data from all the batches. Otherwise you retain big differences in marginal variances and that causes headaches down the line.
2 Likes