Hi @emiruz Thanks for posting those thoughts and questions!
If we were to set out a model, and priors reflecting our knowledge, and then rigorously substantiated that and published it, and then after that we went and collected data to feed the model and calculated the posteriors, it’s easy to see how that would be different; principled; scientific. And how it may give the posteriors theoretical and physical interpretation. But when it’s all considered at once, iteratively until it all makes sense together, what separates that from any other way of tackling a modelling problem by optimising a (PPC driven in this case) objective function?
I think the challenge with your desire for probabilities to make “physical” sense is that there is no physical basis for them. Probability is not a physical thing out there in the world and much of our uncertainty is actually unrelated to sampling variability.
Laplace’s discussion of Bayes got straight to this point—he said, what if you have this coin and you think it may be asymmetrically shaped or weighted, what’s the probability of heads? Well first it is 0.5 because you don’t know which direction it might be weighted in, it could go either way. That value contradicts what you would call the physical probability and what Laplace called the “chances” of heads on a good toss. Once you start observing outcomes you’ll update that probability.
What this implies and what Jeffreys, Keynes, Cox, Jaynes argued was that probability is a logical relation between propositions. Its about making good sense of information. I think the opening pages to Cox’s Algebra of Probable Inference are some of the best (most concise) on the topic.
When Jeffreys was setting out some initial rules for coming up with a theory of probability he argued: “any rule given must be applicable in practice” and thus “must not involve an impossible experiment." When you start imagining that your data is a sample from an infinite population of possible samples and then compare it to that population of samples (which you’ve just conjured up in your mind or on your computer) you’re not really respecting the likelihood principle.
To examine whether a posterior distribution is reasonable, I think one good question ask is: does the model incorporate all of the sources of uncertainty we have about this hypothesis or these parameters? If the observations are from a survey, for example, is our uncertainty about the data being incorporated into the inference? Does the model take any uncertain quantities or decisions and plug them in, without considering alternatives?
In practice, the bootstrap doesn’t always get what you’re looking for. For instance, I’m looking at measurement error models and a popular one — SIMEX — uses a jackknife procedure, and only in limited situations can it come up with proper uncertainty about repeated sampling.