Is this a good good (legitimate) way to check priors?

Interfaces
1

I run the model twice with the code below, once with sample_prior = ‘only’, and the second time with sample_prior = ‘no’ to give two fits. the model is not the best, but it serves as an example.
After checking the summary, etc. for convergence, I run mcmc_intervals for all parameters(except lp_, as it sits at one extreme).
For example, I get these two plots.
For the prior:

image
image700×432 27.7 KB

and for the posterior:
image
image700×432 29.8 KB

The prior posterior cover all the range of the data posterior, so I interpret this as indicating that the priors a no biasing the posterior. BTW, I arrived at the priors by adjusting the priors by trial and error and trying to cover all parameters listed by the get_priors function.

what do you think?

zifamily = zero_inflated_poisson(link = "log", link_zi = "logit")
sp_formula = bf(count ~ 0 + Intercept + sday + (1 +site|species), zi ~ species)
zip_prior <- c(set_prior("normal(0, 5)", class = "b"),
			    set_prior("lognormal(0,1)", class = "Intercept", dpar = "zi"),
				set_prior("normal(-1, 5)", class = "b", dpar = "zi"),
				set_prior("student_t(3,0,5)", class = "sd"))
fit_prior <-
  brm(data = my_data,
      family = zifamily,
      formula = sp_formula,
      prior = zip_prior,
      iter = 2000, warmup = 1000, thin = 1, chains = 4, cores = 4,
      sample_prior = 'only'
  	)

If possible, add also code to simulate data or attach a (subset of) the dataset you work with.

Operating System: Linux Ubuntu 22.04
Interface Version: brms on RStudio
Compiler/Toolkit

2

This is evidence that you don’t have serious prior-data conflict, but is not evidence that the priors are having minimal influence on the posterior. The data are consistent with model configurations that are also consistent with the prior. What you don’t know from these plots is whether the data are also consistent with model configurations that are not consistent with the prior, these configurations being kept out of the posterior via the influence of the prior.

1 Like
3

Thanks, but I’m not sure I understand. Do you mean that these priors may not be consistent with a different model? I already assumed that I need to adjust the priors if I change the model. Maybe you could explain what you mean by “model configurations”

4

I mean estimates of the parameter vector.

5

This is a plot of the counts from the simulated data plotted with the predicted counts for each simulated “species” from the same model fit as above
image
does this address your reservations about the mcmc_intervals plots above?
I’m trying to understand how else I can check the priors. I tried add_predicted_draws with the sample prior only fit, as I did with the data fit, but it gave extreme values. Is something wrong?
I feel I’m missing something.

6

Suppose we have a simple one-parameter model, and you use a prior consistent with values between -3 and 3.

  1. Suppose the data are consistent only with values above 5. As long as the data are strong enough for the likelihood to dominate, you will get a posterior concentrated up above 5, and you will see clear prior-data conflict, because the data are forcing the model towards estimates with vanishingly small prior probability. However, assuming the prior continues to decay rapidly in its tails (rather than flattening out and being long-tailed), it will still pull the estimates hard towards the lower end of what is consistent with the likelihood. Your plots strongly suggest that you are NOT in this situation of prior-data conflict, because your prior tends to cover your posterior.
  2. Now suppose that the data are consistent with values between -6 and 6. The prior strongly regularizes the estimates, but there’s no prior-data conflict. The prior will cover the posterior precisely due to the influence of the prior on the posterior. Relax the prior, get a broader posterior. Tighten the prior, get a tighter posterior. Your plots have little to say about whether this is happening.
  3. Now suppose that the data are consistent with values between 1 and 6. Again, there’s not prior-data conflict. The prior and the likelihood will interact to constrain the estimate (very roughly speaking) to somewhere between 1 and 3. The prior still covers the posterior, and the plots that you looked at will not tell you whether this is because the prior is trimming off configurations that are consistent with the likelihood, or whether it’s because the entire likelihood is concentrated in regions consistent with the prior. Again, your plots have little to say about whether or not this is happening in the places where your posterior estimates approach the edge of your prior estimates.
  4. Suppose that the data are consistent only with values between 1 and 3. Now again you don’t have prior-data conflict, and no regions of the likelihood are getting trimmed off. However, it could still be the case that the prior sees values near 3 as much less probable than values near 1 (but still plausible). And so you could get a posterior that concentrates much more heavily towards 1 than towards 3. Your plots wouldn’t tell you if this is happening.

There are purpose-built tools for efficiently evaluating the sensitivity of your posterior to the prior, if that’s of interest, including adjustr adjustr: Stan Model Adjustments and Sensitivity Analyses using Importance Sampling • adjustr and priorsense GitHub - n-kall/priorsense: priorsense: an R package for prior diagnostics and sensitivity .

That’s probably expected–it says that your prior is consistent with some model configurations that lead to crazy data. You might consider adjusting the prior until it doesn’t display a strong tendency to yield predictions that are inconsistent with your prior domain knowledge. This is one well-trodden road towards building an informative prior.

What is your goal in checking the priors? Note that if your goal is to ensure that they have no influence on your inference, that’s not possible. Even using wildly vague priors or omitting priors altogether doesn’t get you there, because the notion of a “flat” prior depends entirely on how the model is parameterized. For a sobering exposition of this, see https://www.frontiersin.org/articles/10.3389/fevo.2019.00501/full

I think a solid piece of advice here is generally to embrace the idea that priors provide some regularization, which is almost always a good thing as long as the priors are chosen with a bit of thought, and don’t think in terms of whether or not priors “bias” inference.

If I’m understanding correctly, this is not good practice. If I’m understanding right, you relaxed the priors until they cover the posterior. So you started in scenario 1 above, and then relaxed things until you made it to scenario 2 or 3. If you don’t have domain knowledge informing the appropriate degree of prior regularization, then it’s a mistake to stop at scenario 2 or 3; you need to continue relaxing until you are comfortably in scenario 4, and hopefully in a variation where the prior appropriately reflects your prior beliefs about the relative probabilities of the different configurations consistent with the likelihood. If you DO have domain knowledge informing the appropriate degree of prior regularization, then it’s a mistake to ignore that domain knowledge just to relax the prior and get an outcome with no prior data conflict. It is appropriate to reexamine your domain knowledge and to reexamine how you encoded your domain knowledge in a prior, but you need to do so while keeping yourself in the frame of mind as if you had not yet observed your data.

5 Likes
7

Many thanks for such a comprehensive and well explained response. I feel it advances my understanding greatly. Your analytical approach will be useful in other cases as well.
I have one problem concerning conversion of domain knowledge to priors in the zero-inflated poisson mixed models. I have count values, some continuous area values (which I’ll convert to log) and discrete numbered sites and replicates.

8

Using priorsense package is a great tool to see data-prior interactions and a great help to tune the priors to weakly informative