Joint estimation of a test's sensitivity and specificity along with its predictive values

Modeling
1

In order to assess the performance parameters of a biological test, I have a sample for which I have both the results provided by the test and the results provided by a gold standard.

Sensitivity parameters are modelled quite simply using a logistic model.

Se = P(Test+|Gold+)
Sp = P(Test-|Gold-) 

brm(
  bf(test|trials(n) ~ a,
                  a ~ 0 + gold,
                  nl = T
     ), 
  data = data, family =  binomial(), 
  prior = c(
    set_prior("normal(-3, 1.5)", class = "b", coef = "gold0", nlpar = "a"),
    set_prior("normal(3, 1.5)", class = "b", coef = "gold1", nlpar = "a")
                )
    )

Se = \frac{1}{(1+e^{-gold1})}
Sp = 1-\frac{1}{(1+e^{-gold0})}

I also try to estimate the predictive values of the test based on estimates of sensitivity and specificity and the prevalence of the disease (p).

PPV = \frac{Se.p}{Se.p + (1-Sp).(1-p)}
NPV = \frac{Sp.(1-p)}{Sp.(1-p) + (1-Se).p}

At this stage, I estimate the predictive values the test by applying these formulae to a table containing a sample of the posterior sensitivity and specificity predicted values and a sample of the prevalence distribution values that I know a priori.

However, I don’t find this method very elegant and would prefer to estimate all the parameters directly during modelling.

My problem is therefore how to incorporate the prevalence into the model. Since it’s not a point value, I can’t integrate it directly into the non-linear syntax formula. Should I simply define it as a pior? But as I’m not trying to estimate this parameter I doubt that’s the right solution.
Thanks in advance for your insights.

2

From your description, it sounds like the three quantities you need to estimate are totally independent of one another. One quantity is the prevalence. The second is the probability of a positive test given a positive case. The third is the probability of a positive test given a negative case. I don’t see any advantage to modeling these things jointly.

Two exceptions:

  1. If there’s actually some causal relationships between these quantities, for example if the diagnostic thresholds get adjusted based on what’s known about the population prevalence. Then you could think about what a model for this system could look like.
  2. You might have some prior information that is expressible jointly over the parameters, for example that the probability of detection given a positive case is necessarily higher than the probability of detection given a negative case.
3

Thanks @jsocolar for your response.

You’re right that sensitivity and specificity could be estimated separately, since each of these probabilities is based on distinct individuals. However, I’m not trying to estimate prevalence. I’m simply trying to ‘inject’ the prevalence in the form of a beta distribution into the model and combine it with the sensitivity and specificity estimates in order to estimate the predictive values.
In trying to write the model with brms I realised that this would require me to call the parameters gold_0 and gold_1 in the formula. But I don’t really see how to do that.
Have a nice day.

4

Unless I am misunderstanding something, what you are doing estimating prevalence from the prior, with no likelihood. I continue to not see the value in modeling these things jointly, since they are all independent, unless you want to express a joint prior over the quantities.

5

@mitzimorris extended brms so it could handle sensitivity and specificity. She has a case study for this somewhere recreating my and Andrew Gelman’s JRSS C paper on multiple site testing, but I’m not sure where it is.

If you’re in a hierarchical setting and there are multiple test sites, then you have issues of things like setting a PCR test threshold for a Covid test—this often trades off sensitivity for specificity and thus introduces negative correlation into the population model.

This is already captured by sensitivity and specificity.

One typically estimates prevalence given the diagnostic test results and the sensitivity and specificity of the tests. You do this by having a positive test probability defined as follows, where \pi \in (0, 1) is prevalence.

\phi = \Pr[Z = 1 | \theta^\text{sens}, \theta^\text{spec}, \pi] = \pi \cdot \theta^\text{sens} + (1 - \pi) \cdot (1 - \theta^\text{spec}).

Then if you have N tests where n are positive, the likelihood is

n \sim \textrm{binomial}(N, \phi).

If you have covariates, you can model any of sensitivity, specificity, and prevalence with a logistic regression (demographic and spatio-temporal features were common covariates for modeling Covid prevalence, for example). Andrew and I modeled sensitivity and specificity across test sites with a hierarchical model (which gave us a distribution for test sites without calibration data).