Multilevel Bayesian Models of Categorical Data Annotation

Interfaces brms
1
  • Operating System: macOS Mojave 10.14.4
  • brms Version: 2.8.0

I have been reading the paper “Multilevel Bayesian Models of Categorical Data Annotation” by Bob Carpenter, and have been enjoying learning about Bayesian methods for modelling sets of categorisations by multiple raters.

In the paper, analysis of a dataset comprising of 5 rater’s classification of 3869 dental x-rays for the presence or absence of dental caries is provided. By googling around, I managed to source what I think is the dataset from the randomLCA package in R.

library(randomLCA)
#> Loading required package: lattice
data(dentistry)

Created on 2019-05-25 by the reprex package (v0.3.0)

In the paper, four modelling approaches are described:

  1. Binomial model fit (model 4.1), to estimate distributions for prevalence, sensitivity and specificity
  2. Beta-Binomial by Annotator Fit (model 4.2), to estimate prevalence, sensitivity and specificity by rater
  3. Beta-Binomial by Item Fit (model 4.3), to estimate prevalence, sensitivity and specificity by item
  4. Logistic Fit (model 4.4), to estimate prevalence, sensitivity and specificity by rater and item

I would be very grateful, for my learning, whether someone could demonstrate how models models 4.1 to 4.4 might be fit using brms.

Thanks,

2

I currently don’t have the time right now to look at all the models in detail, but I can give you some hints. First of all, binomial models are supported via the binomial family so it’s just a matter of getting the predictors right. The beta-binomial distribution is not natively supported by brms, but you can specify it via a custom family as explained in https://cran.r-project.org/web/packages/brms/vignettes/brms_customfamilies.html

3

Hi Peter,

I’m also interest in using the annotation models proposed by Bob Carpenter in my research.

Have you made any progress/found a solution (brms implementation) yet?

Thanks!

All the best,

Hauke

4

May I suggest you look into the rater package which implements variations of the Dawid-Skene model.

I am the maintainer and very happy to answer any questions.