Two way ANOVA

Modeling
1

I want to perform ANOVA, but now, I cannot come up with any idea.

Motivation
In Radiology, modality means imaging methods , MRI ,CT, PET,…etc,
and reader try to detect lesions from images taken by each modality, and resulting data contain True Positives: TP and False Positives: FP for each reader and modality. To compare modality, we use the AUC calculated for each reader and each modality, that is, AUC is indexed by two subscripts representing reader and modality.
E.g., the AUC for MRI is higher than that of CT, then we can say that MRI is better than CT. If number of modality is two, then it does not need ANOVA, I want to consider the more than 3 modality case.

Notation
Data has form y_{m,r}, where subscript indicates r-th reader and
m-th modality, respectively. And estimated characteristic \theta_{m,r}, indicating AUCs for each r-th reader and m-th modality.

Detail
I want to test the null hypothesis that

H_0: \theta_{1,r} = \theta_{2,r} = \cdots =\theta_{m,r}=\cdots=\theta_{M,r}

for all r-th reader.

Or I want to perform ANOVA with \theta_{m,r}.

My poor Guess
However, \theta_{m,r} is a two indexed family of posterior distributions,
I am not sure, but take a posterior mean of \theta_{m,r}, I get a family of deterministic real numbers as follows ;

y'_{m,r}:= \text{Posterior mean of } \theta_{m,r}

So, I want to perform the ANOVA for the data y'_{m,r}.

I am not sure how to model and how to test. I read Gelman’s book but I cannot understand.

I think it needs something like y'_{m,r} =\mu + \alpha_m + \beta_r +\epsilon_{m,r} but I am not sure. Please let me know How to modeling the two way ANOVA and how to test.

2

You probably want a histogram of [\frac{1}{R}\sum_{r=1}^{r=R}(\theta_{i,m=a} - \text{max}(\theta_{i,m\neq a})] if the same readers are used across modalities rather than any null hypothesis test. If the readers are different then average over readers and difference afterwards. I is the index for iterations so you can calculate the expected value. Depending on how your want to phrase your comparison the exact statistic you want will be different.

1 Like
3

Thank you for reply, I understand your suggesting statistics, i.e.,

\frac{1}{R}\sum_{r=1}^{r=R}(\theta_{i,m=a} - \text{max}(\theta_{i,m\neq a}) )

and I will implement it, such primitive quantities are more intuitive, and I like it.

In past, I tried to another approach based on Bayes factor.
I made two objects of class stanfit , say fitH0 and fitH1, where the first object fitH0 was built by a stan file which was written under the null hypothesis that all modalities are same, and the another object, i.e., fitH1 whose stan file was made under the alternative hypothesis. Then I run the following R scripts to compare the models (i.e., fitH0 and fitH1) based on Bayes factor;

H0 <- bridgesampling::bridge_sampler(fitH0, method = "normal", silent = TRUE)
print(H0)
 

    H1 <- bridgesampling::bridge_sampler(fitH1, method = "normal", silent = TRUE)
    print(H1)
 
    BF10 <- bridgesampling::bf(H1, H0)
    print(BF10)

But I am not sure about the followings;

  • We can write various models (stan files) under null hypothesis and its alternatives, so the model is not unique, thus I think the above R scripts are meaningless.

  • I used improper priors (i.e., whose integral is not 1), and such improper case, I am not sure whether the theory of Bayes factor is available or not.

1 Like
4

Glad to hear you’re considering it. The model comparison approaches make more sense if you’re really interested in inference but metrics like the one I suggested are a better fit with decision theory. When it comes to choosing among similar tools for an applied task I think decision theory is a more appropriate framework. The main issue is coming up with the summary that answers the right question. The one I suggested is good when the true interest is in what works “best” but you would use something different if you were interested in whether any given tool was inappropriate for the task.

1 Like