I am currently writing up an experiment where we asked people on a 0-10 scale how much they expected three different beverages, water, decaf, and coffee, to reduce their caffeine withdrawal symptoms. Initially, using Bayesian parameter estimation, I modelled this as a within-subjects regression beverage type as the categorical predictor. I got great estimates that matched the mean differences from the descriptive stats, but the posterior predictive check of this model returned draws from the joint posterior that were outside the actual range of the data.
So to remedy this, and constrain the draws from the posterior to the actual range of the data I modeled the outcome variable as an ordinal variable, using ordered logistic regression (see this post).
I wanted to get estimates of the difference in log odds between the three beverage types. To do this I simply calculated, at each step in the markov chain, the difference in estimated log odds between each of the three pairwise comparisons of the three groups (water vs decaf, water vs coffee, decaf vs coffee). The parameter estimates from this model are expressed in log odds, which I can graph as histogram/density plots to show the mean and 95% Highest Posterior Density Interval for each pairwise comparison. However log-odds are difficult to interpret even at the best of times, let alone in an ordered logistic regression. So what I did was exponentiate each chain of log-odds estimates for each pairwise comparison to get odds ratios. These are the histograms for log-odds of the difference scores and odds ratios for each pairwise comparison
Each row graphs the histogram/density plots for each pairwise comparison as (i) log-odds (left) and (ii) odds ratios (right). Each graphs also has the mean and HPDI of each markov chain for each comparison.
To my eyes the odds ratio graphs look strange, of course this doesn’t mean they’re wrong, just that I am inexperienced. I have two questions
Are the mean and the 95% HPDI the right statistics to use to summarise the distribution of odds ratios and/or log-odds? Mode or mean would make little difference for the log-odds but in the odds ratio graph the mean is considerably to the right of the the peak of the density plot, due to the heavy skew in the data. Perhaps the mode would be better?
Do these graphs look ok? I understand exponentiating log-odds transforms distributions in unintuitive ways, but I don’t have much experience in reporting distributions in these ways and just wondered if the right-hand column graphs look sensible, given the log odds distributions on their left?
Sorry for taking so long to respond.
I agree with you that log-odds are hard to understand, but odds-ratio on the natural scale are also not great as the skew makes it hard to appreciate when the difference is actually symmetric. I.e. the plot can look quite differently if you choose to make the comparison in the opposite direction. I further think showing the full distribution is often not a great use of screen space, unless you are interested in the fine details. What I’ve found useful is to show posterior intervals of odds ratio on the log scale (i.e. the labels reflect odds ratio, but the scale is log-odds). Here is how it looks like for an analysis I’ve done, the plot is a bit crazy as this shows ALL THE COMPARISONS across seven genes for eight conditions, but I think it illustrates neatly the effectiveness of the representation:
The horizontal axis is the odds ratio on the log scale. Thin lines are 95% central posterior intervals, thick lines are 50% intervals. Color indicates the widest central posterior interval excluding 0 (would probably not be necessary if you have fewer comparisons). I particularly like how 95% interval just excluding 0 and 95% interval just including 0 look almost identical and that no sort of point estimate is visible, making it harder to overinterpret the results.
Here is a less crowded version showing odds ratios for each gene against the average for each condition:
So you think that point estimates are not the right way to do it? If you had a gun to your head and the terrorist demanded a point estimate for the distribution of odds ratios, would you pick the mean or the mode? A strange kind of terrorist who was concerned with this kind of thing I admit.
I wouldn’t put is so strongly and would generally beware of anyone claiming there is a “right” way. For the case I’ve shown, I’ve found point estimates of little interest, because a) the posteriors are quite wide and b) we were mostly interested in qualitative patterns as those could be related to testing/developing theories about biological function of the individual genes. OTOH, if your posteriors are narrow and you do care about magnitudes, point estimates might sensibly take a more prominent role in your reporting. However, if I had to pick a default for a package or something, I would pick displaying intervals/densities without point estimates.
Mean vs. mode (vs. median), really depends on how the pirate will judge if my estimate is good enough to justify my continued existence. Is mean squared error of the parameter estimate important? Then mean is pretty good. Do they care about “most likely explanation”? Then mode might make sense. Is median absolute error relevant? Or is there a large premium on the sign of the estimate being correct? Then median could be best… No single clear “best” candidate without knowing what you need the estimate for.
But really those are just my thoughts so don’t put too much weight on them and make sure what you do makes sense to you :-)
Let’s look at the second row. On the log-odds scale, the mean advantage of water over coffee is -1.3 which translates to an odds ratio of .27. The mean of odds ratios is .4. Because the skew of the odds ratios pulls the mean upwards, I think working on the log odds scale is better. You could still occasionally translate those log-odds values into odds ratios in order to talk characterize effect sizes.
Thank you @tjmahr that is well put. So you think a good way would be to find the mean or mode of the distribution of log-odds and then exponentiate that to get the odds ratio?
Do you think a similar strategy for the HPDI, i.e. get the upper and lower bounds of the HPDI for the distribution of log odds and then exponentiate those values? Or do you think it’s safe to exponentiate the difference in estimated log-odds at each step in the mcmc chain and then take the HPDI of the resultant chain of odds ratios?
Thank you for your thoughtful answer @martinmodrak. It is always difficult in science to serve one’s desire to present a balanced, precise picture while simultaneously presenting the results in a way a reader can understand. I am writing up these data in an article intended for psychologists, most of whom will be non-Bayesians. I would like to try and find a way of presenting the results of my Bayesian analysis in a way that is easy to understand, and hence I have decided to give point estimates. I am also presenting HPDIs along with these point estimates, so hopefully I am walking the tightrope between keeping the Bayesians and the traditionalists engaged. Who knows if I will be successful. It is certainly a labour of love.
That’s been me too, having to briefly explain some Bayesian ideas in two paragraphs in a paper in a psych journal. At the time, I reported medians and 95% quantile “uncertainty” intervals. (I didn’t bother with HPDI intervals.) My strategy for this non-Bayesian audience was to say directly that our goal was to estimate regression effects and estimate the uncertainty about the direction and magnitude of those regression effects and that Bayes lets us accomplish these analysis goals.
