Hi everyone,
This is really a very simple question.
Let’s say I have some data y and that I fit two models:
y \sim Lognormal(\mu, \sigma) and log \ y \sim Normal(\mu, \sigma)
Given that I am technically using different data, can I use loo to compare the predictive accuracy of both models?
Thanks,
Tiago
Do you find any difference at all between the two models? If y \sim LogNormal(\mu, \sigma) then by definition of the LogNormal distribution, \log y follows a Normal distribution. Both models should be equivalent.
Having said that, I will give a short and a long answer to the question, going a little further from the example.
The sort answer is no. You cannot use the loo to compare the predictive accuracy of both models, as they are measuring this accuracy on different data.
The long answer is that you cannot do it directly, because the log likelihood of the first model is in terms of y whereas for the second one it is in terms of z=\log y. In order to compare them, you must first get the second log likelihood in terms of y too, and then compare. I have just uploaded one example on how to do this to github. It uses this example as validation because as both models are equivalent, if the conversion is done properly, they should both yield the same result, but the explanation and the maths are general and then I apply it to this example. It can be used with different examples.
You can include the necessary Jacobian in the transformation as shown in Mesquite bushes example (from Regression and other stories). The code may need the latest loo version from github.
Thank you very much to both (@OriolAbril and @avehtari).
I had the initial idea that a comparison of both models as they are, was not the approach to go, since we would be comparing the loglikelihood over different data (y and log \ y). The detailed examples you shared make it clear how to proceed in a situation like this one.
Thanks a lot again!