Difficulty adding loo criterion to longitudinal truncated normal model

I am analysing data from a clinical trial comparing placebo to active medication at multiple timepoint. One of my outcomes has a pronounced right-skew. The overwhelming majority of observations are between 0 and 0.03 but some are just below 0.5.

I was advised by someone on these forums to try a truncated normal regression. I have never run one of these before and am having trouble with the step adding the loo criterion. I would appreciate some advice on whether my difficulties are simply me not understanding how to set the right priors (for example).

Here are the data.
d.RData (10.6 KB)

And here is the model, a longitudinal truncated normal regression comparing the effect of group (placebo vs x) at each of six timepoints (week 0, 3, 6, 9, 12, and 24) with participant id nested within treatment site as the random effect.

fit_trunc <- brm(formula = outcome | trunc(lb = 0, ub = 1) ~ group*week + (1|site/id),
                 data = df,
                 family = gaussian(), # truncated family
                 prior = c(prior(normal(0.2, 1),
                                 class = Intercept),
                           prior(normal(0, 1),
                                 class = b)),
                 save_pars = save_pars(all=TRUE),
                 seed = 1234)

fit_trunc

Here is the output

# Regression Coefficients:
#               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
# Intercept        -0.39      0.10    -0.60    -0.22 1.00      841     1033
# groupx            0.04      0.07    -0.10     0.17 1.00      416      923
# week3            -0.06      0.03    -0.11    -0.01 1.00     2104     2534
# week6            -0.02      0.03    -0.07     0.03 1.00     1776     2388
# week9            -0.03      0.03    -0.09     0.02 1.00     1989     2529
# week12           -0.08      0.03    -0.13    -0.02 1.00     1908     2081
# week24           -0.07      0.05    -0.17     0.02 1.00     2400     2531
# groupx:week3     -0.04      0.04    -0.11     0.04 1.00     2108     2656
# groupx:week6     -0.13      0.04    -0.21    -0.05 1.00     1844     1951
# groupx:week9     -0.02      0.04    -0.09     0.06 1.00     1652     2372
# groupx:week12     0.04      0.04    -0.04     0.12 1.00     1540     2601
# groupx:week24    -0.03      0.06    -0.15     0.09 1.00     2033     2419

Looks as if there is a notable difference between the groups at week 6. 28 divergent transitions is a bit of a concern so some advice on modelling would be appreciated (different priors or even different family maybe) but is not the main thing I am interested in.

Now, I would like to compare this model to a Gaussian or Gamma model so I need to add the loo criterion with moment matching, with a view to performing loo-cv. However when I add the criterion.

fit_trunc_thc <- add_criterion(x = fit_trunc,
                               criterion = "loo",
                               save_psis = TRUE,
                               moment_match = TRUE)

I get an error message

Error in if (varx == 0) { : missing value where TRUE/FALSE needed

I’ve gone online to source solutions and some suggested using a single core but I still get the error. Any advice much appreciated.

Can you try using the loo package directly instead of using add_criterion? If you extract the log likelihood from the brms fit you can pass it to the loo function. I’d be curious to know if that avoids the error.

Tnank you @jonah. Do you mean, via

loo(x = fit_trunc,
     save_psis = TRUE,
     moment_match = TRUE,
     reloo = TRUE) -> loo_trunc

I tried that but got the same error. I also tried

log_lik_mat <- log_lik(fit_trunc)

loo(x = log_lik_mat,
    save_psis = TRUE,
    moment_match = TRUE,
    reloo = TRUE) -> loo_trunc

And that worked but when I tried to compare it to a non-truncated gaussian version of the same gaussian model using loo_compare it said that the two models had different outcomes. Can you suggest syntax for the fit/loo?

I think what might be happening is that you’re mixing uses of loo::loo with brms’ loo.brmsfit method, which calls loo::loo internally but also does some other stuff. Can you try loo::loo(log_lik_mat) for both models you want to compare and then pass those results to loo::loo_compare()? That said, moment matching and reloo won’t be available without a bunch of extra work because brms provides the loo package with what it needs to do moment matching and it’s trickier to set that up yourself. But for now let’s see if it will at least let you compare the models.

@avehtari @paul.buerkner I think this is an example of why it would be very useful to have a way to turn off computing r_eff in brms. I’m pretty sure the error here is coming from when r_eff is computed, which is why switching to passing the log_lik matrix directly to the loo package bypasses the error. Maybe the fact that the r_eff computation errors is telling us something useful in some cases, but it would be nice to be able to turn it off.

Ok I ran the following code comparing the truncated gaussian to the regular, using the methods you described.

# Example model truncated between 0 and 1
fit_trunc <- brm(formula = outcome | trunc(lb = 0, ub = 1) ~ group*week + (1|site/id),
                 data = df,
                 family = gaussian(), # truncated family
                 prior = c(prior(normal(0.2, 1),
                                 class = Intercept),
                           prior(normal(0, 1),
                                 class = b)),
                 save_pars = save_pars(all=TRUE),
                 seed = 1234)

# now with the log_lik technique we used above
log_lik_mat_trunc <- log_lik(fit_trunc)

loo::loo(x = log_lik_mat_trunc) -> loo_trunc_mat


# gaussian
fit_gauss <- brm(formula = outcome ~ group*week + (1|site/id),
                 data = df,
                 family = gaussian(), # truncated family
                 prior = c(prior(normal(0.2, 1),
                                 class = Intercept),
                           prior(normal(0, 1),
                                 class = b)),
                 save_pars = save_pars(all=TRUE),
                 seed = 1234)

# now with the log_lik technique we used above
log_lik_mat_gauss <- log_lik(fit_gauss)

loo::loo(x = log_lik_mat_gauss) -> loo_gauss_mat

loo_compare(loo_gauss_mat,
            loo_trunc_mat)

But I still get NA output

       elpd_diff se_diff
model1 0.0       0.0    
model2  NA        NA 

Not sure why this fails or why the truncated normal model fails on the add_criterion() step when the regular normal model works. Such matters are beyond my skills.

Which loo version you are using? At least one of this type of error has been fixed in the latest or github version, so it helps to know which version you are using

Hi @avehtari. I was running Version 2.8.0. I updated loo to latest (v2.10.0) and for good measure updated brms to v2.23.0. I reran the code above on the data I supplied and got a similar result

  model elpd_diff se_diff p_worse diag_diff       diag_elpd
 model1       0.0     0.0      NA           21 k_psis > 0.7
 model2        NA      NA      NA           40 k_psis > 0.7

Diagnostic flags present.
See ?`loo-glossary` (sections `diag_diff` and `diag_elpd`)
or https://mc-stan.org/loo/reference/loo-glossary.html.

Can you provide the probelmatic log_lik matrix, so I can investigate?

@avehtari I tried to upload but file size of .RData file is too big (25.5mb). Any other way I can get it to you?

Use any of your preferred file transfer site and send the link (e.g. in private message)

I have sent you the log-likelihood matric for the truncated normal. Thank you.

That log_lik matrix has +Inf values, and loo package silently passes them. +Inf values likely arise from the normalization term of the truncated likelihood (that is the integral of normal from 0 to 1) underflowing to 0, and then the untruncated likelihood values is divided by 0. It seems like truncated normal is not a good model, covariates have large values, or priors are to vague, so that the normal mean is smaller than 0 or larger than 1 and normal sd is so small, that the probability in the range from 0 to 1 is super small. Firstly, this is a problem with your model and secondly truncated likelihood computation in brms could be possibly made numerically better to not underflow or to provide warning in a case of underflow. Tagging @Florence_Bockting, who has recently worked on brms code related to this.

Thank you @avehtari. I won’t pretend I understood all of what you said. You mentioned a problem with my model. Can you suggest any ways it might be improved? The group x time structure is pretty set given that it’s a longitudinal cohort study, but could the priors be improved, or the random effects structure (e.g. removing site for example?)

Thanks for tagging me. I created a corresponding Issue in brms: Improve numerical stability of truncated `log_lik` · Issue #1899 · paul-buerkner/brms · GitHub and loo: Detect non-finite `log_lik` (`Inf` / `NaN`) and explicitely inform the user · Issue #386 · stan-dev/loo · GitHub.
Corresponding PRs will follow.

How I understand the current problem:

Looking at your data, I can see that the outcome is concentrated near zero. For a truncated normal on [0, 1], that often means that for some posterior draws the latent (untruncated) mean is negative while sigma is small. Visually, those draws are a narrow spike mostly to the left of 0. Truncation then keeps only the slice on [0, 1], which is a thin right-tail piece of that spike.

The truncated density renormalizes by dividing by phi(1) - phi(0). When mu is far enough below 0 relative to sigma, both phi(0) and phi(1) are essentially 1 in double precision, so phi(1) - phi(0) underflows to 0. You can try this using e.g., pnorm(0, mean = -0.3, sd = 0.03) and pnorm(1, mean = -0.3, sd = 0.03).

In the post processing, the truncated log_lik becomes +Inf (-(-Inf)) which causes the NA in loo.

Even if you try to make the priors very tight, you probably wont fix the Inf problem. So, probably the truncated normal model might not the right choice in light of the current data. A log normal might be a better choice.