Simplex variable error during bridgesampling

Interfaces brms
, , ,
1

Hi all,

I have been stuck with this problem for a few days now and I would really appreciate anyone’s input:
I would like to compare two brm models with Bayes Factor but something goes wrong during bridesampling.

Dataset: The outcome variable is a likert scale (1-8 → RatingNum) so I opted for using cumulative models. The predictor variables include an ordinal variable with 5 levels (–> LevelN) and a continuous predictor (for the sake of simplicity I describe the reduced model below that does not include the continuous predictor). I also include random slope for the ordinal variable and random intercepts for stimuli and participants.
testdataset.csv (115.9 KB)

Problem: The dataset has 8100 observations (I’m attaching a subset of the data with 15 participants only) so I opted to use within chain parallelization with the cmdstanr backend. The model fit works fine but when I try to use bridgesampling I get this error after recompilation:
Error : Exception: stan::math::simplex_free: Simplex variable is not a valid simplex. sum(Simplex variable) = 0.9999996, but should be 1 (in ‘anon_model’, line 106, column 2 to column 25).

I tried to play around with the prior setting etc. with no luck. Then, I tried NOT using the cmdstanr backend and then the bridgesampling works. So I assume something goes wrong during recompilation?

It would be really good if I could use the within chain parallelization because fitting the model takes many hours and to use Bayes Factors in a acceptable manner, I want to run several simulations and testing
(following the recommendations of: Schad, D. J., Nicenboim, B., Bürkner, P.-C., Betancourt, M., & Vasishth, S. (2022). Workflow techniques for the robust use of bayes factors. Psychological Methods. Advance online publication. APA PsycNet)

Thank you for your input in advance!
Cheers,
Aniko

library(cmdstanr)
library(brms)
library(bridgesampling)

 formulanull = RatingNum ~ 1  + mo(LevelN) +(1+mo(LevelN)|Img) + (1+mo(LevelN) |subject)

priors_mixed.m0 <- c(
    set_prior('normal(-1.1503494, 1)', class = 'Intercept', coef = '1'),
    set_prior('normal(-0.6744898, 1)', class = 'Intercept', coef = '2'),
    set_prior('normal(-0.3186394, 1)', class = 'Intercept', coef = '3'),
    set_prior('normal(0, 1)', class = 'Intercept', coef = '4'),
    set_prior('normal(0.3186394, 1)', class = 'Intercept', coef = '5'),
    set_prior('normal(0.6744898, 1)', class = 'Intercept', coef = '6'),
    set_prior('normal(1.1503494, 1)', class = 'Intercept', coef = '7'),
    set_prior('normal(0, 1)', class = 'b'),
    set_prior('lkj(2)', class = 'cor', group = 'Img'),
    set_prior('lkj(2)', class = 'cor', group = 'subject'),
    set_prior('dirichlet(2, 2, 2, 2)', class = 'simo', coef = 'moLevelN1'),
    set_prior('exponential(.1)', class = 'sd', group = 'Img'),
    set_prior('exponential(.1)', class = 'sd', group = 'subject')) 

modelnull <- brm(formula = formulanull, 
                   family = cumulative,
                   prior = priors_mixed.m0,
                   init = 0.2,
                   chain = 4,
                   cores = 4,
                   iter = 2000, #for actual BF calculations I use 10000 with 2000 warmup 
                   warmup = 1000,
                   control=list(adapt_delta=0.99,max_treedepth=15),
                   save_pars = save_pars(all = TRUE),
                   data = subset)

bridgesampling::bridge_sampler(modelnull)
  
  model1null <- brm(formula = formulanull, 
                    family = cumulative,
                    prior = priors_mixed.m0,
                    init = 0.2,
                    chain = 4,
                    cores = 4, #set-up using  8 CPU cores
                    backend = "cmdstanr", 
                    threads = threading(2),
                    iter = 2000,
                    warmup = 1000,
                    control=list(adapt_delta=0.99,max_treedepth=15),
                    save_pars = save_pars(all = TRUE),
                    data = subset)

bridgesampling::bridge_sampler(model1null)
  • Operating System: macOS Ventura v13.2.1
  • R version 4.2.1
  • brms Version: brms_2.18.0
    *cmdstanr_0.5.3
    *bridgesampling_1.1-2
1 Like
2

This is a known problem with storing the posterior draws in CSV with 6 digits, and then using those in stand-alone generated quantities.

2 Likes
3

Hi Aki,

Thank you so much for the quick response! It was very helpful. Thankfully, brm can use the input ‘sig_figs =’. So I first set it to 8.

modelnulltest2 <- brm(formula = formulanull,
                          family = cumulative,
                          prior = priors_mixed.m0,
                          init = 0.2,
                          chain = 4,
                          cores = 4,
                          backend = "cmdstanr", 
                          threads = threading(2),
                          iter = 2000,
                          warmup = 1000,
                          control = list(adapt_delta = 0.999, max_treedepth = 15),
                          save_pars = save_pars(all = TRUE),
                          sig_figs = 8,
                          data = subset)

This was already promising because now I got a new error message:
Error : Exception: Error transforming variable simo_1: stan::io::simplex_unconstrain: Vector is not a valid simplex. sum(Vector) = 1.000000013, but should be 1 (in ‘model750d5e662de6_5fef8439bf493625f9e42f5088bc477c’ at line 106)

Based on this I have found this bug report:

&

As it says " CONSTRAINT_TOLERANCE is defined as 1E-8 ", I tried with higher setting (e.g. sig_figs = 10) and it works.

Thank you again!
Cheers,
Aniko

2 Likes