fit$loo(moment_match=T) not returning different results from fit$loo()

Modeling
, ,
1

I’m encountering an issue using moment matching on a CmdStanFit object. I’ve followed the helpful info here to recreate a dummy model object, since I am running this on a bunch of models that I’ve fit using cmdstan from the command line.

The issue I’m having is that fit$loo(moment_match = TRUE) is not returning anything different from fit$loo(). Below is the exact code that reproduces these results, with an image of the actual results below this block.

library(cmdstanr)
library(posterior)
library(tidyverse)
library(loo)

options(mc.cores = parallel::detectCores()/2)

mod1 <- cmdstan_model("stan/bivar_trunc_gamma_asym_log.stan", force_recompile = TRUE)
fit_dummy <- mod1$sample(data = "data/independent/low_1.json",
                         iter_warmup = 1,
                         iter_sampling = 1)

start_file_path <- paste0("stan/csv_fits/", "stacking", "/", "independent", "/", "asym_log", "/")
csvfiles <- paste0(start_file_path,
                   list.files(path = start_file_path, 
                              pattern = paste0("low", "_", 1, "_\\d{1}.csv")))
fit_csv <- as_cmdstan_fit(csvfiles)
fit_dummy$.__enclos_env__$private$draws_ <- fit_csv$.__enclos_env__$private$draws_
loo1 <- fit_dummy$loo()
loo2 <- fit_dummy$loo(moment_match = TRUE)

Here’s what I get for both loo1 and loo2:

image
image659×782 68 KB

As you can see, I get the exact same results both with and without moment_match = TRUE. Do you have any ideas about what could be happening?

I’m happy to provide the actual .json file, the model code, and the 3 .csv files from this fit, if that’s helpful. But I didn’t want to overwhelm this initial post.

I’m using cmdstanr v 0.7.1, CmdStan v 2.34.1, and loo v 2.7.0.9000.

Thank you in advance!

1 Like
2

The screenshot shows rounded values. Can you check more digits of k-values for those where the k exceeded the threshold

ids <- pareto_k_ids(loo1)
pareto_k_values(loo1)[ids]
pareto_k_values(loo2)[ids]
3

Thanks for the quick reply!

I did as you suggested, but the values look identical:
image

4

I can look into this if you share the files you mentioned

5

Thank you - I really appreciate that! Here’s a Dropbox folder that contains everything.

1 Like
6

Just following up to see if you had a chance to look into this issue. I really appreciate your help!

7

We had 4 days Easter break, so I did not have time yet

8

My apologies - I had no idea. Hope you had a nice break!

9

I tested with your model and the code is working correctly. The results stay the same, as moment matching is not able to improve. Your model is a strange one with only two parameters, but N=968 transformed parameters. The moment matching is shifting the two parameters only a little, but that shift is transformed to 968 dimensional space, and there even a small change in the proposal can lead to big changes in the density ratios (“curse of the dimensionality”). The problem is likely bigger due to many betas being so close to 1 that within floating point accuracy all the posterior draws are rounded to 1 and thus the transformation of the changes from the two parameters and data is not working well.

As the four k values bigger than 0.7 were only slightly over 0.7, you could try what happens if you increase the number of posterior draws (more chains or more iterations).

1 Like
10

Thank you so much for looking into this! I really appreciate it. I may try what you suggested and increase the number of posterior draws and see what happens.

1 Like