"Global state has changed" warning

Developers
1

I have opened an issue on the GitHub repo: "Global state has changed" warning · Issue #1132 · stan-dev/rstan · GitHub

But I’ve been advised to come to Stan Discourse so I post the issue here again:

We rely on brms / rstan in the report package, and we have a workflow checking for warnings in tests. The workflow is failing due to a warning caused by brms / rstan (see easystats/report#448),

Global state has changed

Essentially, the warning seems to arise from a test simply calling the brms::brms() function as below:

model <- suppressMessages(suppressWarnings(brms::brm(
    mpg ~ qsec + wt,data = mtcars, refresh = 0, iter = 1, seed = 333)))

Generates that “Global state has changed” warning. The detailed error from the workflow is available e.g., here: fix "formal argument "rules" matched by multiple actual arguments" · easystats/report@b5bddf9 · GitHub

── Warning (test-report.brmsfit.R:3:1): report.brms ────────────────────────────
Global state has changed:
     names(before[[4]])                | names(after[[4]])                      
[94] "PIPX_HOME"                       | "PIPX_HOME"                       [94] 
[95] "PKGCACHE_HTTP_VERSION"           | "PKGCACHE_HTTP_VERSION"           [95] 
[96] "PKGLOAD_PARENT_TEMPDIR"          | "PKGLOAD_PARENT_TEMPDIR"          [96] 
                                       - "PKG_CPPFLAGS"                    [97] 
                                       - "PKG_LIBS"                        [98] 
[97] "POWERSHELL_DISTRIBUTION_CHANNEL" | "POWERSHELL_DISTRIBUTION_CHANNEL" [99] 
[98] "PWD"                             | "PWD"                             [100]
[99] "RENV_CONFIG_REPOS_OVERRIDE"      | "RENV_CONFIG_REPOS_OVERRIDE"      [101]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
      names(before[[4]])        | names(after[[4]])              
[153] "TESTTHAT"                | "TESTTHAT"                [155]
[154] "TZ"                      | "TZ"                      [156]
[155] "USER"                    | "USER"                    [157]
                                - "USE_CXX17"               [158]
[156] "VCPKG_INSTALLATION_ROOT" | "VCPKG_INSTALLATION_ROOT" [159]
[157] "XDG_CONFIG_HOME"         | "XDG_CONFIG_HOME"         [160]
[158] "XDG_RUNTIME_DIR"         | "XDG_RUNTIME_DIR"         [161]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
before[[4]][94:99] vs after[[4]][94:101]
  PIPX_BIN_DIR"/opt/pipx_bin"
  PIPX_HOME"/opt/pipx"
  PKGCACHE_HTTP_VERSION"2"
  PKGLOAD_PARENT_TEMPDIR"/tmp/RtmpQHAqXH"
+ PKG_CPPFLAGS"  -I\"/home/runner/work/_temp/Library/Rcpp/include/\"  -I\"/home/runner/work/_temp/Library/RcppEigen/include/\"  -I\"/home/runner/work/_temp/Library/RcppEigen/include/unsupported\"  -I\"/home/runner/work/_temp/Library/BH/include\" -I\"/home/runner/work/_temp/Library/StanHeaders/include/src/\"  -I\"/home/runner/work/_temp/Library/StanHeaders/include/\"  -I\"/home/runner/work/_temp/Library/RcppParallel/include/\"  -I\"/home/runner/work/_temp/Library/rstan/include\" -DEIGEN_NO_DEBUG  -DBOOST_DISABLE_ASSERTS  -DBOOST_PENDING_INTEGER_LOG2_HPP  -DSTAN_THREADS  -DUSE_STANC3 -DSTRICT_R_HEADERS  -DBOOST_PHOENIX_NO_VARIADIC_EXPRESSION  -D_HAS_AUTO_PTR_ETC=0  -include '/home/runner/work/_temp/Library/StanHeaders/include/stan/math/prim/fun/Eigen.hpp'  -D_REENTRANT -DRCPP_PARALLEL_USE_TBB=1 "
+ PKG_LIBS" '/home/runner/work/_temp/Library/rstan/lib//libStanServices.a' -L'/home/runner/work/_temp/Library/StanHeaders/lib/' -lStanHeaders -L'/home/runner/work/_temp/Library/RcppParallel/lib/' -ltbb "
  POWERSHELL_DISTRIBUTION_CHANNEL"GitHub-Actions-ubuntu22"
  PWD"/home/runner/work/report/report"
  RENV_CONFIG_REPOS_OVERRIDE"https://packagemanager.posit.co/cran/__linux__/jammy/latest"
      before[[4]]              | after[[4]]                    
[153] "true"                   | "true"                   [155]
[154] "UTC"                    | "UTC"                    [156]
[155] "runner"                 | "runner"                 [157]
                               - "1"                      [158]
[156] "/usr/local/share/vcpkg" | "/usr/local/share/vcpkg" [159]
[157] "/home/runner/.config"   | "/home/runner/.config"   [160]
[158] "/run/user/1001"         | "/run/user/1001"         [161]

The brms maintainer suggested that this might have to do with rstan and not brms. Would you have any idea as to how we can fix this on our end, or is this something that needs fixing in rstan or in some of its dependencies?

2

I looked up this error and it seems to arise when a test from testthat finds that global state has changed. I agree that it’s bad form to change global state in a test. Tracking this down is going to require finding where RStan or brms or something they call changes the global state.

In particular, this appears to arise from setting a few new flags (PKG_CPPFLAGS, PKG_LIBS and USE_CXX17). One way to deal with this would probably be to set them ahead of time. The other way to deal with it would be to write something that wraps the tests in something that removes warning messages.

Overall, testing for generic state changes like this is probably going to be brittle going forward if it’s not something that RStan or brms care about.

Another alternative might be to switch to the cmdstanr back end. Otherwise, someone needs to track through the code, figure out where those three flags get set, then wrap them safely so that they are guaranteed to be unset (I don’t know if R has exceptions and can support a try/catch/finally pattern).

1 Like
3

Thanks a lot for your reply Bob. I really appreciate that you looked into this and your conclusion is helpful and seems to confirm that the problem is not on our side. I agree the ideal scenario would be to track-down where RStan changes the global state, and change the flags back to the original state. It seems there is little we can do from our side at this point, if only to renounce running this test or accept that it will always fail for now. Is there any way to get more attention to this issue from the other developers?

4

Another alternative might be to switch to the cmdstanr back end.

To follow-up on your suggestion of switching to the cmdstanr back end Bob, I just tried this and the “Global state has changed” warning still occurs, so I’m afraid this isn’t a solution.

5

@bgoodri or @andrjohns or @Jonah would be the ones to fix this.

Why not just turn off the check for warnings?

If no-warnings-in-tests is really a hard requirement for you, you might want to look at a package other than Stan, as I doubt we’ll ever get enough cycles to keep our tests warning-free.

1 Like
6

I’m not sure about rstan (I’ve been mostly focused on cmdstanr for a while), but for cmdstanr I think we always make these sorts
of changes temporary (e.g. using withr::with_makevars when setting any flags like PKG_CPPFLAGS: cmdstanr/R/utils.R at f2e152b88fde5c2cde01ff078d5715b3b6248628 · stan-dev/cmdstanr · GitHub), so I’m surprised this is still an issue with cmdstanr. Perhaps there’s a place where we forgot to do that, but I can’t find it (but right now I only had a chance to look briefly). I would be a bit less surprised if rstan does this, but I’m not sure.

If all it takes is calling brm() then I wonder why this wouldn’t be an issue for all the other packages that have tests that depend on brms, but I don’t think we’ve seen this before (or at least I haven’t). And there are also tons of other packages that have tests calling rstan and I don’t think I’ve seen this reported for those either. Any idea why this warning would show up in your tests but not for those other packages? Or is it possibly just that you’re flagging the warning and they’re ignoring them?

1 Like
7

It’s also strange that this would suddenly start happening now since there isn’t a new version of rstan that was just released that could have changed something affecting this. Or did you only start testing for warning recently? Or maybe did testthat only recently start warning about global state changes?

1 Like
8

Thanks a lot for your replies jonah, I appreciate it a lot!

So indeed, it seems brm() always trigger that warning but is usually ignored, whereas we started implementing this workflow to check for no warnings, and this is why it was not identified earlier.

jonah to help troubleshoot the issue, is there anything you think I could do? Would it be helpful to try to isolate the source of the issue by creating, say, a fork of brms and adding the check for no warnings and see if it triggers there on Github actions?

It is already easy to replicate the issue locally from my side by simply creating a test file named test-test.R (in any package) containing the following code (the first example given in
?brms::brm()) and then clicking on the “Run Tests” in RStudio:

test_that("report-test", {
  suppressWarnings(brms::brm(
    backend = "cmdstanr",
    count ~ zBase * Trt + (1|patient),
    data = brms::epilepsy, family = poisson(),
    prior = brms::prior(normal(0, 10), class = b) +
      brms::prior(cauchy(0, 2), class = sd)
  ))
})

image
image2559×1363 387 KB

The issue is also that the warning occurs even when using suppressWarnings() around the offending function, so it’s not possible for us to suppress it. And it is not running the function per se that triggers a warning, but testthat triggers a warning when it detects the global state has changed. Then, we have a workflow that checks for overall warnings, and it picks-up the higher-level warning generated by testthat…

1 Like
9

Interesting, I don’t see this behavior when I try this. I just added your test code to one of the bayesplot test files and used RStudio’s “Run Tests” and I don’t get any warnings:

Screenshot 2024-10-12 at 1.34.14 PM
Screenshot 2024-10-12 at 1.34.14 PM1920×923 158 KB

This is using testthat version 3.2.1.1 and brms 2.22.0, which I think are the latest versions.

Actually one thing that would be really helpful is if you could test if you still get this warning when just using cmdstanr itself, not with brms. For example, you could put cmdstanr::cmdstanr_example() into a test and that will run an example model using cmdstanr. For me this doesn’t trigger any warning, but I’m also not getting the warning when running your test using brms, so it would be helpful to know what happens when you try it on your end.

10

Thanks a lot jonah, this is very useful! I have the same package version as you, and here’s the results using simply cmdstanr::cmdstanr_example().

image
image2559×1354 331 KB

(this occurs even after deleting the report package folder and cloning a clean version with usethis::create_from_github("easystats/report"))

Interestingly, indeed, I don’t get get this warning when using the same test file from another package project:

image
image2559×1006 111 KB

I also confirmed this by creating a brand new package with the same file and same result:

image
image1297×448 33.6 KB

That is puzzling isn’t it? That is however very good that we were able to locate a specifity of the issue to the report package, though I don’t know how to explain it. Any idea? We’re also getting the warning on the Github actions through this workflow, but exact warning details seem different since it’s Github servers vs my personal computer.

11

I agree, this is quite confusing! I really don’t know why this would be specific to the report package, but it does seem like it is.

One possibility is that rstan and cmdstanr are indeed changing the global state (I’m not sure where cmdstanr would be doing this since I think we only temporarily change it, but it’s possible I’m overlooking it) but the report package is just much stricter in testing for this since you have

which most packages probably don’t use in their tests.

Ideally we’d figure out where the global state is being changed and make it temporary, but if we can’t do that then the documentation for testthat::set_state_inspector suggests modifying the usage of it to ignore certain changes:

(You might discover other packages outside your control are changing the global state, in which case you might want to modify this function to ignore those values.)

So, if it’s the usage of testthat::set_state_inspector that’s picking up the state change (I’m not sure it is, but probably?) then that might be the best option at the moment.

1 Like
12

Bingo!! Fully reproducible example:

usethis::create_package("../globalstate")

usethis::use_test("test")

writeLines(
  'test_that("cmdstanr", {
  cmdstanr::cmdstanr_example()
})', con = "tests/testthat/test-test.R"
)

writeLines(
  'testthat::set_state_inspector(function() {
  envvars = Sys.getenv()
})
', con = "tests/testthat/helper-state.R"
)

image
image2559×1313 334 KB

1 Like
13

Thanks! I can reproduce this now. In this minimal example it looks like it’s just the PATH variable that is changed, but I can’t figure out where CmdStanR is doing this in a way that isn’t temporary. For example, when we compile a model we do this:

which, if withr is working properly, should just result in a temporary change. I’m not yet sure where a permanent change could be happening.

1 Like