Guide: Diagnosing a complete lack of output on Windows

Interfaces CmdStan
1

I have now encountered this bug a few times in the wild so I thought it would be worth documenting. Hopefully it can serve as a reference.

The symptoms

Occasionally, Windows users will run into an issue where a CmdStan model compiles just fine, but running it produces no output. It doesn’t make a file, it doesn’t print anything, it just immediately exits. With cmdstanr, this leads to a error like here :

Warning: Chain 1 finished unexpectedly!
...
Warning messages:
1: All chains finished unexpectedly! Use the $output(chain_id) method for more information.
2: No chains finished successfully. Unable to retrieve the fit.

Additionally, the $output() method it recommends will be completely empty.

The diagnosis

There are potentially other issues which could lead to these errors, but one I have encountered several times is due to another copy of Intel’s Threaded Building Blocks (TBB) library being installed on the machine, and the Stan model trying to dynamically link to that copy instead of the one provided by Stan-math.

Checking this is, luckily, very easy, using a built-in command called where.exe.

Running the command

where.exe tbb.dll

Will search the Windows %PATH% for instances of that file, which is exactly what the dynamic linker does. The output can tell you several things:

  1. If there are no files found, then tbb.dll is not in your path (most likely, CmdStanR/Py are adding it at runtime). The issue you are having is not the one described in this thread, but adding TBB to your path might fix whatever issue you are having.
  2. If there is 1 file found, and it lives inside your $CMDSTAN folder, the issue you are having is not the one described in this thread.
  3. Finally, if you have more than 1 file found, or 1 file found but it is not from $CMDSTAN, then you’re in the case covered by this thread. In my experience, the “other” TBB is often in C:\system32\

Fixes

Fundamentally, the way to fix this problem is to make Stan’s tbb.dll be the one which is accessed first. There are several options, depending on your level of permissions on your own machine and personal feelings:

  1. The local directory is always searched before the %PATH%, so you can copy Stan’s tbb.dll to the same folder as your model .exe and it will use that. Do this FIRST to make sure the issue goes away before considering any more extreme option.

  2. If the other TBB is on your PATH but not in system32 (which is always searched before the rest of the PATH), it’s possible to place Stan’s copy higher up on your path and then it should be picked up. You probably can do this temporarily from inside your favorite programming language that you’re using to call the model.

  3. If you know where the other tbb.dll came from, you may feel comfortable uninstalling/deleting it. NOTE: this WILL break any software which relied on the other TBB, so make sure you know what you’re doing, and possibly make a back up. If you don’t know where it came from, DO NOT DELETE IT

The explanation

CmdStan currently relies on gcc-on-Windows from the MinGW project to compile our C++ code, including the TBB library. If a user has another tbb.dll installed, it is almost assuredly compiled with the Microsoft Visual Studio compilers. This results in there being two shared libraries which are not interchangeable, since they’re compiled against different ABIs, but the dynamic linker has no way to tell them apart or pick one over the other.

Unfortunately, it seems like the dynamic linker failing in this way produces no helpful output and no errors.

4 Likes
2

@charlesm93 this is the post I mentioned during last week’s Stan meeting

1 Like
3

In my case, I only have

where.exe tbb.dll
C:\Windows\System32\tbb.dll

and I do have Visual studio installed and use an Intel CPU but this does seem to relate the the message that I get at the end of building:


NOTE: Please add C:/Users/Polly/Documents/.cmdstan/cmdstan-2.33.1/stan/lib/stan_math/lib/tbb to your PATH variable.

You may call

mingw32-make install-tbb

to automatically update your user configuration.

Is the problem you describe here also the reason why I
a) get the following message when I try to run a brms model via cmdstnr:

Before it says they are not finished, it always gives me the “In the file included message” if I am not working than this message is also red in the building process but I don’t if they are related.

Compiling Stan program...
In file included from stan/lib/stan_math/stan/math/prim/prob/von_mises_lccdf.hpp:5,
                 from stan/lib/stan_math/stan/math/prim/prob/von_mises_ccdf_log.hpp:4,
                 from stan/lib/stan_math/stan/math/prim/prob.hpp:356,
                 from stan/lib/stan_math/stan/math/prim.hpp:16,
                 from stan/lib/stan_math/stan/math/rev.hpp:14,
                 from stan/lib/stan_math/stan/math.hpp:19,
                 from stan/src/stan/model/model_header.hpp:4,
                 from C:/Users/Polly/AppData/Local/Temp/RtmpK6GjaK/model-d684e643b1f.hpp:2:
stan/lib/stan_math/stan/math/prim/prob/von_mises_cdf.hpp: In function 'stan::return_type_t<T_x, T_sigma, T_l> stan::math::von_mises_cdf(const T_x&, const T_mu&, const T_k&)':
stan/lib/stan_math/stan/math/prim/prob/von_mises_cdf.hpp:194: note: '-Wmisleading-indentation' is disabled from this point onwards, since column-tracking was disabled due to the size of the code/headers
  194 |       if (cdf_n < 0.0)
      | 

stan/lib/stan_math/stan/math/prim/prob/von_mises_cdf.hpp:194: note: adding '-flarge-source-files' will allow for more column-tracking support, at the expense of compilation time and memory

Start sampling
Error in cmdstanr::read_cmdstan_csv(out$output_files(), variables = "",  : 
  Assertion on 'files' failed: No file provided.
In addition: Warning message:
No chains finished successfully. Unable to retrieve the fit.

and b) sometimes even rstan seems to fail saying it can access some .txt file.

When using brms how do I do Fix 1? Where is my model .exe in this case?

4

That message is a warning from the C++ compiler that can be ignored.

Most likely that TBB.dll in System32 is the issue. The only thing I have known to resolve the issue is removing it (NOT recommended unless you really know why it is there/what removing it will break!) or placing Stan’s tbb.dll in the folder with your model. As for what that folder is, I’m not sure for BRMS , @andrjohns may know

5

I did further testing and installed the latest version of R and Rtools43 but the results are the same.

6

When I remove tbb.dll from the Windows folder I get this message:

> fit1 <- brm(bf(y ~ x + z), dat, family = mix,
+             prior = prior, chains = 2, backend = "cmdstanr")
Compiling Stan program...
In file included from stan/lib/stan_math/stan/math/prim/prob/von_mises_lccdf.hpp:5,
                 from stan/lib/stan_math/stan/math/prim/prob/von_mises_ccdf_log.hpp:4,
                 from stan/lib/stan_math/stan/math/prim/prob.hpp:356,
                 from stan/lib/stan_math/stan/math/prim.hpp:16,
                 from stan/lib/stan_math/stan/math/rev.hpp:14,
                 from stan/lib/stan_math/stan/math.hpp:19,
                 from stan/src/stan/model/model_header.hpp:4,
                 from C:/Users/Polly/AppData/Local/Temp/Rtmp27DLV2/model-2fbc734830e7.hpp:2:
stan/lib/stan_math/stan/math/prim/prob/von_mises_cdf.hpp: In function 'stan::return_type_t<T_x, T_sigma, T_l> stan::math::von_mises_cdf(const T_x&, const T_mu&, const T_k&)':
stan/lib/stan_math/stan/math/prim/prob/von_mises_cdf.hpp:194: note: '-Wmisleading-indentation' is disabled from this point onwards, since column-tracking was disabled due to the size of the code/headers
  194 |       if (cdf_n < 0.0)
      | 

stan/lib/stan_math/stan/math/prim/prob/von_mises_cdf.hpp:194: note: adding '-flarge-source-files' will allow for more column-tracking support, at the expense of compilation time and memory

Start sampling
Chain 2 Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:
Chain 2 Exception: normal_lpdf: Scale parameter is 0, but must be positive! (in 'C:/Users/Polly/AppData/Local/Temp/Rtmp27DLV2/model-2fbc734830e7.stan', line 70, column 6 to column 63)
Chain 2 If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,
Chain 2 but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.
Chain 2
7

Does sampling complete? That looks like a non-fatal error and may be expected during warmup

8

Interesting, it actually does but why do I get all these extra error messages that I don’t get when running it with the standard backend?

9

I think they may have different default output policies (@jonah ?)