Cmdstanr slow to return results and influenced by internet connection

Interfaces
,
1

I am running the following code and experiencing a long delay after sampling completes. The sampling complete in less than 2 seconds but R does not complete for many seconds after that. The amount of time that it takes to return seems to be influence when the computer is connected to the internet.

Operating System: Windows 11
Interface Version: 2.36 installed using cmdstanr
Compiler/Toolkit: RTools 4.4

library(cmdstanr)
file <- file.path(cmdstan_path(), "examples", "bernoulli", "bernoulli.stan")
mod <- cmdstan_model(file)
system.time({
  data_list <- list(N = 10, y = c(0,1,0,0,0,0,0,0,0,1))
  
  fit <- mod$sample(
    data = data_list,
    seed = 123,
    chains = 4,
    parallel_chains = 4,
    refresh = 500
  )
})
Internet Connection Run Time
Yes 76 sec
No 6 sec

I work remotely, and my company uses Zscaler for network security and VPN. When I run the same code on my personal (non-corporate) computer, it completes in just 2–3 seconds.

This behavior is quite unusual. Is there any reason why sampling would require an internet connection? Could there be telemetry or background data being sent during execution? If so, is there a way to disable it?

Any insight into what might be happening behind the scenes would be greatly appreciated, especially to help our security team determine how best to allow necessary connections without introducing risk. Thank you!

2

Stan does not collect telemetry. Is it possible that, when connected to the internet, your computer is sending files to your company to be scanned before allowing them to be opened? I’ve never heard of such an antivirus protection, but it seems like one that could exist

3

This seems likely @WardBrian – just found https://help.zscaler.com/itdr/about-endpoint-agents

Could be the endpoint agent is sending data back to the company, or I suppose that the endpoint agent is only active when there’s an internet connection.

4

We conducted the same test using both the CmdStan and CmdStanPy interfaces and did not encounter the slow behavior observed previously. Upon examining the temporary directory that cmdstanr uses to store .csv files from the sampling process, we found that these files are written immediately after the sampling is completed.

  • 4 .csv files for the sample from the 4 chains.
  • 1 .json file starting with “file” followed by 12 random characters
  • 1 .json file starting with “standata-” followed by 12 different random characters.

Then a 7th file starting with “file” followed by 12 random characters without an extension is created. The 12 random characters change until the file finally disappears when control is return back to R after the long delay. (see the last file in the screenshot below.)

image
image629×210 3.69 KB

For the cmdstanpy interface the output to the temp directory is different with no .json files created. This interface is not affected by the ZScaler issue.

image
image635×276 13.8 KB

Question: What does the cmdstanr interface do after the writing the .csv files and the .json files that the cmdstanpy interface does not?

Upon reviewing the code for the sample method on the object returned by the cmdstan_model function, it seems that the .csv files are generated during the execution of the run_cmdstan method of the CmdStanRun object. Following this, the CmdStanRun object is passed to the constructor of the CmdStanMCMC class.

Could you provide more details on what occurs after the .csv files are written during the run_cmdstan() method? Also, what processes or transformations take place when the CmdStanRun object is used to instantiate an object of the CmdStanMCMC class?

runset <- CmdStanRun$new(args, procs)
runset$run_cmdstan()
CmdStanMCMC$new(runset)
5

@jonah is probably the right person to ask – I work more on the cmdstanpy side, and would have expected the same behavior in both cases!

6

This is a temporary file that contains information about the variables in the model. It would look something like this:

{
“inputs”: {
“N”: { “type”: “int”, “dimensions”: 0 },
“y”: { “type”: “int”, “dimensions”: 1 }
},
“parameters”: { “theta”: { “type”: “real”, “dimensions”: 0 } },
“transformed parameters”: {},
“generated quantities”: {},
“functions”: [ ],
“distributions”: \[ “bernoulli_lupmf”, “beta_lupdf” \],
“included_files”: [ ]
}

This is a temporary file that contains the data that is passed to cmdstan.

I’m not actually sure what this file is.

We use the processx package to launch multiple processes each running cmdstan. cmdstan itself (not cmdstanr) actually writes the CSVs. After cmdstan finishes we kill the processes and that’s basically it unless something was added I’m not aware of.

Not much. We read in diagnostics (e.g. divergences) from the CSVs:

I think the only time more happens than that is if you’ve compiled the model with compile_model_methods = TRUE or compile_standalone = TRUE. In those cases then there’s some extra code run to expose those to R: