Cmdstanpy WinError 5 Access is denied

I’ve been having a frustrating intermittent problem with cmdstanpy where I frequently get a “[WinError 5] Access is denied” when trying to call methods on my compiled model object. It works sometimes, usually when I have freshly opened a notebook (Jupyter or VSCode), but if I re-compile after making changes to my model script, I get the error essentially every time.

Example:

model   = stan.CmdStanModel(stan_file='model.stan')
MAP_fit = model.optimize(...) 

I’ve tried just about every combination of running as administrator, moving my files to a “safe” location like c:\Users\myusername, etc. I can’t seem to find a pattern. Is cmdstanpy putting a lock on the executable and then forgetting to release it? If so why would it be so intermittent?

My IT admin says he doesn’t see any activity by our security programs (that was my first thought, that our antimalware was killing the .exe).

Thanks ahead for any ideas!

Operating System: Win11 Enterprise
Interface Version: cmdstanpy 2.36
Compiler/Toolkit: RTools40

Full error dump:

13:44:35 - cmdstanpy - INFO - Chain [1] start processing


PermissionError Traceback (most recent call last)
File ~\AppData\Local\anaconda3\envs\LSTM\lib\site-packages\cmdstanpy\model.py:2061, in CmdStanModel._run_cmdstan(self, runset, idx, show_progress, show_console, progress_hook, timeout)
2060 fd_out = open(runset.stdout_files[idx], ‘w’)
→ 2061 proc = subprocess.Popen(
2062 cmd,
2063 bufsize=1,
2064 stdin=subprocess.DEVNULL,
2065 stdout=subprocess.PIPE,
2066 stderr=subprocess.STDOUT, # avoid buffer overflow
2067 env=os.environ,
2068 universal_newlines=True,
2069 )
2070 timer: Optional[threading.Timer]

File ~\AppData\Local\anaconda3\envs\LSTM\lib\subprocess.py:947, in Popen.init(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, user, group, extra_groups, encoding, errors, text, umask)
944 self.stderr = io.TextIOWrapper(self.stderr,
945 encoding=encoding, errors=errors)
→ 947 self._execute_child(args, executable, preexec_fn, close_fds,
948 pass_fds, cwd, env,
949 startupinfo, creationflags, shell,
950 p2cread, p2cwrite,
951 c2pread, c2pwrite,
952 errread, errwrite,
953 restore_signals,
954 gid, gids, uid, umask,
955 start_new_session)
956 except:
957 # Cleanup if the child failed starting.

File ~\AppData\Local\anaconda3\envs\LSTM\lib\subprocess.py:1416, in Popen._execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, unused_restore_signals, unused_gid, unused_gids, unused_uid, unused_umask, unused_start_new_session)
1415 try:
→ 1416 hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
1417 # no special security
1418 None, None,
1419 int(not close_fds),
1420 creationflags,
1421 env,
1422 cwd,
1423 startupinfo)
1424 finally:
1425 # Child is launched. Close the parent’s copy of those pipe
1426 # handles that only the child should have open. You need
(…)
1429 # pipe will not close when the child process exits and the
1430 # ReadFile will hang.

PermissionError: [WinError 5] Access is denied

The above exception was the direct cause of the following exception:

RuntimeError Traceback (most recent call last)
Cell In[21], line 19
2 init_values = {
3 ‘theta_S0’: 0.5,
4 ‘theta_r’: 0.3,
(…)
15 ‘omega_subj_r’: 0.3
16 }
18 # Run optimization
—> 19 MAP_fit = conrado_model.optimize(
20 data=stan_data_train,
21 inits=init_values,
22 algorithm = “lbfgs”,
23 iter=10000,
24 tol_grad=1e-12,
25 tol_param=1e-12,
26 tol_obj=1e-12,
27 init_alpha=1e-3,
28 refresh=50,
29 show_console=True, # equivalent to verbose=True
30 output_dir = “C:/Users/XXXX/stan_models/tmp_output”
31 )

File ~\AppData\Local\anaconda3\envs\LSTM\lib\site-packages\cmdstanpy\model.py:644, in CmdStanModel.optimize(self, data, seed, inits, output_dir, sig_figs, save_profile, algorithm, init_alpha, tol_obj, tol_rel_obj, tol_grad, tol_rel_grad, tol_param, history_size, iter, save_iterations, require_converged, show_console, refresh, time_fmt, timeout, jacobian)
642 dummy_chain_id = 0
643 runset = RunSet(args=args, chains=1, time_fmt=time_fmt)
→ 644 self._run_cmdstan(
645 runset,
646 dummy_chain_id,
647 show_console=show_console,
648 timeout=timeout,
649 )
650 runset.raise_for_timeouts()
652 if not runset._check_retcodes():

File ~\AppData\Local\anaconda3\envs\LSTM\lib\site-packages\cmdstanpy\model.py:2110, in CmdStanModel._run_cmdstan(self, runset, idx, show_progress, show_console, progress_hook, timeout)
2108 except OSError as e:
2109 msg = ‘Failed with error {}\n’.format(str(e))
→ 2110 raise RuntimeError(msg) from e
2111 finally:
2112 fd_out.close()

RuntimeError: Failed with error [WinError 5] Access is denied

We don’t do this manually, though it’s possible that win32 does when running. Usually running multiple processes from the same exe is no issue, though.

Can you describe a bit more about your set-up? Are you running this on a local folder or a network drive? Can you get the issue to happen when running non-interactively (e.g., from the command line)?

I ended up switching my whole setup to WSL to avoid having to deal with this, but for reference here is what my setup looked like:

OS: Win 11 Enterprise 24H2 Build 26100.3476
Files located under c:/Users/username (to avoid any OneDrive issues or any group permissions settings - my IT Admin suggested this)
Python/conda: Conda env with python 3.9.0 (can provide more details if needed)
IDE: VSCode 1.98.2 but issue persisted with Jupyter

I did not explore running via command line because I switched to WSL, so can’t comment on that.

Apparently some colleagues were having a similar issue with NONMEM (.exes getting locked) so it’s entirely possible this is an issue with our IT group policies or some other obscure Windows issue that is not specific to Stan or cmdstanpy. We have all switched to WSL and this has fixed our problems, so not too bothered by it.