Hello. I am having trouble getting a Stan model to compile with PyStan 2.16.0. I have used the same model previously in Rstan, so I’m confident that model itself is okay. The following python code produces the compilation error for me:
from pystan import StanModel
rasch_code = """
data {
int<lower=1> I; // # questions
int<lower=1> J; // # persons
int<lower=1> N; // # observations
int<lower=1, upper=I> ii[N]; // question for n
int<lower=1, upper=J> jj[N]; // person for n
int<lower=0, upper=1> y[N]; // correctness for n
}
parameters {
vector[I] beta;
vector[J] theta;
real<lower=1> sigma;
}
model {
beta ~ normal(0, 3);
theta ~ normal(0, sigma);
sigma ~ exponential(.1);
y ~ bernoulli_logit(theta[jj] - beta[ii]);
}
"""
rasch_model = StanModel(model_code=rasch_code)
The error is:
CompileError: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\cl.exe' failed with exit status 2
For comparison, I am able to compile (and sample from) this example model without problem:
from pystan import StanModel
schools_code = """
data {
int<lower=0> J; // number of schools
real y[J]; // estimated treatment effects
real<lower=0> sigma[J]; // s.e. of effect estimates
}
parameters {
real mu;
real<lower=0> tau;
real eta[J];
}
transformed parameters {
real theta[J];
for (j in 1:J)
theta[j] = mu + tau * eta[j];
}
model {
eta ~ normal(0, 1);
y ~ normal(theta, sigma);
}
"""
schools_model = StanModel(model_code=schools_code)
I’m at a loss regarding next steps for troubleshooting, so I would be grateful for any help. Thank you.
Does some version of Python require MSVC? I thought we’d decided not to support MSVC.
If Python version X.Y requires MSVC, would you mind flagging in the doc whatever won’t work in Python version X, compiler version Y? (If it were already there, presumably someone would’ve just pointed the user there.)
On Windows, Python 3.5 onwards can only use MSVC 14+ compiler. Currently there are no compatible open source compilers due to refactor in MSVC codebase, which has created a copyright issue. Also, on Windows the Python 3.5+ is the only supported version. I do not know if 2.7 or 3.4 will even work (@ariddell) .
The problem with MSVC 14+ is that its template deduction is buggy (for example it can not deduct the Eigen::Matrix types). I have not tested all the cases where this is a problem.
What I see as a viable option is to use httpstan to call CmdStan instead of Cython bindings. Then we could use mingw compiler on Windows. But that said, I don’t have any real solution to this problem on Windows. Currently I recommend to use Docker + Linux, but even that is suboptimal.
I have a VM with Windows 10, so I could set up some tests for different distributions and functions. But what I have tested these before, the errors can be bizarre. Almost anything with Eigen::Matrix will fail.
Example. cpp multiply function works with multiply(A,x); (manually edit the stanc code) but I can not put this in to a variable (in stan code) or use it in function as a parameter.
It’s almost like MSVC precompiler goes over once and uses the resulting type to do its magic. And if these types are e.g. Eigen::Matrix or any function then it will fail.
MSVC probably could be patched, but I’m not even sure if earlier versions of MSVC compiled Stan code. Based on the Google, type deduction has been a problem long before 14 and they (MSVC devs) either don’t see this as a problem / bug or don’t know how to solve it.
PyStan isn’t compatible with Windows and there’s not likely to be a patch any time soon.
Some simple models will run, but nothing with matrix types.
The alternatives seem to be:
tell Windows users to use RStan or CmdStan or get a new OS, or
rebuild PyStan from the ground up to call Stan compilation and sampling in a separate process.
As far as putting Stan in a remote process goes, we could either try to set up some kind of server that does everything or we could build an interface that execs calls to CmdStan.
I’m confused. Things were working at some point with Python>=3.5 and
Windows MSVC 14, right? What changed exactly and when did it change? If
Eigen+MSVC>=14 is a problem, this affects many people, not just us.
I think it’s not this bad. Let’s find out what the problem is.
The documentation is fairly clear about the fact that you must use Python>=3.5 on Windows. (see https://pystan.readthedocs.io/en/latest/windows.html ). We might add a note about being required to use MSVC>=14 at the top too.
I’m sure that Python>=3.5 and MSVC 14 has worked at some point for at
least simple models like 8 schools. PyStan has been testing on Windows
using this configuration for some time. You can see a history of
successful builds and (minimal) tests: https://ci.appveyor.com/project/ariddell/pystan/history
I’m happy dropping support for Windows if we can’t find a way to make
PyStan work with updated versions of MSVC 14. Without someone who knows
their way around the Windows compiler(s), it’s challenging to fix these
problems or respond to bug reports.
Sounds good to me. So they would install all the dependencies
(especially this m2w64 toolchain) using conda and then install the
Windows wheel of PyStan using pip?
Yes. We could also make conda w32/w64 packages so everything can be installed through conda.
I’m trying to figure out if we could somehow force/flag cython to use the gcc so user does not need to edit/create distutils.cfg file. The same flag could turn of the extra arguments for MSVC.
That would be fantastic if we could get PyStan working fully on Windows. Sorry this is such a hassle—I blame it all on the poor C++ support for Windows. We’ve had huge headaches from day 1 getting things to run on Windows, even with g++ on developer machines.
It is helpful to know that the problem is a limitation of the Windows implementation. I’ll keep an eye out for whether this is solved later. Thank you!