I have a research question that implies to add a simple parameter in a function in stan/math (wiener_lpdf), but I’m completely new to c++ and to stan outside of the usual routines. Hence I directly edited the original .hpp file according to my idea to make it easier for me.
Now my question is how do I inform Stan of the modification should I somehow re-compile the function ? Or is there anything else I should do ?
I’m running Stan on a linux machine trhough pystan.
I’m tagging @ariddell and @ahartikainen.
It’s straightforward in CmdStan, but you’re using PyStan, so I don’t know. I don’t think you’ll need to rebuild PyStan, but you’ll need to rebuild the Stan program itself (and make sure it has access to the updated Math).
Yes, you need to recompile pystan.StanModel.
Is there a particular place where the math changes need to go? (folder location for PyStan?)
Yes, they should be under pystan folder.
I recommend that one clones our github / extract source from pypi and install pystan “inplace”
So let’s assume you have a folder ~/github
cd github
git clone --recursive https://github.com/stan-dev/pystan
cd pystan
pip install -e .
Then stan-math files should be under ~/github/pystan/pystan/stan/lib/stan_math
(Or something similar, I’m on my phone)
Thanks both of you for your answers.
Sorry I’m a bit lost but how do you recompile the submodule ?
I meant this
stan_model = pystan.StanModel(model=...)
# do stuff, change .hpp etc
# "recompile"
stan_model = pystan.StanModel(model=...)
Because the header files are read at “compile” time. (Stan lang is translated to cpp --> cpp is compiled to python module --> module is imported to python)
OK I get it but I’m not sure that this applies to my case, sorry if I was unclear about my problem or if I don’t get how your solution applies to it.
I have written a Stan model code, in this Stan code I’m calling a “core” .hpp from stan-math to compute the likelihood (wiener_lpdf.hpp or wiener_lpdf in the code below). I edited this core file and I’m now trying to update Stan so that it takes into account the modified version of the core file when I compile the written Stan model code.
functions {
real wiener_diffusion_lpdf(real y, real mu, real bs, real ndt, real bias,
int dec, real min_rt, real bEs) {
if (dec == 1) {
return wiener_lpdf(y | bs, ndt, bias, drift, bEs); \\bEs is the new parameter
} else {
return wiener_lpdf(y | bs, ndt, 1 - bias, - drift, bEs);
}
In that case, you will have to modify the function_signatures.h in Stan (https://github.com/stan-dev/stan/blob/develop/src/stan/lang/function_signatures.h).
You need to add another signature for the wiener diffusion function with the added parameter type.
Have no experience with Pystan so dont know the exact location there.
I found it under pystan/stan/src/stan/lang
But even after adding the supplementary parameter (either changing the existing signature or adding another) :
for (size_t i = 0; i < vector_types.size(); ++i) {
for (size_t j = 0; j < vector_types.size(); ++j) {
for (size_t k = 0; k < vector_types.size(); ++k) {
for (size_t l = 0; l < vector_types.size(); ++l) {
for (size_t m = 0; m < vector_types.size(); ++m) {
for (size_t n = 0; n < vector_types.size(); ++n) { \\New
add("wiener_lpdf", expr_type(double_type()), vector_types[i],
vector_types[j],vector_types[k], vector_types[l],
vector_types[m], vector_types[n]);
}
}
}
}
}
}
The changes are not taken into account as I get an error on the number of declared parameters as before.
Should I now recompile Stan ? And if so I can I do it inside of Pystan ?
Yes, you need to recompile Stan in that case. I am not sure how you do it in Pystan, sorry.
I know that Stan gets compiled on install, but not much more than that. Summoning @ahartikainen as the Pystan master here :)
I think reinstalling pystan should work
In cloned pystan folder
pip install -e . --force-reinstall --ignore-installed --no-binary :all: .
If you are on windows, remove boost docs or enable long paths from windows registry
Thanks a lot @rok_cesnovar and @ahartikainen. I was able to modify function_signatures.h and to recompile stan but it is still not compiling the Stan model.
I get the following error :
Traceback (most recent call last):
File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/unixccompiler.py", line 118, in _compile
extra_postargs)
File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/ccompiler.py", line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/spawn.py", line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/spawn.py", line 159, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "t0vDDMFit_SC_custom.py", line 30, in <module>
t0v_DDM_SC = stan_utility.compile_model('DDM/t0v_DDM_customlpdf.stan', path="DDM/", model_name="t0v_DDM_SC")
File "/home/gabriel/Stan/utilities_and_data/stan_utility.py", line 96, in compile_model
sm = pystan.StanModel(model_code=model_code)
File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/site-packages/pystan/pystan/model.py", line 349, in __init__
build_extension.run()
File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension
depends=ext.depends)
File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/ccompiler.py", line 574, in compile
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/unixccompiler.py", line 120, in _compile
raise CompileError(msg)
distutils.errors.CompileError: command 'gcc' failed with exit status 1
I tried to keep only one signature for wiener_lpdf or having both signatures (with 5 or 6 arguments) but in both cases I get this error. I’m pretty sure that everything is correct in both the function_signatures.h and the wiener_lpdf.hpp files so the error must lie somewhere else.
In case it is useful I’m on Debian 9.8 and gcc = 6.3.0
Do you need to edit e.g. https://github.com/stan-dev/math/blob/develop/stan/math/prim/scal.hpp
edit. Make sure you don’t try to read cached file
I don’t think I need to edit it because I replace an existing function, but perhaps I’m wrong.
For the cached file I assume it shouldn’t be a problem as I create new environments in anaconda to be sure to start from a fresh install (new conda env -> git clone pystan -> edit signatures and hpp -> git install -e . -> run)
Do you have the compilation information? Try with verbose=True or on jupyter check the terminal output. There is probably more information what failed.
btw, the earlier advice with -no-binary can be tricky. I don’t normally touch stan / stan-math so not sure if pip install -e . -U is enough to update binaries.
Using verbose I figured out that there was a type in my .hpp file !! Thanks a lot @ahartikainen, now it is running :-)