How to update a modified c++ stan code of a function in stan/math

#1

I have a research question that implies to add a simple parameter in a function in stan/math (wiener_lpdf), but I’m completely new to c++ and to stan outside of the usual routines. Hence I directly edited the original .hpp file according to my idea to make it easier for me.
Now my question is how do I inform Stan of the modification should I somehow re-compile the function ? Or is there anything else I should do ?

I’m running Stan on a linux machine trhough pystan.

0 Likes

#2

I’m tagging @ariddell and @ahartikainen.

It’s straightforward in CmdStan, but you’re using PyStan, so I don’t know. I don’t think you’ll need to rebuild PyStan, but you’ll need to rebuild the Stan program itself (and make sure it has access to the updated Math).

1 Like

#3

Yes, you need to recompile pystan.StanModel.

1 Like

#4

Is there a particular place where the math changes need to go? (folder location for PyStan?)

0 Likes

#5

Yes, they should be under pystan folder.

I recommend that one clones our github / extract source from pypi and install pystan “inplace”

So let’s assume you have a folder ~/github

cd github
git clone --recursive https://github.com/stan-dev/pystan
cd pystan
pip install -e .

Then stan-math files should be under ~/github/pystan/pystan/stan/lib/stan_math
(Or something similar, I’m on my phone)

2 Likes

#6

Thanks both of you for your answers.

Sorry I’m a bit lost but how do you recompile the submodule ?

0 Likes

#7

I meant this

stan_model = pystan.StanModel(model=...)
# do stuff, change .hpp etc
# "recompile"
stan_model = pystan.StanModel(model=...)

Because the header files are read at “compile” time. (Stan lang is translated to cpp --> cpp is compiled to python module --> module is imported to python)

0 Likes

#8

OK I get it but I’m not sure that this applies to my case, sorry if I was unclear about my problem or if I don’t get how your solution applies to it.

I have written a Stan model code, in this Stan code I’m calling a “core” .hpp from stan-math to compute the likelihood (wiener_lpdf.hpp or wiener_lpdf in the code below). I edited this core file and I’m now trying to update Stan so that it takes into account the modified version of the core file when I compile the written Stan model code.

functions {
   real wiener_diffusion_lpdf(real y, real mu, real bs, real ndt, real bias,
                               int dec, real min_rt, real bEs) {
	if (dec == 1) {
	  return wiener_lpdf(y | bs, ndt, bias,  drift, bEs); \\bEs is the new parameter
	} else {
	  return wiener_lpdf(y | bs, ndt, 1 - bias, - drift, bEs);
  }
0 Likes

#9

In that case, you will have to modify the function_signatures.h in Stan (https://github.com/stan-dev/stan/blob/develop/src/stan/lang/function_signatures.h).
You need to add another signature for the wiener diffusion function with the added parameter type.
Have no experience with Pystan so dont know the exact location there.

0 Likes

#10

I found it under pystan/stan/src/stan/lang

But even after adding the supplementary parameter (either changing the existing signature or adding another) :

for (size_t i = 0; i < vector_types.size(); ++i) {
  for (size_t j = 0; j < vector_types.size(); ++j) {
    for (size_t k = 0; k < vector_types.size(); ++k) {
      for (size_t l = 0; l < vector_types.size(); ++l) {
	for (size_t m = 0; m < vector_types.size(); ++m) {
  	  for (size_t n = 0; n < vector_types.size(); ++n) { \\New
	    add("wiener_lpdf", expr_type(double_type()), vector_types[i],
	        vector_types[j],vector_types[k], vector_types[l],
	        vector_types[m], vector_types[n]);
	  }
	}
      }
    }
  }
 }

The changes are not taken into account as I get an error on the number of declared parameters as before.
Should I now recompile Stan ? And if so I can I do it inside of Pystan ?

0 Likes

#11

Yes, you need to recompile Stan in that case. I am not sure how you do it in Pystan, sorry.

I know that Stan gets compiled on install, but not much more than that. Summoning @ahartikainen as the Pystan master here :)

0 Likes

#12

I think reinstalling pystan should work

In cloned pystan folder

pip install -e . --force-reinstall --ignore-installed --no-binary :all: .

If you are on windows, remove boost docs or enable long paths from windows registry

0 Likes

#13

Thanks a lot @rok_cesnovar and @ahartikainen. I was able to modify function_signatures.h and to recompile stan but it is still not compiling the Stan model.

I get the following error :

Traceback (most recent call last):
  File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/unixccompiler.py", line  118, in _compile
extra_postargs)
  File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/ccompiler.py", line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
  File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/spawn.py", line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/spawn.py", line 159, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "t0vDDMFit_SC_custom.py", line 30, in <module>
  t0v_DDM_SC = stan_utility.compile_model('DDM/t0v_DDM_customlpdf.stan', path="DDM/", model_name="t0v_DDM_SC")
  File "/home/gabriel/Stan/utilities_and_data/stan_utility.py", line 96, in compile_model
sm = pystan.StanModel(model_code=model_code)
  File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/site-packages/pystan/pystan/model.py", line 349, in __init__
build_extension.run()
  File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
  File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
  File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
  File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension
depends=ext.depends)
  File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/ccompiler.py", line 574, in compile
   self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
      File "/home/gabriel/anaconda2/envs/wiener4/lib/python3.7/distutils/unixccompiler.py", line 120, in _compile
    raise CompileError(msg)
distutils.errors.CompileError: command 'gcc' failed with exit status 1

I tried to keep only one signature for wiener_lpdf or having both signatures (with 5 or 6 arguments) but in both cases I get this error. I’m pretty sure that everything is correct in both the function_signatures.h and the wiener_lpdf.hpp files so the error must lie somewhere else.

In case it is useful I’m on Debian 9.8 and gcc = 6.3.0

0 Likes

#14

Do you need to edit e.g. https://github.com/stan-dev/math/blob/develop/stan/math/prim/scal.hpp

edit. Make sure you don’t try to read cached file

0 Likes

#15

I don’t think I need to edit it because I replace an existing function, but perhaps I’m wrong.

For the cached file I assume it shouldn’t be a problem as I create new environments in anaconda to be sure to start from a fresh install (new conda env -> git clone pystan -> edit signatures and hpp -> git install -e . -> run)

0 Likes

#16

Do you have the compilation information? Try with verbose=True or on jupyter check the terminal output. There is probably more information what failed.

btw, the earlier advice with -no-binary can be tricky. I don’t normally touch stan / stan-math so not sure if pip install -e . -U is enough to update binaries.

0 Likes

#17

Using verbose I figured out that there was a type in my .hpp file !! Thanks a lot @ahartikainen, now it is running :-)

1 Like