Help with using an external C++ file

bfinley · January 19, 2020, 8:36am

My partner and I are trying to utilize Stan’s autograd function in the example found at this page:

https://pystan.readthedocs.io/en/latest/external_cpp.html

We are doing this in an attempt to increase the number of arguments that can be passed in at a given time. However, we are running into several errors when we try to pass in more than five arguments to our external c++ function. The code we are trying to implement is provided below.

This is the external c++ file:

template <typename T1, typename T2, typename T3, typename T4, typename T5>
typename boost::math::tools::promote_args<T1, T2, T3, T4, T5>::type
my_other_func (const T1& A, const T2& B, const T3& C, const T4& X, const T5& Y, std::ostream* pstream) {
  typedef typename boost::math::tools::promote_args<T1, T2, T3, T4, T5>::type T;

  
  T Z = A*X*X+B*X+C+Y;

  return Z;
}

This is the PyStan code we are using, we are running PyStan 2.19.1.1:

import pystan
import os

model_code = """
functions {

    real my_other_func(real A, real B, real C, real X, real Y);
}
data {

    real X;
    real Y;
}
parameters {
    real A;
    real B;
    real C;
}
transformed parameters {

    real E = my_other_func(A, B, C, X, Y);
}
model {

    E ~ std_normal();
}
             """

include_dir = [os.path.join(".", "external_cpp")]
include_files = ["external_manual.hpp", "external_autograd.hpp"]
stan_model = pystan.StanModel(model_code=model_code,
                              verbose=True,
                              allow_undefined=True,
                              includes=include_files,
                              include_dirs=include_dir,
                              )

dict = {'X': 4,
        'Y': 1}

fit = stan_model.sampling(data=dict, iter=1000, chains=4)
print(fit)

These are some of the errors output to the terminal on Ubuntu 18.04.3. These errors occur multiple times but they all reference the same type of error. I could post the entire terminal output if necessary; however, it is exceptionally long.

/tmp/pystan_i2r5z6dp/anon_model_aa7fce6b9639a550505fc2f8eeaafc68.hpp: In member function ‘void anon_model_aa7fce6b9639a550505fc2f8eeaafc68_namespace::anon_model_aa7fce6b9639a550505fc2f8eeaafc68::write_array(RNG&, std::vector<double>&, std::vector<int>&, std::vector<double>&, bool, bool, std::ostream*) const’:
/tmp/pystan_i2r5z6dp/anon_model_aa7fce6b9639a550505fc2f8eeaafc68.hpp:335:72: error: call of overloaded ‘my_other_func(double&, double&, double&, const double&, const double&, std::ostream*&)’ is ambiguous
             stan::math::assign(E,my_other_func(A, B, C, X, Y, pstream__));
                                                                        ^
/tmp/pystan_i2r5z6dp/anon_model_aa7fce6b9639a550505fc2f8eeaafc68.hpp:231:47: error: call of overloaded ‘my_other_func(local_scalar_t__&, local_scalar_t__&, local_scalar_t__&, const double&, const double&, std::ostream*&)’ is ambiguous
             stan::math::assign(E,my_other_func(A, B, C, X, Y, pstream__));
                                  ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~

We appreciate any help or advice that can be provided! If there is any information that we failed to provide, please let us know and we will get back with you as soon as possible.

ahartikainen · January 19, 2020, 1:47pm

Hi, does this work with 4 inputs?

bfinley · January 19, 2020, 4:46pm

Yes, the issue arises whenever we try implement more than 4 inputs to function.

ahartikainen · January 19, 2020, 4:51pm

Interesting.

Does is work if you pack/unpack an array / vector?

bfinley · January 19, 2020, 6:11pm

We’ll give that a try and get back with you.

bfinley · January 21, 2020, 6:54am

While attempting to implement packing/unpacking into our solution we found that it may not be viable for our use case. We think the issue may be related to the boost library as it does not seem to allow for greater than 4 arguments to be input. Here is the specific header file we are referring to:

https://www.boost.org/doc/libs/1_69_0/boost/math/special_functions/math_fwd.hpp

For the time being, we have moved over to RStan, however, if a solution is determined please let me know.

andrjohns · January 21, 2020, 7:03am

We think the issue may be related to the boost library as it does not seem to allow for greater than 4 arguments to be input

If the problem is that promote_args doesn’t allow for more than four arguments, you could instead use stan::return_type_t, since that does the same job and works with an arbitrary number of arguments:

template <typename T1, typename T2, typename T3, typename T4, typename T5>
stan::return_type_t<T1, T2, T3, T4, T5>
my_other_func (const T1& A, const T2& B, const T3& C, const T4& X, const T5& Y, std::ostream* pstream) {
  using T = stan::return_type_t<T1, T2, T3, T4, T5>;
  
  T Z = A*X*X+B*X+C+Y;

  return Z;
}

matthew_ay · January 21, 2020, 7:37am

Hey Andrew,

@bfinley partner here. Using the PyStan file outlined in @bfinley’s initial post along with an .hpp file using the code you provided, we run into errors similar to the ones prior. However, with the exception of one error:

In file included from /var/folders/jh/ksyr7q296sldxds02llwkyv80000gn/T/pystan_jekvntlg/stanfit4anon_model_aa7fce6b9639a550505fc2f8eeaafc68_8093633944278173207.cpp:680:
In file included from /var/folders/jh/ksyr7q296sldxds02llwkyv80000gn/T/pystan_jekvntlg/anon_model_aa7fce6b9639a550505fc2f8eeaafc68.hpp:18:
./external_cpp/external_autograd.hpp:2:1: error: no template named 'return_type_t' in namespace 'stan::math'; did
      you mean 'stan::return_type'?
stan::math::return_type_t<T1, T2, T3, T4, T5>
^~~~~~~~~~~~~~~~~~~~~~~~~
stan::return_type
/usr/local/lib/python3.7/site-packages/pystan/stan/lib/stan_math/stan/math/prim/scal/meta/return_type.hpp:33:8: note: 
      'stan::return_type' declared here
struct return_type {
       ^

I attempted changing

stan::math::return_type_t<T1, T2, T3, T4, T5>

and

using T = stan::math::return_type_t<T1, T2, T3, T4, T5>;

to

stan::return_type<T1, T2, T3, T4, T5>

and

using T = stan::return_type<T1, T2, T3, T4, T5>;

in order to the address the error above, but I ended up with an output similar to the errors present in @bfinley’s original post.

andrjohns · January 21, 2020, 7:57am

Ah sorry, that should be stan::return_type_t<>, note the _t at the end.

Does that change anything?

matthew_ay · January 21, 2020, 8:13am

Andrew,

Using the C++ code you outlined (along with the PyStan code in @bfinley’s original post):

template <typename T1, typename T2, typename T3, typename T4, typename T5>
stan::return_type_t<T1, T2, T3, T4, T5>
my_other_func (const T1& A, const T2& B, const T3& C, const T4& X, const T5& Y, std::ostream* pstream) {
  using T = stan::return_type_t<T1, T2, T3, T4, T5>;
  
  T Z = A*X*X+B*X+C+Y;

  return Z;
}

I still received the error (along with the errors outlined in @bfinley’s original post):

In file included from /var/folders/jh/ksyr7q296sldxds02llwkyv80000gn/T/pystan_f6yzkn7y/stanfit4anon_model_aa7fce6b9639a550505fc2f8eeaafc68_321933308182462563.cpp:680:
In file included from /var/folders/jh/ksyr7q296sldxds02llwkyv80000gn/T/pystan_f6yzkn7y/anon_model_aa7fce6b9639a550505fc2f8eeaafc68.hpp:18:
./external_cpp/external_autograd.hpp:2:7: error: no template named 'return_type_t' in namespace 'stan'; did you
      mean 'return_type'?
stan::return_type_t<T1, T2, T3, T4, T5>
~~~~~~^~~~~~~~~~~~~
      return_type
/usr/local/lib/python3.7/site-packages/pystan/stan/lib/stan_math/stan/math/prim/scal/meta/return_type.hpp:33:8: note: 
      'return_type' declared here
struct return_type {
       ^
In file included from /var/folders/jh/ksyr7q296sldxds02llwkyv80000gn/T/pystan_f6yzkn7y/stanfit4anon_model_aa7fce6b9639a550505fc2f8eeaafc68_321933308182462563.cpp:680:
In file included from /var/folders/jh/ksyr7q296sldxds02llwkyv80000gn/T/pystan_f6yzkn7y/anon_model_aa7fce6b9639a550505fc2f8eeaafc68.hpp:18:
./external_cpp/external_autograd.hpp:4:19: error: no template named 'return_type_t' in namespace 'stan'; did you
      mean 'return_type'?
  using T = stan::return_type_t<T1, T2, T3, T4, T5>;
            ~~~~~~^~~~~~~~~~~~~
                  return_type

bgoodri · January 21, 2020, 8:15am

Using auto as the return type may work under C++14.

matthew_ay · January 21, 2020, 8:33am

Ben,

Unfortunately, I have fairly limited experience with C++ and don’t completely understand what you mean. Did you mean…

template <typename T1, typename T2, typename T3, typename T4, typename T5>
auto<T1, T2, T3, T4, T5>
my_other_func (const T1& A, const T2& B, const T3& C, const T4& X, const T5& Y, std::ostream* pstream) {
  using T = auto<T1, T2, T3, T4, T5>;
  
  T Z = A*X*X+B*X+C+Y;

  return Z;
}

bgoodri · January 21, 2020, 9:04am

Just

template <typename T1, typename T2, typename T3, typename T4, typename T5>
auto
my_other_func (const T1& A, const T2& B, const T3& C, const T4& X, const T5& Y, std::ostream* pstream) {
  
  auto Z = A*X*X+B*X+C+Y;
  return Z;
}

matthew_ay · January 21, 2020, 9:15am

Ben,

Went ahead and tried that, still have the same errors as with the original code in @bfinley’s initial post. For my output, the errors are (with some variance as to exactly which line the error is occurring):

/var/folders/jh/ksyr7q296sldxds02llwkyv80000gn/T/pystan_uolwiug5/anon_model_aa7fce6b9639a550505fc2f8eeaafc68.hpp:231:34: error: 
      call to 'my_other_func' is ambiguous
            stan::math::assign(E,my_other_func(A, B, C, X, Y, pstream__));
                                 ^~~~~~~~~~~~~

and

n file included from /var/folders/jh/ksyr7q296sldxds02llwkyv80000gn/T/pystan_uolwiug5/stanfit4anon_model_aa7fce6b9639a550505fc2f8eeaafc68_2596437037328481539.cpp:678:
In file included from /usr/local/lib/python3.7/site-packages/pystan/stan_fit.hpp:24:
In file included from /usr/local/lib/python3.7/site-packages/pystan/stan/src/stan/services/optimize/bfgs.hpp:11:
In file included from /usr/local/lib/python3.7/site-packages/pystan/stan/src/stan/optimization/bfgs.hpp:5:
/usr/local/lib/python3.7/site-packages/pystan/stan/src/stan/model/log_prob_propto.hpp:45:28: error: no matching
      member function for call to 'log_prob'
          = model.template log_prob<true, jacobian_adjust_transform>
            ~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

bgoodri · January 21, 2020, 9:23am

Oh. I think the last argument needs to be called pstream__ rather than pstream in your C++ file. That and you would need a return Z.eval(); if you use the auto keyword.

matthew_ay · January 21, 2020, 10:33am

Ben,

Using the code below, we are encountering the same errors as before. Also, thank you for assisting us with this issue.

template <typename T1, typename T2, typename T3, typename T4, typename T5>
auto 
my_other_func (const T1& A, const T2& B, const T3& C, const T4& X, const T5& Y, std::ostream* pstream__) {
  
  auto Z = A*X*X+B*X+C+Y;
  return Z.eval();
}

Topic		Replies	Views
External C++ Code with Precomputed Gradients CmdStan	4	1146	February 7, 2022
Pystan support external C++ function? Developers	4	454	August 16, 2019
Including boost function into Stan by Pystan 2.18.0 Modeling techniques	5	725	August 27, 2018
Problems while calling an external C++ function Modeling	2	489	September 13, 2023
Linker error when calling external C++ within Stan function General techniques , specification , performance	2	924	June 17, 2020

Help with using an external C++ file

Related topics