Hoping for some guidance / help with implementing custom log likelihood and gradient for research project (details below)

Hi,

First off I wasn’t quite sure what topic to put this under. Sorry.

Some background.

I have been working on a research project using Stan to sample from the joint state and parameter posterior for dynamic state space models (these could be thought of as an extension of stochastic volatility model to hidden states of higher dimension and sometimes more structure). This is a topic of some interest to the control and system identification community as historically it has been hard to achieve for anything but low order models and hence maximum likelihood approaches have been most common. The results using stan are very promising. See the following preprint and github for current progress

Extending this work

It is well known within the sysid community that a Kalman smoother (KS) can be used to compute the log likelihood and the gradient given a sample of the parameters. We would like to implement this within stan in order to compare sampling only the parameters using the KS vs jointly sampling the parameters and states.

Problem (or where I am stuck)

I have C code to compute the log likelihood and the gradient. I have downloaded and installed stan from source and been able to run the examples using it. I have also read the documentation on contributing to stan. However, I am not experienced in C++ and so this is where I get stuck.

Request

I am really hoping that someone might be able to take the time to explain / provide a short demo of how I can implement my own log likelihood and gradients in stan. I would be very happy for either a demo here, or even a video chat (or potential more involvement if someone is interested in the research).

Thanks very much in advance for any help

2 Likes

Hi it looks like you are interesting in adding a new log probability density function. The below guide for the math library may be helpful.

http://mc-stan.org/math/d5/db4/group__prob__dists.html#new_distribution

If you have functions that you would like to implement check out the Stan math getting started guide

http://mc-stan.org/math/getting_started.html

Once these functions pass our test suite that checks your gradients against finite difference for correctness you can expose those functions to the stan language by adding the signature here in the stanc3 compiler. Alternatively you can use them by including the function signature in the stan file and compiling with the --allow-undefined flag. Then when you compile the generated C++ include the path to the C++ with your function.

1 Like

Thanks for the prompt response. That first link looks very helpful, not sure how I havent found it in the past. I will have a go at this over the next few days and come back here with questions if i get stuck.

1 Like

Hey,

Would it be a possible to get a little bit more guidance on the last part of what you mentioned.

Once I have written a new log probability density function how do i go about using it? Do i need to recompile cmdstan / stan math? I would like to try the second option you mentioned (the --allow-undefined) as it seems the simplest to get started. With the include the signature in the stan file, am I right in guessing that for the new_normal_lpdf example that would mean including

functions {
real new_normal_lpdf(vector y, vector loc, real scale);
}

at the top of the stan model file?

These are probably really dumb questions, but with my limited C++ experience this is where I am struggling the most and hopefully it will all click soon.

I found this 15 stanc: Translating Stan to C++ | CmdStan User’s Guide and realised I needed to include the USER_HEADER when running make. The compiler now finds the new .hpp file but I get an error that to me is a wall of text. All i am trying to do at this stage is to get the example new_normal_lpdf working (I even tried just copying the contents from normal_lpdf and renaming the functions and the #ifndef parts.

Within the massive error message is the bit
undefined reference to `boost::math::tools::promote_args<stan::value_type<Eigen::Matrix<double, -1, 1, 0, -1, 1>,
which I gather means i need to point it to Eigen in some way but I’m really not sure how I do that

Are you able to post your code / an environment to replicate your error? It’s rather hard to tell where the error comes from without seeing how its made

Hey,

I’m not entirely sure what the best way of doing this would be. I cloned the cmdstan and have followed the instructions in the first link you shared (Stan Math Library: Probability Distributions). My goal was to exactly copy the ‘new_normal’ distribution, because i figured if i could get that working it wouldn’t be too hard to modify the math part of the code.

When I cloned cmdstan I did end up with a different folder structure so I’m wondering if i went fundamentally wrong from the start? for instance the normal_lpdf code is located in stan > lib > stan_math > stan > math > prim > prob and also I don’t have stanc3 / src / middle

I’ll share in a second reply below the new_normal.hpp code I have, but it should be the same as in the link.

my stan model code

functions{
    real new_normal_lpdf(vector y, vector loc, real scale);
}

data {
    int<lower=0> N;
    vector[N] u;
    vector[N] y;
}

parameters {
    real<lower=0.,upper=1.> a;
    real b;
    vector[N] x;
    real<lower=1e-8> q;
    real<lower=1e-8> r;
}

transformed parameters {
    vector[N] mu = a * x + b * u;
}

model {
    // priors
    q ~ cauchy(0., 1.);
    r ~ cauchy(0., 1.);

    // likelihoods
//    x[2:N] ~ normal(mu[1:N-1], q);
    target += new_normal_lpdf(x[2:N] | mu[1:N-1], q);
    y ~ normal(x, r);

}

generated quantities {
    vector[N] yhat = x + normal_rng(0, r);
}

Which I have tried compiling using both

make STANCFLAGS=--allow-undefined USER_HEADER=./stan/lib/stan_math/stan/math/prim/prob/new_normal_lpdf.hpp models/ssm
make STANCFLAGS=--allow-undefined 
USER_HEADER=~/git/cmdstan/stan/lib/stan_math/stan/math/prim/prob/new_normal_lpdf.hpp models/ssm

Is there an alternative way I should be trying to share my environment with you? I feel certain the problem is probably something quite basic about how I should be setting up the environment / including the right things and could be sorted out by a 10 min discussion if that would be possible.

EDIT: @maxbiostat edited this post for syntax highlighting.

Adding this although I don’t think its where the issue is

#ifndef STAN_MATH_PRIM_PROB_NEW_NORMAL_LPDF_HPP
#define STAN_MATH_PRIM_PROB_NEW_NORMAL_LPDF_HPP

#include <stan/math/prim/meta.hpp>
#include <stan/math/prim/err.hpp>
#include <stan/math/prim/fun/as_column_vector_or_scalar.hpp>
#include <stan/math/prim/fun/as_array_or_scalar.hpp>
#include <stan/math/prim/fun/as_value_column_array_or_scalar.hpp>
#include <stan/math/prim/fun/constants.hpp>
#include <stan/math/prim/fun/log.hpp>
#include <stan/math/prim/fun/max_size.hpp>
#include <stan/math/prim/fun/size.hpp>
#include <stan/math/prim/fun/size_zero.hpp>
#include <stan/math/prim/fun/to_ref.hpp>
#include <stan/math/prim/fun/value_of.hpp>
#include <stan/math/prim/functor/operands_and_partials.hpp>
#include <cmath>


namespace stan {
namespace math {

template <bool propto, typename T_y, typename T_loc, typename T_scale>
inline return_type_t<T_y, T_loc, T_scale> new_normal_lpdf(const T_y& y,
                                                      const T_loc& mu,
                                                      const T_scale& sigma) {

  using T_partials_return = partials_return_t<T_y, T_loc, T_scale>;
  using T_y_ref = ref_type_if_t<!is_constant<T_y>::value, T_y>;
  using T_mu_ref = ref_type_if_t<!is_constant<T_loc>::value, T_loc>;
  using T_sigma_ref = ref_type_if_t<!is_constant<T_scale>::value, T_scale>;
  static const char* function = "new_normal_lpdf";
  check_consistent_sizes(function, "Random variable", y, "Location parameter",
                         mu, "Scale parameter", sigma);
  T_y_ref y_ref = y;
  T_mu_ref mu_ref = mu;
  T_sigma_ref sigma_ref = sigma;

  decltype(auto) y_val = to_ref(as_value_column_array_or_scalar(y_ref));
  decltype(auto) mu_val = to_ref(as_value_column_array_or_scalar(mu_ref));
  decltype(auto) sigma_val = to_ref(as_value_column_array_or_scalar(sigma_ref));

  check_not_nan(function, "Random variable", y_val);
  check_finite(function, "Location parameter", mu_val);
  check_positive(function, "Scale parameter", sigma_val);

  if (size_zero(y, mu, sigma)) {
    return 0.0;
  }
  if (!include_summand<propto, T_y, T_loc, T_scale>::value) {
    return 0.0;
  }

  operands_and_partials<T_y_ref, T_mu_ref, T_sigma_ref> ops_partials(
      y_ref, mu_ref, sigma_ref);

  const auto& inv_sigma
      = to_ref_if<!is_constant_all<T_y, T_scale, T_loc>::value>(inv(sigma_val));
  const auto& y_scaled = to_ref((y_val - mu_val) * inv_sigma);
  const auto& y_scaled_sq
      = to_ref_if<!is_constant_all<T_scale>::value>(y_scaled * y_scaled);

  size_t N = max_size(y, mu, sigma);
  T_partials_return logp = -0.5 * sum(y_scaled_sq);
  if (include_summand<propto>::value) {
    logp += NEG_LOG_SQRT_TWO_PI * N;
  }
  if (include_summand<propto, T_scale>::value) {
    logp -= sum(log(sigma_val)) * N / size(sigma);
  }

  if (!is_constant_all<T_y, T_scale, T_loc>::value) {
    auto scaled_diff = to_ref_if<!is_constant_all<T_y>::value
                                     + !is_constant_all<T_scale>::value
                                     + !is_constant_all<T_loc>::value
                                 >= 2>(inv_sigma * y_scaled);
    if (!is_constant_all<T_y>::value) {
      ops_partials.edge1_.partials_ = -scaled_diff;
    }
    if (!is_constant_all<T_scale>::value) {
      ops_partials.edge3_.partials_ = inv_sigma * y_scaled_sq - inv_sigma;
    }
    if (!is_constant_all<T_loc>::value) {
      ops_partials.edge2_.partials_ = std::move(scaled_diff);
    }
  }
  return ops_partials.build(logp);
}

template <typename T_y, typename T_loc, typename T_scale>
inline return_type_t<T_y, T_loc, T_scale> new_normal_lpdf(const T_y& y,
                                                      const T_loc& mu,
                                                      const T_scale& sigma) {
  return new_normal_lpdf<false>(y, mu, sigma);
}

}  // namespace math
}  // namespace stan
#endif

EDIT: @maxbiostat edited this post for syntax highlighting.

If you have added the new custom log likelihood inside the stan::math namespace in Stan Math, then you must not set it as the USER_HEADER. The USER_HEADER is for external code not part of the stan::math namespace.

The only thing I think you are missing is including the new file in the math/prim/prob.hpp file: math/prob.hpp at develop · stan-dev/math · GitHub

1 Like

Hey,

Included the new file in prob.hpp and then

Removing the USER_HEADER but leaving in the --allow-undefined flag gave me the fatal error:models/user_header.hpp: No such file or directory

Either removing the --allow-undefined flag or adding an empty models/user_header.hpp file gets me back to the same error where it looks like its boost and eigen references aren’t defined.

Its starting to seem like it might be worth trying to alternate approach of exposing the new function to stanc. This seems like it would require recompiling stanc. However, when I cloned cmdstan it doesn’t seem like i got the source for stanc, how do i go about doing that and compiling it so that it is used by cmdstan?

Ok, So i have something of a solution. It doesn’t seem very ‘neat’ at all, but it appears to work. I am hoping that someone could tell me how to make it a bit nicer to use.

Present set up for cmdstan

  1. cloned cmdstan and built according to instructions on cmdstan getting started page
  2. added a new distribution 'new_normal" following the guide on the stan math page (Stan Math Library: Probability Distributions)
  3. added this file into the header prob.hpp

Set up of stanc3

  1. cloned stanc3 into a new directory
  2. added the signature for new_normal to middle/Stan_math_signatures
  3. built stanc3 which gives the stanc.exe file

Setting up the model

  1. can now use new_normal in the exact same way I would have previously used the normal distribution i.e. y ~ normal(mu, std);

Building the model
Ok, so this is where it got a bit yuck.

  1. cd into the stanc directory
  2. translate the model.stan to a model.hpp file by using
    ./_build/default/src/stanc/stanc.exe path/to/model.stan
  3. change directory back to cmdstan
  4. call make path/to/model.hpp
  5. provided that model.hpp is newer than model.stan it doesnt try to recreate the hpp file using cmdstan’s version of stanc and instead compiles the hpp file we created using our newly built stanc.exe
  6. can now sample from this model as per normal

So the yuck bit is that cmdstan doesn’t use the modified version of stanc3, any suggestions on how to fix this?

Regards,
Johannes

For that part you should just be able to copy the stanc3 binary from the stanc repo into the /bin folder of the cmdstan repository

Here is another example without having to change stanc3:

Stan model:

hpp file: cmdstanpy/make_odds.hpp at 2ed87fe2780e80d04d3dc783fac4ddb4762adb22 · stan-dev/cmdstanpy · GitHub

Note that the function is inside the namespace bernoulli_external_model which is the model name + the _model suffix.

Then you need to set the following in the make/local file:

USER_HEADER=path/to/hpp/file.hpp
STANCFLAGS += --allow-undefined

an example with cmdstanr:

library(cmdstanr)

stan_file <- "bernoulli_external.stan"

mod <- cmdstan_model(
  stan_file,
  cpp_options = list(USER_HEADER="/full/path/to/the/external/file/make_odds.hpp"),
  stanc_options = list("allow-undefined")
)
data_list <- list(N = 10, y = c(0,1,0,0,0,0,0,0,0,1))
fit <- mod$sample(data = data_list)

Note that at the moment the USER_HEADER at least for cmdstanr has to be the full absolute path.

1 Like

I tried that with no success.

I did already go through this example, however, it doesn’t help very much with the goal of implementing a custom distribution function and doesn’t help with following the guide provided at the other link (the new_normal function described in the other guide requires access to stan math namespace)

Can you see if this thread helps to get the normal example working?

If that does work then you’ll have to define your new distribution and the derivatives. However, if you can get it running in pure Stan first autodiff will take care of the derivatives (though this will be slower).

1 Like

This thread looks like it should be exactly what I want. Based on it I implement the following function

#include <stan/math/rev/core.hpp>

inline stan::math::var new_normal_2_lpdf(const stan::math::var& y,
const stan::math::var& mu,
const stan::math::var& sigma,
std::ostream* pstream__){
// extract values (not derivatives of inputs)
double mu_val = mu.val();
double y_val = y.val();
double sigma_val = sigma.val();

double t0 = 1/sigma_val;
double t1 = t0 * t0;
double t2 = y_val - mu_val;
double t3 = t2 / t1;

double log_prob = -0.5 * t2 * t2 / t1 - std::log(sigma_val);

double df_dy = -t3;
double df_dmu = t3;
double df_dsigma = t2*t2/t1*t0 - t0;

return stan::math::var(new stan::math::precomp_vvv_vari(log_prob, y.vi_, mu.vi_, sigma.vi_, df_dy, df_dmu, df_dsigma));

}

but then when I try to compile I get the an error saying undefined references

— Compiling, linking C++ code —
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -I stan/lib/stan_math/lib/tbb_2020.3/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_5.7.0/include -DBOOST_DISABLE_ASSERTS -c -Wno-ignored-attributes -include /home/johannes/git/cmdstan/new_normal_2.hpp -x c++ -o normal_test.o normal_test.hpp
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes -I stan/lib/stan_math/lib/tbb_2020.3/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_5.7.0/include -DBOOST_DISABLE_ASSERTS -Wl,-L,"/home/johannes/git/cmdstan/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/home/johannes/git/cmdstan/stan/lib/stan_math/lib/tbb" normal_test.o src/cmdstan/main.o -Wl,-L,"/home/johannes/git/cmdstan/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/home/johannes/git/cmdstan/stan/lib/stan_math/lib/tbb" stan/lib/stan_math/lib/sundials_5.7.0/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_5.7.0/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_5.7.0/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_5.7.0/lib/libsundials_kinsol.a stan/lib/stan_math/lib/tbb/libtbb.so.2 -o normal_test
/usr/bin/ld: normal_test.o: in function stan::model::model_base_crtp<normal_test_model_namespace::normal_test_model>::log_prob_propto(Eigen::Matrix<double, -1, 1, 0, -1, 1>&, std::ostream*) const': normal_test.hpp:(.text._ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE15log_prob_proptoERN5Eigen6MatrixIdLin1ELi1ELi0ELin1ELi1EEEPSo[_ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE15log_prob_proptoERN5Eigen6MatrixIdLin1ELi1ELi0ELin1ELi1EEEPSo]+0x156): undefined reference to boost::math::tools::promote_args<double, double, double, float, float, float>::type normal_test_model_namespace::new_normal_2_lpdf<false, double, double, double>(double const&, double const&, double const&, std::ostream*)’
/usr/bin/ld: normal_test.o: in function stan::model::model_base_crtp<normal_test_model_namespace::normal_test_model>::log_prob(Eigen::Matrix<double, -1, 1, 0, -1, 1>&, std::ostream*) const': normal_test.hpp:(.text._ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE8log_probERN5Eigen6MatrixIdLin1ELi1ELi0ELin1ELi1EEEPSo[_ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE8log_probERN5Eigen6MatrixIdLin1ELi1ELi0ELin1ELi1EEEPSo]+0x187): undefined reference to boost::math::tools::promote_args<double, double, double, float, float, float>::type normal_test_model_namespace::new_normal_2_lpdf<false, double, double, double>(double const&, double const&, double const&, std::ostream*)’
/usr/bin/ld: normal_test.o: in function stan::model::model_base_crtp<normal_test_model_namespace::normal_test_model>::log_prob_jacobian(Eigen::Matrix<double, -1, 1, 0, -1, 1>&, std::ostream*) const': normal_test.hpp:(.text._ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE17log_prob_jacobianERN5Eigen6MatrixIdLin1ELi1ELi0ELin1ELi1EEEPSo[_ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE17log_prob_jacobianERN5Eigen6MatrixIdLin1ELi1ELi0ELin1ELi1EEEPSo]+0x198): undefined reference to boost::math::tools::promote_args<double, double, double, float, float, float>::type normal_test_model_namespace::new_normal_2_lpdf<false, double, double, double>(double const&, double const&, double const&, std::ostream*)’
/usr/bin/ld: normal_test.o: in function stan::model::model_base_crtp<normal_test_model_namespace::normal_test_model>::log_prob_propto_jacobian(Eigen::Matrix<double, -1, 1, 0, -1, 1>&, std::ostream*) const': normal_test.hpp:(.text._ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE24log_prob_propto_jacobianERN5Eigen6MatrixIdLin1ELi1ELi0ELin1ELi1EEEPSo[_ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE24log_prob_propto_jacobianERN5Eigen6MatrixIdLin1ELi1ELi0ELin1ELi1EEEPSo]+0x166): undefined reference to boost::math::tools::promote_args<double, double, double, float, float, float>::type normal_test_model_namespace::new_normal_2_lpdf<false, double, double, double>(double const&, double const&, double const&, std::ostream*)’
/usr/bin/ld: normal_test.o: in function stan::model::model_base_crtp<normal_test_model_namespace::normal_test_model>::log_prob(std::vector<double, std::allocator<double> >&, std::vector<int, std::allocator<int> >&, std::ostream*) const': normal_test.hpp:(.text._ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE8log_probERSt6vectorIdSaIdEERS5_IiSaIiEEPSo[_ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE8log_probERSt6vectorIdSaIdEERS5_IiSaIiEEPSo]+0x1a7): undefined reference to boost::math::tools::promote_args<double, double, double, float, float, float>::type normal_test_model_namespace::new_normal_2_lpdf<false, double, double, double>(double const&, double const&, double const&, std::ostream*)’
/usr/bin/ld: normal_test.o:normal_test.hpp:(.text._ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE17log_prob_jacobianERSt6vectorIdSaIdEERS5_IiSaIiEEPSo[_ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE17log_prob_jacobianERSt6vectorIdSaIdEERS5_IiSaIiEEPSo]+0x1b0): more undefined references to boost::math::tools::promote_args<double, double, double, float, float, float>::type normal_test_model_namespace::new_normal_2_lpdf<false, double, double, double>(double const&, double const&, double const&, std::ostream*)' follow /usr/bin/ld: normal_test.o: in function stan::model::model_base_crtp<normal_test_model_namespace::normal_test_model>::log_prob_jacobian(std::vector<stan::math::var_value<double, void>, std::allocator<stan::math::var_value<double, void> > >&, std::vector<int, std::allocator >&, std::ostream*) const’:
normal_test.hpp:(.text._ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE17log_prob_jacobianERSt6vectorINS_4math9var_valueIdvEESaIS8_EERS5_IiSaIiEEPSo[_ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE17log_prob_jacobianERSt6vectorINS_4math9var_valueIdvEESaIS8_EERS5_IiSaIiEEPSo]+0x32e): undefined reference to boost::math::tools::promote_args<double, stan::math::var_value<double, void>, stan::math::var_value<double, void>, float, float, float>::type normal_test_model_namespace::new_normal_2_lpdf<false, double, stan::math::var_value<double, void>, stan::math::var_value<double, void> >(double const&, stan::math::var_value<double, void> const&, stan::math::var_value<double, void> const&, std::ostream*)' /usr/bin/ld: normal_test.o: in function stan::model::model_base_crtp<normal_test_model_namespace::normal_test_model>::log_prob(std::vector<stan::math::var_value<double, void>, std::allocator<stan::math::var_value<double, void> > >&, std::vector<int, std::allocator >&, std::ostream*) const’:
normal_test.hpp:(.text._ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE8log_probERSt6vectorINS_4math9var_valueIdvEESaIS8_EERS5_IiSaIiEEPSo[_ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE8log_probERSt6vectorINS_4math9var_valueIdvEESaIS8_EERS5_IiSaIiEEPSo]+0x310): undefined reference to boost::math::tools::promote_args<double, stan::math::var_value<double, void>, stan::math::var_value<double, void>, float, float, float>::type normal_test_model_namespace::new_normal_2_lpdf<false, double, stan::math::var_value<double, void>, stan::math::var_value<double, void> >(double const&, stan::math::var_value<double, void> const&, stan::math::var_value<double, void> const&, std::ostream*)' /usr/bin/ld: normal_test.o: in function stan::model::model_base_crtp<normal_test_model_namespace::normal_test_model>::log_prob_propto_jacobian(std::vector<stan::math::var_value<double, void>, std::allocator<stan::math::var_value<double, void> > >&, std::vector<int, std::allocator >&, std::ostream*) const’:
normal_test.hpp:(.text._ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE24log_prob_propto_jacobianERSt6vectorINS_4math9var_valueIdvEESaIS8_EERS5_IiSaIiEEPSo[_ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE24log_prob_propto_jacobianERSt6vectorINS_4math9var_valueIdvEESaIS8_EERS5_IiSaIiEEPSo]+0x32e): undefined reference to boost::math::tools::promote_args<double, stan::math::var_value<double, void>, stan::math::var_value<double, void>, float, float, float>::type normal_test_model_namespace::new_normal_2_lpdf<false, double, stan::math::var_value<double, void>, stan::math::var_value<double, void> >(double const&, stan::math::var_value<double, void> const&, stan::math::var_value<double, void> const&, std::ostream*)' /usr/bin/ld: normal_test.o: in function stan::model::model_base_crtp<normal_test_model_namespace::normal_test_model>::log_prob_propto(std::vector<stan::math::var_value<double, void>, std::allocator<stan::math::var_value<double, void> > >&, std::vector<int, std::allocator >&, std::ostream*) const’:
normal_test.hpp:(.text._ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE15log_prob_proptoERSt6vectorINS_4math9var_valueIdvEESaIS8_EERS5_IiSaIiEEPSo[_ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE15log_prob_proptoERSt6vectorINS_4math9var_valueIdvEESaIS8_EERS5_IiSaIiEEPSo]+0x310): undefined reference to boost::math::tools::promote_args<double, stan::math::var_value<double, void>, stan::math::var_value<double, void>, float, float, float>::type normal_test_model_namespace::new_normal_2_lpdf<false, double, stan::math::var_value<double, void>, stan::math::var_value<double, void> >(double const&, stan::math::var_value<double, void> const&, stan::math::var_value<double, void> const&, std::ostream*)' /usr/bin/ld: normal_test.o: in function stan::model::model_base_crtp<normal_test_model_namespace::normal_test_model>::log_prob_propto_jacobian(Eigen::Matrix<stan::math::var_value<double, void>, -1, 1, 0, -1, 1>&, std::ostream*) const’:
normal_test.hpp:(.text._ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE24log_prob_propto_jacobianERN5Eigen6MatrixINS_4math9var_valueIdvEELin1ELi1ELi0ELin1ELi1EEEPSo[_ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE24log_prob_propto_jacobianERN5Eigen6MatrixINS_4math9var_valueIdvEELin1ELi1ELi0ELin1ELi1EEEPSo]+0x318): undefined reference to boost::math::tools::promote_args<double, stan::math::var_value<double, void>, stan::math::var_value<double, void>, float, float, float>::type normal_test_model_namespace::new_normal_2_lpdf<false, double, stan::math::var_value<double, void>, stan::math::var_value<double, void> >(double const&, stan::math::var_value<double, void> const&, stan::math::var_value<double, void> const&, std::ostream*)' /usr/bin/ld: normal_test.o:normal_test.hpp:(.text._ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE8log_probERN5Eigen6MatrixINS_4math9var_valueIdvEELin1ELi1ELi0ELin1ELi1EEEPSo[_ZNK4stan5model15model_base_crtpIN27normal_test_model_namespace17normal_test_modelEE8log_probERN5Eigen6MatrixINS_4math9var_valueIdvEELin1ELi1ELi0ELin1ELi1EEEPSo]+0x326): more undefined references to boost::math::tools::promote_args<double, stan::math::var_value<double, void>, stan::math::var_value<double, void>, float, float, float>::type normal_test_model_namespace::new_normal_2_lpdf<false, double, stan::math::var_value<double, void>, stan::math::var_value<double, void> >(double const&, stan::math::var_value<double, void> const&, stan::math::var_value<double, void> const&, std::ostream*)’ follow
collect2: error: ld returned 1 exit status
make: *** [make/program:56: normal_test] Error 1

Sorry, I know this is probably frustrating that I am stuck on something that seems like it should be simple (believe me I am frustrated too)

Try replacing #include <stan/math/rev/core.hpp> with #include <stan/math.hpp>

Same error as before. Looking closely at part of the error

undefined reference to `boost::math::tools::promote_args<double, stan::math::var_value<double, void>, stan::math::var_value<double, void>, float, float, float>::type normal_test_model_namespace::new_normal_2_lpdf<false, double, stan::math::var_value<double, void>, stan::math::var_value<double, void> >(double const&, stan::math::var_value<double, void> const&, stan::math::var_value<double, void> const&, std::ostream*)'

it looks like its looking for new_normal_2_lpdf (which is what I called my function) in the normal_test_model_namespace namespace but unable to find it. So maybe I need to somehow make the function part of that namespace? however, the normal_test_model_namespace namespace isn’t currently defined within new_normal_2.hpp and I’m not sure how I would add it in

Here is my current hpp file with the new distribution function:


#ifndef NEW_NORMAL_2_LPDF_HPP
#define NEW_NORMAL_2_LPDF_HPP


#include <stan/math/prim.hpp>
//#include <stan/math/rev/core.hpp>
#include <stan/math.hpp>

inline stan::math::var new_normal_2_lpdf(const stan::math::var& y,
                                        const stan::math::var& mu,
                                        const stan::math::var& sigma,
                                         std::ostream* pstream__){
    // extract values (not derivatives of inputs)
    double mu_val = mu.val();
    double y_val = y.val();
    double sigma_val = sigma.val();

    double t0 = 1/sigma_val;
    double t1 = t0 * t0;
    double t2 = y_val - mu_val;
    double t3 = t2 / t1;

    double log_prob = -0.5 * t2 * t2 / t1 - std::log(sigma_val);

    double df_dy = -t3;
    double df_dmu = t3;
    double df_dsigma = t2*t2/t1*t0 - t0;

    return stan::math::var(new stan::math::precomp_vvv_vari(log_prob, y.vi_, mu.vi_, sigma.vi_, df_dy, df_dmu, df_dsigma));
}

#endif

Here is the stan model file

functions {
//    real new_normal_2_lpdf(real y, real mu, real sigma){
//        return -0.5 * pow((y - mu)/ sigma, 2) - log(sigma) - 0.5 * log(2*pi());
//    }
    real new_normal_2_lpdf(real y, real mu, real sigma);
}

data {
    int <lower=0> N;
    vector[N] x;
    vector[N] y;

}
parameters {
    real b;
    real<lower=1e-6> r;
}
model {
    r ~ cauchy(0, 1);
//    y ~ normal(b * x, r);
    for (n in 1:N){
        target += new_normal_2_lpdf(y[n] | b * x[n], r);
    }
}

I am compiling with the command

make STANCFLAGS=--allow-undefined USER_HEADER=~/git/cmdstan/new_normal_2.hpp normal_test

Here is the model hpp file that is being generated by stanc: Where it does look like the new_normal_2 function is defined inside the normal_test_model_namespace ?

// Code generated by stanc a8f502d5
#include <stan/model/model_header.hpp>
namespace normal_test_model_namespace {

using stan::io::dump;
using stan::model::assign;
using stan::model::index_uni;
using stan::model::index_max;
using stan::model::index_min;
using stan::model::index_min_max;
using stan::model::index_multi;
using stan::model::index_omni;
using stan::model::model_base_crtp;
using stan::model::rvalue;
using namespace stan::math;

stan::math::profile_map profiles__;
static constexpr std::array<const char*, 12> locations_array__ = 
{" (found before start of program)",
 " (in 'normal_test.stan', line 16, column 4 to column 11)",
 " (in 'normal_test.stan', line 17, column 4 to column 23)",
 " (in 'normal_test.stan', line 20, column 4 to column 21)",
 " (in 'normal_test.stan', line 23, column 8 to column 56)",
 " (in 'normal_test.stan', line 22, column 18 to line 24, column 5)",
 " (in 'normal_test.stan', line 22, column 4 to line 24, column 5)",
 " (in 'normal_test.stan', line 10, column 4 to column 20)",
 " (in 'normal_test.stan', line 11, column 11 to column 12)",
 " (in 'normal_test.stan', line 11, column 4 to column 16)",
 " (in 'normal_test.stan', line 12, column 11 to column 12)",
 " (in 'normal_test.stan', line 12, column 4 to column 16)"};


template <bool propto__, typename T0__, typename T1__, typename T2__>
stan::promote_args_t<T0__, T1__,
T2__>
new_normal_2_lpdf(const T0__& y, const T1__& mu, const T2__& sigma,
                  std::ostream* pstream__) ;

class normal_test_model final : public model_base_crtp<normal_test_model> {

 private:
  int N;
  Eigen::Matrix<double, -1, 1> x__;
  Eigen::Matrix<double, -1, 1> y__; 
  Eigen::Map<Eigen::Matrix<double, -1, 1>> x{nullptr, 0};
  Eigen::Map<Eigen::Matrix<double, -1, 1>> y{nullptr, 0};
 
 public:
  ~normal_test_model() { }
  
  inline std::string model_name() const final { return "normal_test_model"; }

  inline std::vector<std::string> model_compile_info() const noexcept {
    return std::vector<std::string>{"stanc_version = stanc3 a8f502d5", "stancflags = --allow-undefined"};
  }
  
  
  normal_test_model(stan::io::var_context& context__,
                    unsigned int random_seed__ = 0,
                    std::ostream* pstream__ = nullptr) : model_base_crtp(0) {
    int current_statement__ = 0;
    using local_scalar_t__ = double ;
    boost::ecuyer1988 base_rng__ = 
        stan::services::util::create_rng(random_seed__, 0);
    (void) base_rng__;  // suppress unused var warning
    static constexpr const char* function__ = "normal_test_model_namespace::normal_test_model";
    (void) function__;  // suppress unused var warning
    local_scalar_t__ DUMMY_VAR__(std::numeric_limits<double>::quiet_NaN());
    (void) DUMMY_VAR__;  // suppress unused var warning
    try {
      int pos__;
      pos__ = 1;
      current_statement__ = 7;
      context__.validate_dims("data initialization","N","int",
           std::vector<size_t>{});
      N = std::numeric_limits<int>::min();
      
      current_statement__ = 7;
      N = context__.vals_i("N")[(1 - 1)];
      current_statement__ = 7;
      check_greater_or_equal(function__, "N", N, 0);
      current_statement__ = 8;
      validate_non_negative_index("x", "N", N);
      current_statement__ = 9;
      context__.validate_dims("data initialization","x","double",
           std::vector<size_t>{static_cast<size_t>(N)});
      x__ = Eigen::Matrix<double, -1, 1>(N);
      new (&x) Eigen::Map<Eigen::Matrix<double, -1, 1>>(x__.data(), N);
      
      {
        std::vector<local_scalar_t__> x_flat__;
        current_statement__ = 9;
        x_flat__ = context__.vals_r("x");
        current_statement__ = 9;
        pos__ = 1;
        current_statement__ = 9;
        for (int sym1__ = 1; sym1__ <= N; ++sym1__) {
          current_statement__ = 9;
          assign(x, x_flat__[(pos__ - 1)],
            "assigning variable x", index_uni(sym1__));
          current_statement__ = 9;
          pos__ = (pos__ + 1);
        }
      }
      current_statement__ = 10;
      validate_non_negative_index("y", "N", N);
      current_statement__ = 11;
      context__.validate_dims("data initialization","y","double",
           std::vector<size_t>{static_cast<size_t>(N)});
      y__ = Eigen::Matrix<double, -1, 1>(N);
      new (&y) Eigen::Map<Eigen::Matrix<double, -1, 1>>(y__.data(), N);
      
      {
        std::vector<local_scalar_t__> y_flat__;
        current_statement__ = 11;
        y_flat__ = context__.vals_r("y");
        current_statement__ = 11;
        pos__ = 1;
        current_statement__ = 11;
        for (int sym1__ = 1; sym1__ <= N; ++sym1__) {
          current_statement__ = 11;
          assign(y, y_flat__[(pos__ - 1)],
            "assigning variable y", index_uni(sym1__));
          current_statement__ = 11;
          pos__ = (pos__ + 1);
        }
      }
    } catch (const std::exception& e) {
      stan::lang::rethrow_located(e, locations_array__[current_statement__]);
    }
    num_params_r__ = 1 + 1;
    
  }
  
  template <bool propto__, bool jacobian__ , typename VecR, typename VecI, 
  stan::require_vector_like_t<VecR>* = nullptr, 
  stan::require_vector_like_vt<std::is_integral, VecI>* = nullptr> 
  inline stan::scalar_type_t<VecR> log_prob_impl(VecR& params_r__,
                                                 VecI& params_i__,
                                                 std::ostream* pstream__ = nullptr) const {
    using T__ = stan::scalar_type_t<VecR>;
    using local_scalar_t__ = T__;
    T__ lp__(0.0);
    stan::math::accumulator<T__> lp_accum__;
    stan::io::deserializer<local_scalar_t__> in__(params_r__, params_i__);
    int current_statement__ = 0;
    local_scalar_t__ DUMMY_VAR__(std::numeric_limits<double>::quiet_NaN());
    (void) DUMMY_VAR__;  // suppress unused var warning
    static constexpr const char* function__ = "normal_test_model_namespace::log_prob";
    (void) function__;  // suppress unused var warning
    
    try {
      local_scalar_t__ b;
      current_statement__ = 1;
      b = in__.template read<local_scalar_t__>();
      local_scalar_t__ r;
      current_statement__ = 2;
      r = in__.template read_constrain_lb<local_scalar_t__, jacobian__>(1e-6,
            lp__);
      {
        current_statement__ = 3;
        lp_accum__.add(cauchy_lpdf<propto__>(r, 0, 1));
        current_statement__ = 6;
        for (int n = 1; n <= N; ++n) {
          current_statement__ = 4;
          lp_accum__.add(
            new_normal_2_lpdf<false>(rvalue(y, "y", index_uni(n)),
              (b * rvalue(x, "x", index_uni(n))), r, pstream__));
        }
      }
    } catch (const std::exception& e) {
      stan::lang::rethrow_located(e, locations_array__[current_statement__]);
    }
    lp_accum__.add(lp__);
    return lp_accum__.sum();
    } // log_prob_impl() 
    
  template <typename RNG, typename VecR, typename VecI, typename VecVar, 
  stan::require_vector_like_vt<std::is_floating_point, VecR>* = nullptr, 
  stan::require_vector_like_vt<std::is_integral, VecI>* = nullptr, 
  stan::require_std_vector_vt<std::is_floating_point, VecVar>* = nullptr> 
  inline void write_array_impl(RNG& base_rng__, VecR& params_r__,
                               VecI& params_i__, VecVar& vars__,
                               const bool emit_transformed_parameters__ = true,
                               const bool emit_generated_quantities__ = true,
                               std::ostream* pstream__ = nullptr) const {
    using local_scalar_t__ = double;
    stan::io::deserializer<local_scalar_t__> in__(params_r__, params_i__);
    stan::io::serializer<local_scalar_t__> out__(vars__);
    static constexpr bool propto__ = true;
    (void) propto__;
    double lp__ = 0.0;
    (void) lp__;  // dummy to suppress unused var warning
    int current_statement__ = 0; 
    stan::math::accumulator<double> lp_accum__;
    local_scalar_t__ DUMMY_VAR__(std::numeric_limits<double>::quiet_NaN());
    constexpr bool jacobian__ = false;
    (void) DUMMY_VAR__;  // suppress unused var warning
    static constexpr const char* function__ = "normal_test_model_namespace::write_array";
    (void) function__;  // suppress unused var warning
    
    try {
      double b;
      current_statement__ = 1;
      b = in__.template read<local_scalar_t__>();
      double r;
      current_statement__ = 2;
      r = in__.template read_constrain_lb<local_scalar_t__, jacobian__>(1e-6,
            lp__);
      out__.write(b);
      out__.write(r);
      if (logical_negation((primitive_value(emit_transformed_parameters__) ||
            primitive_value(emit_generated_quantities__)))) {
        return ;
      } 
      if (logical_negation(emit_generated_quantities__)) {
        return ;
      } 
    } catch (const std::exception& e) {
      stan::lang::rethrow_located(e, locations_array__[current_statement__]);
    }
    } // write_array_impl() 
    
  template <typename VecVar, typename VecI, 
  stan::require_std_vector_t<VecVar>* = nullptr, 
  stan::require_vector_like_vt<std::is_integral, VecI>* = nullptr> 
  inline void transform_inits_impl(VecVar& params_r__, VecI& params_i__,
                                   VecVar& vars__,
                                   std::ostream* pstream__ = nullptr) const {
    using local_scalar_t__ = double;
    stan::io::deserializer<local_scalar_t__> in__(params_r__, params_i__);
    stan::io::serializer<local_scalar_t__> out__(vars__);
    int current_statement__ = 0;
    local_scalar_t__ DUMMY_VAR__(std::numeric_limits<double>::quiet_NaN());
    
    try {
      int pos__;
      pos__ = 1;
      local_scalar_t__ b;
      b = in__.read<local_scalar_t__>();
      out__.write(b);
      local_scalar_t__ r;
      r = in__.read<local_scalar_t__>();
      out__.write_free_lb(1e-6, r);
    } catch (const std::exception& e) {
      stan::lang::rethrow_located(e, locations_array__[current_statement__]);
    }
    } // transform_inits_impl() 
    
  inline void get_param_names(std::vector<std::string>& names__) const {
    
    names__ = std::vector<std::string>{"b", "r"};
    
    } // get_param_names() 
    
  inline void get_dims(std::vector<std::vector<size_t>>& dimss__) const {
    
    dimss__ = std::vector<std::vector<size_t>>{std::vector<size_t>{},
      std::vector<size_t>{}};
    
    } // get_dims() 
    
  inline void constrained_param_names(
                                      std::vector<std::string>& param_names__,
                                      bool emit_transformed_parameters__ = true,
                                      bool emit_generated_quantities__ = true) const
    final {
    
    param_names__.emplace_back(std::string() + "b");
    param_names__.emplace_back(std::string() + "r");
    if (emit_transformed_parameters__) {
      
    }
    
    if (emit_generated_quantities__) {
      
    }
    
    } // constrained_param_names() 
    
  inline void unconstrained_param_names(
                                        std::vector<std::string>& param_names__,
                                        bool emit_transformed_parameters__ = true,
                                        bool emit_generated_quantities__ = true) const
    final {
    
    param_names__.emplace_back(std::string() + "b");
    param_names__.emplace_back(std::string() + "r");
    if (emit_transformed_parameters__) {
      
    }
    
    if (emit_generated_quantities__) {
      
    }
    
    } // unconstrained_param_names() 
    
  inline std::string get_constrained_sizedtypes() const {
    
    return std::string("[{\"name\":\"b\",\"type\":{\"name\":\"real\"},\"block\":\"parameters\"},{\"name\":\"r\",\"type\":{\"name\":\"real\"},\"block\":\"parameters\"}]");
    
    } // get_constrained_sizedtypes() 
    
  inline std::string get_unconstrained_sizedtypes() const {
    
    return std::string("[{\"name\":\"b\",\"type\":{\"name\":\"real\"},\"block\":\"parameters\"},{\"name\":\"r\",\"type\":{\"name\":\"real\"},\"block\":\"parameters\"}]");
    
    } // get_unconstrained_sizedtypes() 
    
  
    // Begin method overload boilerplate
    template <typename RNG>
    inline void write_array(RNG& base_rng,
                            Eigen::Matrix<double,Eigen::Dynamic,1>& params_r,
                            Eigen::Matrix<double,Eigen::Dynamic,1>& vars,
                            const bool emit_transformed_parameters = true,
                            const bool emit_generated_quantities = true,
                            std::ostream* pstream = nullptr) const {
      const size_t num_params__ = 
  (1 + 1);
      const size_t num_transformed = 0;
      const size_t num_gen_quantities = 0;
      std::vector<double> vars_vec(num_params__
       + (emit_transformed_parameters * num_transformed)
       + (emit_generated_quantities * num_gen_quantities));
      std::vector<int> params_i;
      write_array_impl(base_rng, params_r, params_i, vars_vec,
          emit_transformed_parameters, emit_generated_quantities, pstream);
      vars = Eigen::Map<Eigen::Matrix<double,Eigen::Dynamic,1>>(
        vars_vec.data(), vars_vec.size());
    }

    template <typename RNG>
    inline void write_array(RNG& base_rng, std::vector<double>& params_r,
                            std::vector<int>& params_i,
                            std::vector<double>& vars,
                            bool emit_transformed_parameters = true,
                            bool emit_generated_quantities = true,
                            std::ostream* pstream = nullptr) const {
      const size_t num_params__ = 
  (1 + 1);
      const size_t num_transformed = 0;
      const size_t num_gen_quantities = 0;
      vars.resize(num_params__
        + (emit_transformed_parameters * num_transformed)
        + (emit_generated_quantities * num_gen_quantities));
      write_array_impl(base_rng, params_r, params_i, vars, emit_transformed_parameters, emit_generated_quantities, pstream);
    }

    template <bool propto__, bool jacobian__, typename T_>
    inline T_ log_prob(Eigen::Matrix<T_,Eigen::Dynamic,1>& params_r,
                       std::ostream* pstream = nullptr) const {
      Eigen::Matrix<int, -1, 1> params_i;
      return log_prob_impl<propto__, jacobian__>(params_r, params_i, pstream);
    }

    template <bool propto__, bool jacobian__, typename T__>
    inline T__ log_prob(std::vector<T__>& params_r,
                        std::vector<int>& params_i,
                        std::ostream* pstream = nullptr) const {
      return log_prob_impl<propto__, jacobian__>(params_r, params_i, pstream);
    }


    inline void transform_inits(const stan::io::var_context& context,
                         Eigen::Matrix<double, Eigen::Dynamic, 1>& params_r,
                         std::ostream* pstream = nullptr) const final {
      std::vector<double> params_r_vec(params_r.size());
      std::vector<int> params_i;
      transform_inits(context, params_i, params_r_vec, pstream);
      params_r = Eigen::Map<Eigen::Matrix<double,Eigen::Dynamic,1>>(
        params_r_vec.data(), params_r_vec.size());
    }

  inline void transform_inits(const stan::io::var_context& context,
                              std::vector<int>& params_i,
                              std::vector<double>& vars,
                              std::ostream* pstream__ = nullptr) const {
     constexpr std::array<const char*, 2> names__{"b", "r"};
     const std::array<Eigen::Index, 2> num_params__{1, 1};
    
     std::vector<double> params_r_flat__(num_params_r__);
     Eigen::Index size_iter__ = 0;
     Eigen::Index flat_iter__ = 0;
     for (auto&& param_name__ : names__) {
       const auto param_vec__ = context.vals_r(param_name__);
       for (Eigen::Index i = 0; i < num_params__[size_iter__]; ++i) {
         params_r_flat__[flat_iter__] = param_vec__[i];
         ++flat_iter__;
       }
       ++size_iter__;
     }
     vars.resize(params_r_flat__.size());
     transform_inits_impl(params_r_flat__, params_i, vars, pstream__);
    } // transform_inits() 
    
};
}
using stan_model = normal_test_model_namespace::normal_test_model;

#ifndef USING_R

// Boilerplate
stan::model::model_base& new_model(
        stan::io::var_context& data_context,
        unsigned int seed,
        std::ostream* msg_stream) {
  stan_model* m = new stan_model(data_context, seed, msg_stream);
  return *m;
}

stan::math::profile_map& get_stan_profile_data() {
  return normal_test_model_namespace::profiles__;
}

#endif

EDIT: @maxbiostat struggled to get the syntax highlighting to work in this post.