Trouble including an integrand as part of the model

I’m trying to estimate a model in which the DGP involves integrating a custom function. When I try to compile the model (below), I get a Error in sink(type = "output") : invalid connection error (full error message below). The issue is very clearly coming from the call to integrate_1d().

Sample model below. Anyone have an idea of what might be going wrong?


functions{
  real cost_density(real x,          // Function argument
                    real xc,         // Complement of function argument
                                     //  on the domain (defined later)
                    real[] theta,    // parameters
                    real[] x_r,      // data (real)
                    int[] x_i) {     // data (integer)
  real mu = theta[1];
  real sigma = theta[2];
  real max_c = theta[3];
  int num_bidders = x_i[1];
  real out;

  out = (num_bidders - 1)*(lognormal_lcdf(x | mu, sigma) - lognormal_lcdf(max_c | mu, sigma));
  return(exp(out));
  }
  
  vector getGenerativeTotalAuctionHazard(int N,
                               int[] J, // num bidders in each auction
                               vector costs,
                               matrix X,
                               vector beta_costs, 
                               real sigma,
                               real c_max
                               ){
        vector[N] mus;
        vector[N] numerators;
        vector[N] denominators;
        vector[N] out;

        mus = X * beta_costs;
        
        // NOTE: This assumes homogeneous bidders so that g_j(b_i) = g(b_i | X_j) = g(b_i | X_i) = g_i(b_i)
        for (n in 1:N){
          
          numerators[n] = integrate_1d(cost_density,
                             0.0, // left limit
                             c_max, // right limit
                             { mus[n], sigma, costs[n] }, //theta
                             { 0.0 }, //x_r
                             { J[n] }, //x_i
                             0.001
                             );
          
          denominators[n] = exp(
                                (J[n] - 1)*(lognormal_lcdf(costs[n] | mus[n], sigma) - lognormal_lcdf(c_max | mus[n], sigma))
                                );
          

          out[n] = numerators[n] / denominators[n];
        }
        
        return(out);
   }
 
}

data {
  int<lower=0> N; // Number of auction-bidder obs
  int<lower=0> K; // number of auction-bidder feaures
  int J[N]; // Number of bidders in each auction
  matrix[N, K] X; // Auction-bidder features
  vector[N] bids;
}

parameters {
  vector[K] beta;
  real<lower=0> sigma;
  real<lower = 0> c_max;
  vector[N] costs;
}

model {
  vector[N] inv_hazards;
  //prior
  to_vector(beta) ~ normal(0,1);
  sigma ~ normal(1,0.01);
  c_max ~ normal(3, 1);
  
  //likelihood
  for(i in 1:N){
    costs[i] ~ normal(X[i] * beta, sigma) T[,c_max]; //empirical bid distribution
  }
  // 
  inv_hazards = getGenerativeTotalAuctionHazard(
                           N,
                           J, // num bidders in each auction
                           costs,
                           X,
                           beta,
                           sigma,
                           c_max
                           );
   (bids - costs) ~ normal(inv_hazards , 1);

}

Full error message:

Error in compileCode(f, code, language = language, verbose = verbose) : 
  Compilation ERROR, function(s)/method(s) not created! In file included from file3d89436904cd.cpp:8:
In file included from /Library/Frameworks/R.framework/Versions/3.6/Resources/library/StanHeaders/include/src/stan/model/model_header.hpp:4:
In file included from /Library/Frameworks/R.framework/Versions/3.6/Resources/library/StanHeaders/include/stan/math.hpp:4:
In file included from /Library/Frameworks/R.framework/Versions/3.6/Resources/library/StanHeaders/include/stan/math/rev/mat.hpp:4:
In file included from /Library/Frameworks/R.framework/Versions/3.6/Resources/library/StanHeaders/include/stan/math/rev/core.hpp:5:
In file included from /Library/Frameworks/R.framework/Versions/3.6/Resources/library/StanHeaders/include/stan/math/rev/core/build_vari_array.hpp:4:
In file included from /Library/Frameworks/R.framework/Versions/3.6/Resources/library/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:4:
In file included from /Library/Frameworks/R.framework/Versions/3.6/Resources/l
In addition: Warning message:
In system(cmd, intern = !verbose) :
  running command '/Library/Frameworks/R.framework/Resources/bin/R CMD SHLIB file3d89436904cd.cpp 2> file3d89436904cd.cpp.err.txt' had status 1
Error in sink(type = "output") : invalid connection
1 Like

You need to call it with verbose = TRUE to see the actual compiler error.

Ah! Error below:

/Library/Frameworks/R.framework/Versions/3.6/Resources/library/StanHeaders/include/stan/math/rev/arr/functor/integrate_1d.hpp:120:1: note: candidate function template not viable: no known conversion from 'vector<local_scalar_t__>' to 'const vector<double>' for 5th argument
integrate_1d(const F &f, const T_a &a, const T_b &b,
^
/Library/Frameworks/R.framework/Versions/3.6/Resources/library/StanHeaders/include/stan/math/prim/arr/functor/integrate_1d.hpp:195:15: note: candidate function template not viable: no known conversion from 'const stan::math::var' to 'const double' for 3rd argument
inline double integrate_1d(
              ^
file4f60bc4249b.cpp:535:45: error: no matching function for call to 'getGenerativeTotalAuctionHazard'
            stan::math::assign(inv_hazards, getGenerativeTotalAuctionHazard(N, J, costs, X, beta, sigma, c_max, pstream__));
                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Library/Frameworks/R.framework/Versions/3.6/Resources/library/StanHeaders/include/src/stan/model/log_prob_grad.hpp:44:28: note: in instantiation of function template specialization 'model4f60717efaac_generative_model_namespace::model4f60717efaac_generative_model::log_prob<true, false, stan::math::var>' requested here
          = model.template log_prob<propto, jacobian_adjust_transform>
                           ^
/Library/Frameworks/R.framework/Versions/3.6/Resources/library/StanHeaders/include/src/stan/services/util/initialize.hpp:166:31: note: in instantiation of function template specialization 'stan::model::log_prob_grad<true, false, model4f60717efaac_generative_model_namespace::model4f60717efaac_generative_model>' requested here
      log_prob = stan::model::log_prob_grad<true, Jacobian>
                              ^
/Library/Frameworks/R.framework/Versions/3.6/Resources/library/StanHeaders/include/src/stan/services/optimize/newton.hpp:54:21: note: in instantiation of function template specialization 'stan::services::util::initialize<false, model4f60717efaac_generative_model_namespace::model4f60717efaac_generative_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014, 0, 2147483563>, boost::random::linear_congruential_engine<unsigned int, 40692, 0, 2147483399> > >' requested here
            = util::initialize<false>(model, init, rng, init_radius, false,
                    ^
/Library/Frameworks/R.framework/Versions/3.6/Resources/library/rstan/include/rstan/stan_fit.hpp:488:35: note: in instantiation of function template specialization 'stan::services::optimize::newton<model4f60717efaac_generative_model_namespace::model4f60717efaac_generative_model>' requested here
      = stan::services::optimize::newton(model, *init_context_ptr,
                                  ^
/Library/Frameworks/R.framework/Versions/3.6/Resources/library/rstan/include/rstan/stan_fit.hpp:1201:11: note: in instantiation of function template specialization 'rstan::(anonymous namespace)::command<model4f60717efaac_generative_model_namespace::model4f60717efaac_generative_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014, 0, 2147483563>, boost::random::linear_congruential_engine<unsigned int, 40692, 0, 2147483399> > >' requested here
    ret = command(args, model_, holder, names_oi_tidx_,
          ^
file4f60bc4249b.cpp:741:140: note: in instantiation of member function 'rstan::stan_fit<model4f60717efaac_generative_model_namespace::model4f60717efaac_generative_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014, 0, 2147483563>, boost::random::linear_congruential_engine<unsigned int, 40692, 0, 2147483399> > >::call_sampler' requested here
            &rstan::stan_fit<model4f60717efaac_generative_model_namespace::model4f60717efaac_generative_model, boost::random::ecuyer1988>::call_sampler)
                                                                                                                                           ^
file4f60bc4249b.cpp:108:1: note: candidate template ignored: substitution failure [with T2__ = stan::math::var, T3__ = double, T4__ = stan::math::var, T5__ = stan::math::var, T6__ = stan::math::var]
getGenerativeTotalAuctionHazard(const int& N,
^
In file included from file4f60bc4249b.cpp:730:
In file included from /Library/Frameworks/R.framework/Versions/3.6/Resources/library/rstan/include/rstan/rstaninc.hpp:3:
In file included from /Library/Frameworks/R.framework/Versions/3.6/Resources/library/rstan/include/rstan/stan_fit.hpp:36:
In file included from /Library/Frameworks/R.framework/Versions/3.6/Resources/library/StanHeaders/include/src/stan/services/optimize/bfgs.hpp:11:
In file included from /Library/Frameworks/R.framework/Versions/3.6/Resources/library/StanHeaders/include/src/stan/optimization/bfgs.hpp:5:
/Library/Frameworks/R.framework/Versions/3.6/Resources/library/StanHeaders/include/src/stan/model/log_prob_propto.hpp:45:28: error: no matching member function for call to 'log_prob'
          = model.template log_prob<true, jacobian_adjust_transform>
            ~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Library/Frameworks/R.framework/Versions/3.6/Resources/library/rstan/include/rstan/stan_fit.hpp:1124:40: note: in instantiation of function template specialization 'stan::model::log_prob_propto<false, model4f60717efaac_generative_model_namespace::model4f60717efaac_generative_model>' requested here
        return Rcpp::wrap(stan::model::log_prob_propto<false>(model_, par_r, par_i, &rstan::io::rcout));
                                       ^
file4f60bc4249b.cpp:759:140: note: in instantiation of member function 'rstan::stan_fit<model4f60717efaac_generative_model_namespace::model4f60717efaac_generative_model, boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014, 0, 2147483563>, boost::random::linear_congruential_engine<unsigned int, 40692, 0, 2147483399> > >::log_prob' requested here
            &rstan::stan_fit<model4f60717efaac_generative_model_namespace::model4f60717efaac_generative_model, boost::random::ecuyer1988>::log_prob)
                                                                                                                                           ^
file4f60bc4249b.cpp:464:9: note: candidate template ignored: substitution failure [with propto__ = true, jacobian__ = false, T__ = stan::math::var]
    T__ log_prob(std::vector<T__>& params_r__,
        ^
file4f60bc4249b.cpp:550:8: note: candidate function template not viable: requires at most 2 arguments, but 3 were provided
    T_ log_prob(Eigen::Matrix<T_,Eigen::Dynamic,1>& params_r,
       ^
18 warnings and 3 errors generated.
make: *** [file4f60bc4249b.o] Error 1

ERROR(s) during compilation: source code errors or compiler configuration errors!

shosh.stan (3.0 KB) compiles. The parser shouldn’t be generating C++ code that can’t compile, but it has been known to happen. Thus, I ended up having to pass x_r from transformed data to your function with the data argument qualifier to get it to work.

2 Likes

Thanks very much, Ben :)