Compile errors with stanc3 for higher-order functions

charlesm93 · July 20, 2020, 12:20am

I’ve been getting odd compilation errors with cmdstanr when using higher-order functions. I just downloaded a new version of cmdstan today.

First with integrate_ode_rk45, I get the following message:

> mod <- cmdstan_model(paste0("model/", modelName))
Compiling Stan program...
Uncaught exception:
  
  (Invalid_argument "length mismatch in map2_exn: 5 <> 4 ")

Raised at file "stdlib.ml", line 34, characters 20-45
Called from file "src/list.ml", line 130, characters 13-76
Called from file "src/list.ml" (inlined), line 378, characters 2-36
Called from file "src/middle/UnsizedType.ml", line 68, characters 13-165
Called from file "src/middle/UnsizedType.ml", line 86, characters 12-68
Called from file "list.ml", line 134, characters 35-42
Called from file "src/list0.ml" (inlined), line 32, characters 40-75
Called from file "src/list.ml" (inlined), line 373, characters 27-49
Called from file "src/list.ml" (inlined), line 379, characters 2-18
Called from file "src/middle/UnsizedType.ml", line 84, characters 7-212
Called from file "src/list.ml", line 204, characters 19-22
Called from file "src/list.ml" (inlined), line 209, characters 22-39
Called from file "src/middle/Stan_math_signatures.ml", line 249, characters 4-128
Called from file "src/frontend/Semantic_check.ml", line 322, characters 4-121
Called from file "src/frontend/Semantic_check.ml", line 334, characters 8-38
Called from file "src/frontend/Semantic_check.ml", line 695, characters 4-57
Called from file "src/frontend/Semantic_check.ml", line 931, characters 12-51
Called from file "src/frontend/Semantic_check.ml", line 1406, characters 9-80
Called from file "src/frontend/Semantic_check.ml", line 1434, characters 4-58
Called from file "src/frontend/Semantic_check.ml", line 1667, characters 18-51
Called from file "list.ml", line 117, characters 24-34
Called from file "src/list0.ml" (inlined), line 22, characters 40-81
Called from file "src/frontend/Semantic_check.ml", line 1666, characters 6-128
Called from file "src/frontend/Semantic_check.ml", line 1728, characters 13-63
Called from file "src/frontend/Frontend_utils.ml", line 24, characters 10-51
Called from file "src/stanc/stanc.ml", line 187, characters 18-53
Called from file "src/stanc/stanc.ml", line 244, characters 9-16
make: *** No rule to make target `/var/folders/ty/36ws994x4l33ksszcdwjtrv40000gp/T/RtmpEluPgF/model-47401e5c31e2.d', needed by `/var/folders/ty/36ws994x4l33ksszcdwjtrv40000gp/T/RtmpEluPgF/model-47401e5c31e2'.  Stop.
Error in processx::run(command = make_cmd(), args = c(tmp_exe, include_paths,  : 
  System command 'make' failed, exit status: 2, stderr (last 10 lines):
E> Called from file "src/frontend/Semantic_check.ml", line 1434, characters 4-58
E> Called from file "src/frontend/Semantic_check.ml", line 1667, characters 18-51
E> Called from file "list.ml", line 117, characters 24-34
E> Called from file "src/list0.ml" (inlined), line 22, characters 40-81
E> Called from file "src/frontend/Semantic_check.ml", line 1666, characters 6-128
E> Called from file "src/frontend/Semantic_check.ml", line 1728, characters 13-63
E> Called from file "src/frontend/Frontend_utils.ml", line 24, characters 10-51
E> Called from file "src/stanc/stanc.ml", line 187, characters 18-53
E> Called from file "src/stanc/stanc.ml", line 244, characters 9-16
E> make: *** No rule to make target `/var/folders/ty/36ws994x4l33ksszcdwjtrv40000gp/T/RtmpEluPgF/model-47401e5c31e2.d', needed by `/var/folders/ty/36ws994x4l33ksszcdwjtrv40000gp
[...]
Type .Last.error.trace to see where the error occured

Really not quite sure what causes this error. Also, I don’t think the unit tests would catch this.

Next, when I try using a prototype function, laplace_marginal_poisson_log(), from the branch try_laplace-approximation2 (note the 2) I get the following:

../model/disease_map_ela.hpp:422:19: error: no matching function for call to
      'laplace_marginal_poisson_log'
        density = laplace_marginal_poisson_log<false>(y, n_samples, ye,
                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../model/disease_map_ela.hpp:661:14: note: in instantiation of function template
      specialization
      'disease_map_ela_model_namespace::disease_map_ela_model::log_prob<false,
      false, double>' requested here
      return log_prob<propto__,jacobian__,T_>(vec_params_r, vec_params_i...
             ^
/Users/charlesm/Desktop/Code/laplace_approximation/writeup/StanCon2020/script/cmdstan/stan/src/stan/model/model_base_crtp.hpp:90:50: note: 
      in instantiation of function template specialization
      'disease_map_ela_model_namespace::disease_map_ela_model::log_prob<false,
      false, double>' requested here
    return static_cast<const M*>(this)->template log_prob<false, false, double>(
                                                 ^
../model/disease_map_ela.hpp:162:3: note: in instantiation of member function
      'stan::model::model_base_crtp<disease_map_ela_model_namespace::disease_map_ela_model>::log_prob'
      requested here
  ~disease_map_ela_model() final { }
  ^
/Users/charlesm/Desktop/Code/laplace_approximation/writeup/StanCon2020/script/cmdstan/stan/lib/stan_math/stan/math/laplace/laplace_marginal_poisson_log.hpp:39:6: note: 
      candidate template ignored: invalid explicitly-specified argument for
      template parameter 'T0'
  T1 laplace_marginal_poisson_log
     ^
/Users/charlesm/Desktop/Code/laplace_approximation/writeup/StanCon2020/script/cmdstan/stan/lib/stan_math/stan/math/laplace/laplace_marginal_poisson_log.hpp:58:6: note: 
      candidate template ignored: invalid explicitly-specified argument for
      template parameter 'T0'
  T1 laplace_marginal_poisson_log

A look at the created C++ file shows the call to laplace_marginal_poisson_log is the following

density = laplace_marginal_poisson_log<false>(y, n_samples, ye,
                    K_functor__(), phi, x, delta, delta_int, theta_0,
                    pstream__);

Everything here is fine except the <false> part. The C++ code I wrote is not designed to handle such a boolean. The code exposing the function to the parser is at line 1600 of stan_math_signature.ml. The unit tests I wrote pass, but obviously they wouldn’t catch such a mistranslation.

rok_cesnovar · July 20, 2020, 3:33am

Huh, with the integrate one you have probably uncovered a bug that was maybe added with the variadic ODE PR that was merged today.

I will look into this. it would really help if could you post a minimal example where this fails.
Thanks!

EDIT: I tried all the test models we have for the old integrate interface and they all compile fine, is this maybe on your branch? If not, I would need a minimal example.

EDIT2: I took the liberty of changing the thread title to reflect the issue more (its not something cmdstanr specific.Hope you dont mind.

rok_cesnovar · July 20, 2020, 3:48am

For the laplace one, the issue is that for laplace the propto template parameter is not used and should not get generated. The logic to handle that is here:

github.com

stan-dev/stanc3/blob/3960cc8265c79b3c420a4f549b6244483835411d/src/stan_math_backend/Expression_gen.ml#L128


  ends_with_any user_dist_suffices s
  && not (ends_with_any ["_cdf_log"; "_ccdf_log"] s)

let is_user_lp s = ends_with "_lp" s

let suffix_args f =
  if ends_with "_rng" f then ["base_rng__"]
  else if ends_with "_lp" f then ["lp__"; "lp_accum__"]
  else []

let demangle_propto_name udf f =
  if f = "multiply_log" || f = "binomial_coefficient_log" then f
  else if Utils.is_propto_distribution f then
    Utils.stdlib_distribution_name f ^ "<propto__>"
  else if
    Utils.is_distribution_name f || (udf && (is_user_dist f || is_user_lp f))
  then f ^ "<false>"
  else f

let fn_renames =
  List.map

Since it has the _log suffix that was used for distributions, the parser treats it like any distibution and prepends the template parameter. If the _log suffix will stay, this should not be handled as a distribution I guess: https://github.com/stan-dev/stanc3/blob/master/src/middle/Utils.ml#L16

Something along the lines of:

let is_distribution_name s =
  (not
     ( String.is_suffix s ~suffix:"_cdf_log"
     || String.is_suffix s ~suffix:"_ccdf_log" 
     || String.equal s "laplace_marginal_poisson_log"))
  && List.exists
       ~f:(fun suffix -> String.is_suffix s ~suffix)
       distribution_suffices

Best solution would be if the functon would not end on _log if possible.

~~This was an oversight in the laplace PR.~~ (Ignore me, this is on a branch)
This would get caught if we had tests I am proposing here: https://github.com/stan-dev/stanc3/issues/636
(Sorry for taking this opportunity for pushing my agenda here, but I think its important we add this).
That or if the model was also added to the stan repo. But I would prefer not to duplicate stuff.

nhuurre · July 20, 2020, 9:07am

Nope, that is an older bug. The error points here

github.com

stan-dev/stanc3/blob/3960cc8265c79b3c420a4f549b6244483835411d/src/middle/UnsizedType.ml#L74


      
          
          let check_of_same_type_mod_conv name t1 t2 =
            if String.is_prefix name ~prefix:"assign_" then t1 = t2
            else
              match (t1, t2) with
              | UReal, UInt -> true
              | UFun (l1, rt1), UFun (l2, rt2) ->
                  rt1 = rt2
                  && List.for_all
                       ~f:(fun x -> x = true)
                       (List.map2_exn
                          ~f:(fun (at1, ut1) (at2, ut2) ->
                            ut1 = ut2 && autodifftype_can_convert at2 at1 )
                          l1 l2)
              | _ -> t1 = t2
          
          let rec check_of_same_type_mod_array_conv name t1 t2 =
            match (t1, t2) with
            | UArray t1elt, UArray t2elt ->
                check_of_same_type_mod_array_conv name t1elt t2elt
            | _ -> check_of_same_type_mod_conv name t1 t2

It was fixed on the closures branch.

github.com

nhuurre/stanc3/blob/610ed76f63cb71a4a54887787530e7e9b92dee71/src/middle/UnsizedType.ml#L70


      
          
          let check_of_same_type_mod_conv name t1 t2 =
            if String.is_prefix name ~prefix:"assign_" then t1 = t2
            else
              match (t1, t2) with
              | UReal, UInt -> true
              | UFun (l1, rt1, _), UFun (l2, rt2, _) -> (
                  rt1 = rt2
                  &&
                  match
                    List.for_all2
                      ~f:(fun (at1, ut1) (at2, ut2) ->
                        ut1 = ut2 && autodifftype_can_convert at2 at1 )
                      l1 l2
                  with
                  | List.Or_unequal_lengths.Ok ok -> ok
                  | Unequal_lengths -> false )
              | _ -> t1 = t2
          
          let rec check_of_same_type_mod_array_conv name t1 t2 =
            match (t1, t2) with

rok_cesnovar · July 20, 2020, 9:14am

Oh, thanks! ~~When does it come up?~~ Nevermind, I can see where it might come up.

nhuurre · July 20, 2020, 9:23am

The bug is that when checking if a function signature matches what a higher-order-function expects it only checks that each argument has the expected type but crashes if the number of arguments does not match. integrate_ode_rk45 takes an ODE system that has five input arguments (time, state, params, real data, int data). Presumably @charlesm93 defined an ODE system function that only takes four arguments.

charlesm93 · July 20, 2020, 1:35pm

Thanks everyone for the help.

laplace_marginal_poisson_log is really the name I’d like to use to be consistent with the density named poisson_log. Can I create an exception for certain functions? If needed, I’ll change the name for now (my goal being to have something usable for StanCon, i.e. August 13th).

@nhuurre Your analysis is correct. The function I passed had the wrong number of arguments and the cmdstan version I used was outdated.

rok_cesnovar · July 20, 2020, 3:38pm

Will this be used as laplace_marginal_poisson_log_lpmf similar to poisson_log_lpmf?

charlesm93 · July 20, 2020, 7:34pm

No. It isn’t an lpmf in the traditional sense, because the data isn’t passed as a single object (scalar or vector). Rather, the users passes sufficient statistics.

So no laplace_marginal_poisson_log_lpmf.

rok_cesnovar · July 20, 2020, 8:25pm

Ok. Then if its not intended for use with the tilde statement, we can special case it in is_distribution in Utils.ml I think that should cover everything.

charlesm93 · July 21, 2020, 1:05pm

Sounds good. I’ll take a look at the file.

rok_cesnovar · July 21, 2020, 1:24pm

Sorry, I was responding on a phone yesterday, so didnt go into detail.

Basically this: https://github.com/stan-dev/stanc3/blob/master/src/middle/Utils.ml#L16

Should be:

let is_distribution_name s =
  (not
     ( String.is_suffix s ~suffix:"_cdf_log"
     || String.is_suffix s ~suffix:"_ccdf_log" 
     || String.equal s "laplace_marginal_poisson_log" ))
  && List.exists
       ~f:(fun suffix -> String.is_suffix s ~suffix)
       distribution_suffices

or something close to that.

Topic		Replies	Views
Compiling error cmdstan example code CmdStan	14	1394	November 10, 2023
Cmdstanr compilation error with no detailed message General fitting-issues	13	1477	March 1, 2021
Compilation error - how to debug it? CmdStan matrix	3	778	February 10, 2022
CmdStanR throws error on compiling model General compiler	7	1267	July 22, 2020
12 warnings and 14 errors generated with Stan CmdStan cmdstanr	7	541	November 13, 2023

Compile errors with stanc3 for higher-order functions

Related topics