R package compile problems on certain platforms

This is pretty specific and @bgoodri may be the only one who can help.
R package ctsem has caught the Ripley virus and I’ve been given very limited time to resolve every CRAN check error shown here: https://cran.r-project.org/web/checks/check_results_ctsem.html

Basically I’m having trouble building on:
r-devel-linux-x86_64-debian-clang
r-patched-linux-x86_64
r-patched-solaris-x86

I’m at a bit of a loss and pretty ready to support CRAN alternatives or therapy sessions, but maybe somebody can help…

I had similar trouble a while ago with OncoBayes2. Look at the change logs from the R package

https://cran.r-project.org/web/packages/OncoBayes2/index.html

there you see that I had to do multiple attempts to fix the sanitiser issues. You should compare the stan model between the releases where it was fixed - it’s some problematic indexing stuff. There is an issue in stan somewhere, but we have not yet gotten to the bottom of it.

So basically slicing with indices into data-structures is the root cause which you have to avoid somehow.

Thanks… are you saying that any references of e.g. matrices like:
mymatrix[intarray, intarray]
is going to be problematic? If so then I guess I should give up now…

avoiding these resolved my problem, yes. You should look up the issue for this for more details (sorry, I don’t find it right now).

I’ve got to do a presentation in an hour but I will help figure it out this afternoon.

2 Likes

‘Maybe’ I found a problem – instead of the following in makevars:
-DBOOST_MATH_OVERFLOW_ERROR_POLICY=errno_on_error
I had the windows argument:
-DRCPP_PARALLEL_USE_TBB=1

The original form compiled on my ubuntu with gcc and clang, but it’s my only guess / hope so far.

You can have both and probably should. Are we talking about

https://www.r-project.org/nosvn/R.check/r-patched-solaris-x86/ctsem-00install.html

or something else?

There is also:
https://www.r-project.org/nosvn/R.check/r-devel-linux-x86_64-debian-clang/ctsem-00install.html
https://www.r-project.org/nosvn/R.check/r-patched-linux-x86_64/ctsem-00install.html

Those look like it either timed out or ran out of RAM. One thing that I think all of the R packages can move toward is having a separate generated quantities block that is drawn from by calling rstan::gqs. Have you worked with standalone generated quantities before?

No, but I already compile a separate object for the complex generated quantities… I notice when compiling with clang 10 on Ubuntu that I am getting a hang when anything greater than -O1 is set, though this doesn’t seem to be the case on CRAN…

Yes actually it seems to hang at the same point where it is killed on the debian clang box, when I use -O2 or greater with clang on ubuntu:

* installing *source* package ‘ctsem’ ...
** using staged installation
** libs
"/usr/lib/R/bin/Rscript" -e "source(file.path('..', 'tools', 'make_cc.R')); make_cc(commandArgs(TRUE))" stan_files/ctsm.stan
Wrote C++ file "stan_files/ctsm.cc"


clang++ -std=gnu++14 -I"/usr/share/R/include" -DNDEBUG -I"../inst/include" -I"/home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src" -DBOOST_DISABLE_ASSERTS -DEIGEN_NO_DEBUG -DBOOST_MATH_OVERFLOW_ERROR_POLICY=errno_on_error -I'/home/driver/R/x86_64-pc-linux-gnu-library/4.0/BH/include' -I'/home/driver/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include' -I'/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include' -I'/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppParallel/include' -I'/home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include' -I'/home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include'    -I'/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppParallel/include' -D_REENTRANT -DSTAN_THREADS   -fpic  -mtune=native -march=native -fPIC -fpic  -O2 -Wall  -c stan_files/ctsm.cc -o stan_files/ctsm.o
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:3:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:535:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:10:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:3:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:2:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/LU:47:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:10:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:3:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:3:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Cholesky:12:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Jacobi:29:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:10:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:3:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:3:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Cholesky:43:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:10:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:3:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/QR:17:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Householder:27:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:10:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:3:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:5:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/SVD:48:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:10:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:3:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:6:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Geometry:58:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:10:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:3:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:7:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Eigenvalues:58:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:10:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:3:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:14:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Sparse:26:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/SparseCore:66:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:10:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:3:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:14:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Sparse:27:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/OrderingMethods:71:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:10:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:3:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:14:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Sparse:29:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/SparseCholesky:43:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:10:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:3:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:14:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Sparse:32:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/SparseQR:35:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:10:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:3:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:14:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Sparse:33:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/IterativeLinearSolvers:46:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:10:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/stan_fit.hpp:22:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/RcppEigen.h:25:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/RcppEigenForward.h:32:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/CholmodSupport:45:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:10:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/stan_fit.hpp:22:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/RcppEigen.h:25:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/RcppEigenForward.h:35:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported/Eigen/KroneckerProduct:34:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported/Eigen/../../Eigen/src/Core/util/ReenableStupidWarnings.h:10:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/stan_fit.hpp:22:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/RcppEigen.h:25:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/RcppEigenForward.h:39:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported/Eigen/Polynomials:135:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported/Eigen/../../Eigen/src/Core/util/ReenableStupidWarnings.h:10:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/stan_fit.hpp:22:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/RcppEigen.h:25:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/RcppEigenForward.h:40:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported/Eigen/SparseExtra:51:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported/Eigen/../../Eigen/src/Core/util/ReenableStupidWarnings.h:10:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
    #pragma clang diagnostic pop
                             ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/stan_fit.hpp:35:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/services/diagnose/diagnose.hpp:10:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/model/test_gradients.hpp:7:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/model/log_prob_grad.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/mat.hpp:12:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat.hpp:283:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/prob/gaussian_dlm_obs_rng.hpp:138:7: warning: unused variable 'n' [-Wunused-variable]
  int n = G.rows();  // number of states
      ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/stan_fit.hpp:35:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/services/diagnose/diagnose.hpp:10:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/model/test_gradients.hpp:7:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/model/log_prob_grad.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/mat.hpp:17:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/mat/fun/columns_dot_self.hpp:8:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/mat/fun/dot_self.hpp:38:41: warning: all paths through this function will call itself [-Winfinite-recursion]
  inline static double square(double x) { return square(x); }
                                        ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/stan_fit.hpp:35:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/services/diagnose/diagnose.hpp:10:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/model/test_gradients.hpp:7:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/model/log_prob_grad.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/mat.hpp:70:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/mat/functor/integrate_ode_adams.hpp:5:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/mat/functor/cvodes_integrator.hpp:126:18: warning: unused variable 'coupled_size' [-Wunused-variable]
    const size_t coupled_size = cvodes_data.coupled_ode_.size();
                 ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/stan_fit.hpp:35:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/services/diagnose/diagnose.hpp:10:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/model/test_gradients.hpp:7:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/model/log_prob_grad.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/mat.hpp:6:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/core.hpp:46:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/core/set_zero_all_adjoints.hpp:14:13: warning: unused function 'set_zero_all_adjoints' [-Wunused-function]
static void set_zero_all_adjoints() {
            ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/stan_fit.hpp:35:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/services/diagnose/diagnose.hpp:10:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/model/test_gradients.hpp:7:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/model/log_prob_grad.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/mat.hpp:6:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/core.hpp:47:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/core/set_zero_all_adjoints_nested.hpp:17:13: warning: 'static' function 'set_zero_all_adjoints_nested' declared in header file should be declared 'static inline' [-Wunneeded-internal-declaration]
static void set_zero_all_adjoints_nested() {
            ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/stan_fit.hpp:35:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/services/diagnose/diagnose.hpp:10:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/model/test_gradients.hpp:7:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/model/log_prob_grad.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/mat.hpp:12:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat.hpp:278:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/prob/dirichlet_log.hpp:5:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/prob/dirichlet_lpmf.hpp:60:9: warning: unused type alias 'T_partials_vec' [-Wunused-local-typedef]
  using T_partials_vec = typename Eigen::Matrix<T_partials_return, -1, 1>;
        ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/stan_fit.hpp:35:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/services/diagnose/diagnose.hpp:10:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/model/test_gradients.hpp:7:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/model/log_prob_grad.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/mat.hpp:12:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat.hpp:328:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/prob/poisson_log_glm_log.hpp:5:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/prob/poisson_log_glm_lpmf.hpp:52:9: warning: unused type alias 'T_alpha_val' [-Wunused-local-typedef]
  using T_alpha_val = typename std::conditional_t<
        ^
In file included from stan_files/ctsm.cc:3:
In file included from stan_files/ctsm.hpp:18:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/rstaninc.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/rstan/include/rstan/stan_fit.hpp:35:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/services/diagnose/diagnose.hpp:10:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/model/test_gradients.hpp:7:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/stan/model/log_prob_grad.hpp:4:
In file included from /home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/mat.hpp:51:
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/mat/fun/squared_distance.hpp:27:11: warning: unused type alias 'idx_t' [-Wunused-local-typedef]
    using idx_t = typename index_type<matrix_v>::type;
          ^
/home/driver/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/rev/mat/fun/squared_distance.hpp:64:11: warning: unused type alias 'idx_t' [-Wunused-local-typedef]
    using idx_t = typename index_type<matrix_d>::type;
          ^
In file included from stan_files/ctsm.cc:3:
stan_files/ctsm.hpp:1712:24: warning: unused typedef 'local_scalar_t__' [-Wunused-local-typedef]
        typedef double local_scalar_t__;
                       ^

clang++ is usually faster to compile but it can sometimes be slower on giant files. For me, clang++-10 eventually installs ctsem under -O3 but ctsem.o is 11MB. The largest one in rstanarm is about 7.5 MB.

What object were you referring to when you said you had a separate generated quantities?

ok. ctsmgen.stan is used for prior / posterior checks. ctsm.stan (the main file) contains about 60 lines of generated quantities.

OK. Let me see if I can put together a PR.

gosh. good luck :/ and thanks!

I started to do this on the gqs branch that I just invited you to on GitHub. At this point, you could get the generated quantities that were formerly in the ctsm.stan file by calling

gqs(stanmodels$ctsm_gqs, data_list, draws = as.matrix(post))

where post is produced by sampling(stanmodels$ctsm, data_list, ...).

The next step I am afraid I will mess up without understanding the details of the model, but basically everything that is in the transformed parameters block of ctsm.stan that isn’t needed to evaluate the posterior kernel needs to get moved into the generated quantities block of ctsm_gqs.stan (and then deleted from its parameters block). This way, ctsm.stan will get smaller and ctsm_gqs.stan will get bigger but the compilation should be easier.

Ok. I am working on condensing some things. I can’t really see the more major step you propose working though – yes there is a bunch of stuff that can be taken out of transformed pars and put in gen quantities, but gen quantities then depends on nearly everything else in the model anyway so would have to get very large. I’ll think on it and try my luck I guess. Thanks.

That is the idea. But the things that are in the transformed parameters block of ctsm.stan don’t have to be recalculated for ctsm_gqs.stan. You can just pass them back their realizations as part of that big matrix called draws in rstan::gqs. So, it is mostly just a manner of redistributing the code across ctsm.stan and ctsm_gqs.stan so that each is bite-sized.

Sure. But I’m not storing all the process / measurement matrices that may change at each time point, because that generates headaches of it’s own, I think – and they all have to be regenerated to compute the interesting quantities. No doubt some improvement is possible, I just don’t think it’s so straightforward unfortunately :/

It isn’t so straightforward, which is why I am scared to do it. But try it and then I’ll run over it again.

2 Likes