How to update a package that has StanHeaders and rstan in its LinkingTo

This thread is for maintainers of R packages that ship with C++ code generated by Stan. Due to the move to C++14 by Stan, many such packages need to get updated ASAP because they currently do not build from source. I will make PRs on GitHub for the applicable packages, but the basic idea in each case is to install rstan 2.18.1 from CRAN and then in your package make the following changes:

  • In DESCRIPTION,

    • Under Depends: edit it to R >= 3.4.0, Rcpp (>= 0.12.0), methods (plus perhaps other packages)
    • Under LinkingTo: edit it to StanHeaders (>= 2.18.0), rstan (>= 2.18.1), BH (>= 1.66.0), Rcpp (>= 0.12.0), RcppEigen (>= 0.3.3.3.0)
    • Under Imports: edit it to rstan (>= 2.18.1) (plus perhaps other packages
    • Increase the Version number and Date
  • For src/Makevars and src/Makevars.win, see How to update a package that has StanHeaders and rstan in its LinkingTo, part II

Then, rebuild your package (making sure the C++ code gets regenerated) and upload it to CRAN ASAP to avoid having your package kicked out.

Be on the lookout for PRs, and post on this thread if you have questions.

P.S. If you are tempted to write C++ code by hand that utilizes integrate_1d in Stan Math, be aware that it will fail on Solaris until Boost 1.69 is released. The stanc executable currently will not let you utilize integrate_1d in a Stan program, so this is unlikely to be a problem in practice until Stan 2.19 is released.

3 Likes

Has this been checked on Windows? I get

installing the source package ‘rstan’

trying URL 'https://cran.rstudio.com/src/contrib/rstan_2.18.1.tar.gz'
Content type 'application/x-gzip' length 833755 bytes (814 KB)
downloaded 814 KB

* installing *source* package 'rstan' ...
** Paket 'rstan' erfolgreich entpackt und MD5 Summen überprüft
** libs
Fehler in .shlib_internal(args) : 
  C++14 standard requested but CXX14 is not defined
* removing 'C:/R/library/rstan'
* restoring previous 'C:/R/library/rstan'
In R CMD INSTALL
Warning in install.packages :
  installation of package ‘rstan’ had non-zero exit status

I have seen https://github.com/stan-dev/rstantools/issues/40, following @bgoodri it should be fine once 2.18.1 is approved.

I think you need to make your R installation aware of C++14. @bgoodri has described it a few times on discourse here. It boils down to having to edit your ~/.R/Makevars

Ok, this works. I had removed it because in

there is no CXX14 = g++

1 Like

It sort of exists

Ubuntu

CXX14 = g++ # g++ or clang++ if you have that
CXX14FLAGS = -O3

in user Makevars. Project Makevars copied from rstanarm.
After a lengthy list of warnings:

In file included from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/mat/prob/bernoulli_logit_glm_lpmf.hpp:15:0,
                 from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/mat.hpp:267,
                 from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/mat.hpp:12,
                 from /usr/local/lib/R/site-library/StanHeaders/include/src/stan/model/log_prob_grad.hpp:4,
                 from /usr/local/lib/R/site-library/StanHeaders/include/src/stan/model/test_gradients.hpp:7,
                 from /usr/local/lib/R/site-library/StanHeaders/include/src/stan/services/diagnose/diagnose.hpp:10,
                 from /usr/local/lib/R/site-library/rstan/include/rstan/stan_fit.hpp:34,
                 from /usr/local/lib/R/site-library/rstan/include/rstan/rstaninc.hpp:3,
                 from stan_files/breath_test_1.hpp:5,
                 from stan_files/breath_test_1.cc:3:
/usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/scal/fun/size_zero.hpp:18:13: note: candidate: template<class T> bool stan::math::size_zero(T&)
 inline bool size_zero(T& x) {
             ^
/usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/scal/fun/size_zero.hpp:18:13: note:   template argument deduction/substitution failed:
In file included from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/scal/prob/normal_log.hpp:5:0,
                 from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/scal.hpp:340,
                 from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/arr.hpp:49,
                 from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/mat.hpp:325,
                 from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/mat.hpp:12,
                 from /usr/local/lib/R/site-library/StanHeaders/include/src/stan/model/log_prob_grad.hpp:4,
                 from /usr/local/lib/R/site-library/StanHeaders/include/src/stan/model/test_gradients.hpp:7,
                 from /usr/local/lib/R/site-library/StanHeaders/include/src/stan/services/diagnose/diagnose.hpp:10,
                 from /usr/local/lib/R/site-library/rstan/include/rstan/stan_fit.hpp:34,
                 from /usr/local/lib/R/site-library/rstan/include/rstan/rstaninc.hpp:3,
                 from stan_files/breath_test_1.hpp:5,
                 from stan_files/breath_test_1.cc:3:
/usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/scal/prob/normal_lpdf.hpp:55:16: note:   candidate expects 1 argument, 3 provided
   if (size_zero(y, mu, sigma))
                ^
In file included from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/mat/prob/bernoulli_logit_glm_lpmf.hpp:15:0,
                 from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/mat.hpp:267,
                 from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/mat.hpp:12,
                 from /usr/local/lib/R/site-library/StanHeaders/include/src/stan/model/log_prob_grad.hpp:4,
                 from /usr/local/lib/R/site-library/StanHeaders/include/src/stan/model/test_gradients.hpp:7,
                 from /usr/local/lib/R/site-library/StanHeaders/include/src/stan/services/diagnose/diagnose.hpp:10,
                 from /usr/local/lib/R/site-library/rstan/include/rstan/stan_fit.hpp:34,
                 from /usr/local/lib/R/site-library/rstan/include/rstan/rstaninc.hpp:3,
                 from stan_files/breath_test_1.hpp:5,
                 from stan_files/breath_test_1.cc:3:
/usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/scal/fun/size_zero.hpp:31:13: note: candidate: template<class T, class ... Ts> bool stan::math::size_zero(T&, <type error>)
 inline bool size_zero(T& x, Ts&&... xs) {
             ^
/usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/scal/fun/size_zero.hpp:31:13: note:   template argument deduction/substitution failed:
/usr/lib/R/etc/Makeconf:166: die Regel für Ziel „stan_files/breath_test_1.o“ scheiterte
rm stan_files/breath_test_1.cc
make: *** [stan_files/breath_test_1.o] Fehler 1
ERROR: compilation failed for package ‘breathteststan’
* removing ‘/usr/local/lib/R/site-library/breathteststan’
* restoring previous ‘/usr/local/lib/R/site-library/breathteststan’
Warning in install.packages :
  installation of package ‘breathteststan’ had non-zero exit status
1 Like

There is a variadic version of size_zero now in Math


What version of g++ do you have?

Windows 10: Makevars as above.

C:/R/library/StanHeaders/include/src/stan/lang/ast/node/map_rect.hpp: In static member function 'static std::vector<std::pair<int, std::basic_string<char> > >& stan::lang::map_rect::registered_calls()':
C:/R/library/StanHeaders/include/src/stan/lang/ast/node/map_rect.hpp:28:50: error: '>>' should be '> >' within a nested template argument list
     static std::vector<std::pair<int, std::string>> REGISTERED_CALLS_;
                                                  ^
make: *** [C:/R/etc/x64/Makeconf:215: lang__grammars__bare_type_grammar_inst.o] Error 1
make: *** [C:/R/etc/x64/Makeconf:215: lang__grammars__expression07_grammar_inst.o] Error 1
ERROR: compilation failed for package 'rstan'
* removing 'C:/R/library/rstan'
* restoring previous 'C:/R/library/rstan'
In R CMD INSTALL
Warning in install.packages :
  installation of package ‘rstan’ had non-zero exit status

Windows:
gcc version 4.9.3 (x86_64-posix-seh, Built by MinGW-W64 project)

Also tried clang++ on windows, as downloaded today. Different error.

Ubuntu
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)

That is strange. The compiler should know better. You can try

install.packages("https://win-builder.r-project.org/8Q3LUki5kcHs/rstan_2.18.1.zip", 
                 repos = NULL)

until there are proper binaries.

1 Like

breathteststan is working for me with g++-6 and clang++-3.9. I’ll try in a little while on a computer that only has g++-5.x.

Installed gcc 6.4.0 on Ubuntu. Build works there. Trying Windows next

Hi Ben,

First, thanks very much for the pull request to update our package.

I accepted the pull request, and the package builds well on my local OS X. However, R CMD check gives the following warning:

* DONE
Status: 1 WARNING
checking compiled code ... WARNING
File ‘bmlm/libs/bmlm.so’:
  Found ‘_abort’, possibly from ‘abort’ (C)
    Object: ‘Modules.o’

Compiled code should not call entry points which might terminate R nor
write to stdout/stderr instead of to the console, nor use Fortran I/O
nor system RNGs.

See
  ‘/bmlm.Rcheck/00check.log’
for details.

R CMD check results
0 errors | 1 warning  | 0 notes

R CMD check succeeded

However, results from win builder R CMD check (again thank you for sending this in!) give no warnings:

Your package bmlm_1.3.8.tar.gz has been built (if working) and checked for Windows.
Please check the log files and (if working) the binary package at:
https://win-builder.r-project.org/x23M090H5ANc
The files will be removed after roughly 72 hours.
Installation time in seconds: 137
Check time in seconds: 124
Status: OK
R Under development (unstable) (2018-10-12 r75434)

Is the local warning about bmlm.so something that I can ignore, or is there something else I can and should do about it? Nothing in the pkg source has changed apart from the PR since last CRAN submission, and there were no issues back then.

Thanks very much

> devtools::session_info()
Session info -----------------------------------------------------------------------------------------------------------------------
 setting  value                       
 version  R version 3.5.1 (2018-07-02)
 system   x86_64, darwin15.6.0        
 ui       RStudio (1.2.907)           
 language (EN)                        
 collate  en_US.UTF-8                 
 tz       America/New_York            
 date     2018-10-16                  

Packages ---------------------------------------------------------------------------------------------------------------------------
 package   * version date       source        
 base      * 3.5.1   2018-07-05 local         
 compiler    3.5.1   2018-07-05 local         
 datasets  * 3.5.1   2018-07-05 local         
 devtools    1.13.6  2018-06-27 CRAN (R 3.5.0)
 digest      0.6.18  2018-10-10 CRAN (R 3.5.0)
 graphics  * 3.5.1   2018-07-05 local         
 grDevices * 3.5.1   2018-07-05 local         
 memoise     1.1.0   2017-04-21 CRAN (R 3.5.0)
 methods   * 3.5.1   2018-07-05 local         
 packrat     0.4.9-3 2018-06-01 CRAN (R 3.5.0)
 stats     * 3.5.1   2018-07-05 local         
 tools       3.5.1   2018-07-05 local         
 utils     * 3.5.1   2018-07-05 local         
 withr       2.1.2   2018-03-15 CRAN (R 3.5.0)

@matti R gets better and better at detecting objectionable things. We’ll have to figure that out before you can upload. @jonah can you replicate this if you install master from https://github.com/mvuorre/bmlm/ ?

Edit: @jonah Never mind.

Sorry, I only noted your PR now, I had made the changes by hand, but looks they are the same. PR now included in master

Builds better on Windows with gcc version 6.2.1 20160825 (GCC). No great chaos, but:

* installing *source* package 'breathteststan' ...
** libs
g++   -I"C:/R/include" -DNDEBUG -I"../inst/include" -I"`"C:/R/bin/x64/Rscript" --vanilla -e "cat(system.file('include', 'src', package = 'StanHeaders'))"`" -I"`"C:/R/bin/x64/Rscript" --vanilla -e "cat(system.file('include', 'boost_not_in_BH', package = 'rstan'))"`" -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -DEIGEN_NO_DEBUG -DBOOST_NO_CXX11_RVALUE_REFERENCES -I"C:/R/library/StanHeaders/include" -I"C:/R/library/rstan/include" -I"C:/R/library/BH/include" -I"C:/R/library/Rcpp/include" -I"C:/R/library/RcppEigen/include"        -O3 -mtune=native -Wno-unused-variable -Wno-deprecated-declarations -Wno-ignored-attributes -c stan_files/breath_test_1.cc -o stan_files/breath_test_1.o
g++   -I"C:/R/include" -DNDEBUG -I"../inst/include" -I"`"C:/R/bin/x64/Rscript" --vanilla -e "cat(system.file('include', 'src', package = 'StanHeaders'))"`" -I"`"C:/R/bin/x64/Rscript" --vanilla -e "cat(system.file('include', 'boost_not_in_BH', package = 'rstan'))"`" -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -DEIGEN_NO_DEBUG -DBOOST_NO_CXX11_RVALUE_REFERENCES -I"C:/R/library/StanHeaders/include" -I"C:/R/library/rstan/include" -I"C:/R/library/BH/include" -I"C:/R/library/Rcpp/include" -I"C:/R/library/RcppEigen/include"        -O3 -mtune=native -Wno-unused-variable -Wno-deprecated-declarations -Wno-ignored-attributes -c stan_files/breath_test_group_1.cc -o stan_files/breath_test_group_1.o
g++   -I"C:/R/include" -DNDEBUG -I"../inst/include" -I"`"C:/R/bin/x64/Rscript" --vanilla -e "cat(system.file('include', 'src', package = 'StanHeaders'))"`" -I"`"C:/R/bin/x64/Rscript" --vanilla -e "cat(system.file('include', 'boost_not_in_BH', package = 'rstan'))"`" -DBOOST_RESULT_OF_USE_TR1 -DBOOST_NO_DECLTYPE -DBOOST_DISABLE_ASSERTS -DEIGEN_NO_DEBUG -DBOOST_NO_CXX11_RVALUE_REFERENCES -I"C:/R/library/StanHeaders/include" -I"C:/R/library/rstan/include" -I"C:/R/library/BH/include" -I"C:/R/library/Rcpp/include" -I"C:/R/library/RcppEigen/include"        -O3 -mtune=native -Wno-unused-variable -Wno-deprecated-declarations -Wno-ignored-attributes -c init.cpp -o init.o
In file included from C:/R/library/StanHeaders/include/stan/math/prim/mat/functor/map_rect.hpp:13:0,
                 from C:/R/library/StanHeaders/include/stan/math/prim/mat.hpp:262,
                 from C:/R/library/StanHeaders/include/stan/math/rev/mat.hpp:12,
                 from C:/R/library/StanHeaders/include/src/stan/model/log_prob_grad.hpp:4,
                 from C:/R/library/StanHeaders/include/src/stan/model/test_gradients.hpp:7,
                 from C:/R/library/StanHeaders/include/src/stan/services/diagnose/diagnose.hpp:10,
                 from C:/R/library/rstan/include/rstan/stan_fit.hpp:34,
                 from C:/R/library/rstan/include/rstan/rstaninc.hpp:3,
                 from stan_files/breath_test_group_1.hpp:5,
                 from stan_files/breath_test_group_1.cc:3:
C:/R/library/StanHeaders/include/stan/math/prim/mat/functor/map_rect_concurrent.hpp: In function 'Eigen::Matrix<typename stan::return_type<T_shared_param, T_job_param>::type, -1, 1> stan::math::internal::map_rect_concurrent(const Eigen::Matrix<T_job_param, -1, 1>&, const std::vector<Eigen::Matrix<T_job_param, -1, 1> >&, const std::vector<std::vector<double> >&, const std::vector<std::vector<int> >&, std::ostream*)':
C:/R/library/StanHeaders/include/stan/math/prim/mat/functor/map_rect_concurrent.hpp:108:59: error: invalid use of incomplete type '__gnu_cxx::__alloc_traits<std::allocator<std::future<std::vector<Eigen::Matrix<double, -1, -1> > > > >::value_type {aka class std::future<std::vector<Eigen::Matrix<double, -1, -1> > >}'
     const std::vector<matrix_d>& chunk_result = futures[i].get();

@denne That may be a bug in Stan and / or g++-6.2 on Windows. Is it working with g++ from RTools?

No, that was my earlier setup reported above which created mass-messages.

Also, @matti are you use that you have -O2 or -O3 in your CXX14FLAGS of your ~/.R/Makevars file. Sometimes these abort things get generated but cannot actually be called and there needs to be sufficient optimization to eliminate the dead code.

Drat. What is in your ~/.R/Makevars ?

@paul.buerkner Can you try to install / check from master at https://github.com/dmenne/breathteststan/ to see if you can replicate / solve @denne’s problems?