RStan in .Rmd fails to knit to html, but pdf and interactive work, and no error messages

Interfaces RStan
1

As in the title, any time I try to run a code block in a .Rmd containing the stan_model() function to compile Stan code into a model, if the output format is set to output: html_document, when I knit the file, it runs all the chunks up to the one containing that command and then just stops, as if completed, but without continuing on to the next chunks or outputing a file, and no error messages produced. It doesn’t seem to just be hanging; the little stop sign in Rstudio indicating code is running disappears as i it had completed and Rstudio is afterwards free to run any command. The exact same code, in a fully cleared environment, runs fine interactively with no errors, and if I set the header to output: pdf_document, it knits without any problems. Knitting to html on files that do not contain Stan code also produces output as expected with no problems. I found this when coming back to code that worked two months ago, so something must have changed, but I don’t have session info for the versions I was running then, so the best I can do is provide my current session info and system environment. I will say that I did not change my R version, and my Rstan version was up to date as of a few months ago, but there have probably been many minor updates in between. The code below, set as the full content of a .Rmd file, is a minimum working example, though any Stan model I’ve tried produces this same outcome.

library(rstan) #Access Stan from R
stan_code_bernoulli <- " 
data { 
  int T;                     // The sample size
  int<lower=0,upper=1> y[T]; // The observations (array of length T)
  real<lower=0> prior_a;     // The a parameter of the prior: data because user allowed to enter value
  real<lower=0> prior_b;     // The b parameter of the prior
} 
parameters { 
  real<lower=0,upper=1> theta; // The probability of observing a one
} 
model { 
  theta ~ beta(prior_a,prior_b);        // Prior
  for (t in 1:T) 
    y[t] ~ bernoulli(theta);  // Likelihood: consists of N independent Bernoulli samples
} 
generated quantities { 
  int<lower=0,upper=1> y_tilde; // Samples from posterior predictive distribution
  y_tilde = bernoulli_rng(theta);  //Simulate draw from (Bernoulli) conditional likelihood 
}"

# Compile this code as a model. Warning: compiling takes a long time
stan_model_bernoulli<-stan_model(model_code = stan_code_bernoulli,model_name = "BernoulliModel")

My session info (when knitting to HTML) is below: I tried comparing running these commands in knit html documents versus interactively to see if there is a difference, and aside from the working directory I don’t see anything that stands out. I searched the Stan forums and Google before posting this, but found nobody with a comparable problem, except for my old post on a similar problem, which did not get a full explanation. In between the time of that post and this one, I upgraded R to 4.0 and also Stan, which fixed the problem at least for other files (I eventually gave up on that VAR model), but now it seems to be back. It’s possible this is an RStudio or knitr issue, not a Stan issue, though running Stan is the only place I’ve been able to make the problem happen. My first guess is that something with the C++ compiler accessed with different knitting engines changes. I don’t see particular signs of that but could be missing something.

Edit: Found an interesting difference: running Sys.getenv() when compiling to html does produce different results than when compiling to PDF or when running interactively. The part that shows up in these but doesn’t show up in the html knit (when blocks containing stan_compile are set to eval=FALSE, as otherwise it doesn’t knit at all) is the following

PKG_CPPFLAGS                -I"/Library/Frameworks/R.framework/Versions/4.0/Resources/library/Rcpp/include/"
                            -I"/Library/Frameworks/R.framework/Versions/4.0/Resources/library/RcppEigen/include/"
                            -I"/Library/Frameworks/R.framework/Versions/4.0/Resources/library/RcppEigen/include/unsupported"
                            -I"/Library/Frameworks/R.framework/Versions/4.0/Resources/library/BH/include"
                            -I"/Library/Frameworks/R.framework/Versions/4.0/Resources/library/StanHeaders/include/src/"
                            -I"/Library/Frameworks/R.framework/Versions/4.0/Resources/library/StanHeaders/include/"
                            -I"/Library/Frameworks/R.framework/Versions/4.0/Resources/library/RcppParallel/include/"
                            -I"/Library/Frameworks/R.framework/Versions/4.0/Resources/library/rstan/include"
                            -DEIGEN_NO_DEBUG -DBOOST_DISABLE_ASSERTS
                            -DBOOST_PENDING_INTEGER_LOG2_HPP -DSTAN_THREADS
                            -DBOOST_NO_AUTO_PTR -include
                            '/Library/Frameworks/R.framework/Versions/4.0/Resources/library/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp'
                            -D_REENTRANT -DRCPP_PARALLEL_USE_TBB=1
PKG_LIBS                    '/Library/Frameworks/R.framework/Versions/4.0/Resources/library/rstan/lib//libStanServices.a'
                            -L'/Library/Frameworks/R.framework/Versions/4.0/Resources/library/StanHeaders/lib/'
                            -lStanHeaders
                            -L'/Library/Frameworks/R.framework/Versions/4.0/Resources/library/RcppParallel/lib/'
                            -ltbb -ltbbmalloc -ltbbmalloc_proxy

So, it definitely looks like the C++ settings are somehow different in the html file compiler and the pdf or interactive version. Is there some way I can manually adjust or reset these? Reading the Makevars file produces the same output in html or pdf. If this is a knitr or Rstudio thing and I should be asking elsewhere I’ll happily move this question wherever it belongs.

The rest of the system results are below:

sessionInfo()
## R version 4.0.3 (2020-10-10)
## Platform: x86_64-apple-darwin17.0 (64-bit)
## Running under: macOS Catalina 10.15.6
## 
## Matrix products: default
## BLAS:   /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRblas.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] rstan_2.21.2         ggplot2_3.3.3        StanHeaders_2.21.0-7
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.6         pillar_1.5.1       compiler_4.0.3     prettyunits_1.1.1 
##  [5] tools_4.0.3        pkgbuild_1.2.0     digest_0.6.27      jsonlite_1.7.2    
##  [9] evaluate_0.14      lifecycle_1.0.0    tibble_3.1.0       gtable_0.3.0      
## [13] debugme_1.1.0      pkgconfig_2.0.3    rlang_0.4.10       cli_2.4.0         
## [17] DBI_1.1.1          parallel_4.0.3     curl_4.3           yaml_2.2.1        
## [21] xfun_0.20          loo_2.4.1          gridExtra_2.3      withr_2.4.1       
## [25] stringr_1.4.0      dplyr_1.0.4        knitr_1.30         generics_0.1.0    
## [29] vctrs_0.3.7        stats4_4.0.3       grid_4.0.3         tidyselect_1.1.0  
## [33] inline_0.3.17      glue_1.4.2         R6_2.5.0           processx_3.4.5    
## [37] fansi_0.4.2        rmarkdown_2.6      callr_3.5.1        purrr_0.3.4       
## [41] magrittr_2.0.1     codetools_0.2-18   matrixStats_0.57.0 ps_1.6.0          
## [45] scales_1.1.1       ellipsis_0.3.1     htmltools_0.5.0    assertthat_0.2.1  
## [49] colorspace_2.0-0   V8_3.4.0           utf8_1.2.1         stringi_1.5.3     
## [53] RcppParallel_5.0.2 munsell_0.5.0      crayon_1.4.1

Sys.getenv()
## __CF_USER_TEXT_ENCODING
##                         0x1F5:0x0:0x0
## CLICOLOR_FORCE          1
## DISPLAY                 :0
## DYLD_FALLBACK_LIBRARY_PATH
##                         /Library/Frameworks/R.framework/Resources/lib:/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib/server:/Library/Frameworks/R.framework/Resources/lib:/Users/dchilder/lib:/usr/local/lib:/usr/lib:::/lib:/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home/jre/lib/server
## EDITOR                  vi
## GIT_ASKPASS             rpostback-askpass
## HOME                    /Users/dchilder
## LANG                    en_US.UTF-8
## LC_CTYPE                en_US.UTF-8
## LN_S                    ln -s
## LOGNAME                 dchilder
## MAKE                    make
## MPLENGINE               tkAgg
## NOT_CRAN                true
## PAGER                   /usr/bin/less
## PATH                    /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Library/TeX/texbin:/Library/Apple/usr/bin
## PWD                     /Users/dchilder/Dropbox/Forecasting/Website
##                         Version/Forecasting
## R_ARCH                  
## R_BROWSER               /usr/bin/open
## R_BZIPCMD               /usr/bin/bzip2
## R_DOC_DIR               /Library/Frameworks/R.framework/Resources/doc
## R_GZIPCMD               /usr/bin/gzip
## R_HOME                  /Library/Frameworks/R.framework/Resources
## R_INCLUDE_DIR           /Library/Frameworks/R.framework/Resources/include
## R_LIBS                  /Library/Frameworks/R.framework/Versions/4.0/Resources/library
## R_LIBS_SITE             
## R_LIBS_USER             ~/Library/R/4.0/library
## R_PAPERSIZE             a4
## R_PAPERSIZE_USER        a4
## R_PDFVIEWER             /usr/bin/open
## R_PLATFORM              x86_64-apple-darwin17.0
## R_PRINTCMD              lpr
## R_QPDF                  /Library/Frameworks/R.framework/Resources/bin/qpdf
## R_RD4PDF                times,inconsolata,hyper
## R_SESSION_TMPDIR        /var/folders/nz/znjnbmrn0qj846n629k4j69m0000gn/T//Rtmp6dx9IH
## R_SHARE_DIR             /Library/Frameworks/R.framework/Resources/share
## R_STRIP_SHARED_LIB      strip -x
## R_STRIP_STATIC_LIB      strip -S
## R_SYSTEM_ABI            macos,gcc,gxx,gfortran,gfortran
## R_TEXI2DVICMD           /usr/local/bin/texi2dvi
## R_UNZIPCMD              /usr/bin/unzip
## R_ZIPCMD                /usr/bin/zip
## RMARKDOWN_MATHJAX_PATH
##                         /Applications/RStudio.app/Contents/Resources/resources/mathjax-26
## RMARKDOWN_PREVIEW_DIR   /var/folders/nz/znjnbmrn0qj846n629k4j69m0000gn/T//RtmpfmpcdW
## RS_PPM_FD_READ          13
## RS_PPM_FD_WRITE         16
## RS_RPOSTBACK_PATH       /Applications/RStudio.app/Contents/MacOS/rpostback
## RS_SHARED_SECRET        9849436581144108930470211272
## RSTUDIO                 1
## RSTUDIO_CONSOLE_COLOR   256
## RSTUDIO_CONSOLE_WIDTH   110
## RSTUDIO_PANDOC          /Applications/RStudio.app/Contents/MacOS/pandoc
## RSTUDIO_SESSION_PORT    34332
## RSTUDIO_USER_IDENTITY   dchilder
## RSTUDIO_VERSION         1.2.5033
## RSTUDIO_WINUTILS        bin/winutils
## SED                     /usr/bin/sed
## SHELL                   /bin/zsh
## SHLVL                   0
## SQLITE_EXEMPT_PATH_FROM_VNODE_GUARDS
##                         /Users/dchilder/Library/WebKit/Databases
## SSH_ASKPASS             rpostback-askpass
## SSH_AUTH_SOCK           /private/tmp/com.apple.launchd.KICEkTTlsJ/Listeners
## TAR                     /usr/bin/tar
## TERM                    xterm-256color
## TMPDIR                  /var/folders/nz/znjnbmrn0qj846n629k4j69m0000gn/T/
## TZDIR                   macOS
## USER                    dchilder
## XPC_FLAGS               0x0
## XPC_SERVICE_NAME        org.rstudio.RStudio.3700

version
##                _                           
## platform       x86_64-apple-darwin17.0     
## arch           x86_64                      
## os             darwin17.0                  
## system         x86_64, darwin17.0          
## status                                     
## major          4                           
## minor          0.3                         
## year           2020                        
## month          10                          
## day            10                          
## svn rev        79318                       
## language       R                           
## version.string R version 4.0.3 (2020-10-10)
## nickname       Bunny-Wunnies Freak Out

writeLines(readLines(file.path(Sys.getenv("HOME"), ".R/Makevars")))
## CFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
## CCFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
## CXXFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
## # clang: end
1 Like
2

So, I spent another couple hours on this, reinstalled Stan, reinstalled knitr, tried a few more things, none of which fixed it. But, I rewrote my markdown file to, instead of calling an R chunk which calls stan_compile, just include the Stan program in a stan chunk, i.e.

```{stan, output.var="stan_bernoulli_model"}
\\ Stan code follows

And then the document compiles and samples with no problem, either in pdf or html. So the underlying issue, whatever it is, is still there, but I can at least make an html file which contains Stan code and runs it, which is what I needed for the website I was building. I presume this problem will come back to bite me in 6 months, so I’ll leave the post up in case somebody finds a fix before then.

1 Like
3

I honestly have no idea why that could be a problem, but thanks very much for sharing your workaround!