Dealing with Catalina

bigmanj · October 17, 2019, 3:37am

I don’t know if this matters, but when I download rstan (since I uninstalled it to see if that would work), it is going into an unfamiliar folder to me (The downloaded binary packages are in
/var/folders/bp/4z337q_5263ghp8njdx0_48h0000gp/T//RtmpRtWwIC/downloaded_packages), which is not the same folder that packages are saved to when I use the install.packages 'button" in the file, plots, packages, etc. pane (/Library/Frameworks/R.framework/Versions/3.5/Resources/library/[Default]).

bgoodri · October 17, 2019, 3:58am

That is fine. When you do

library(rstan)
example(stan_model, run.dontrun = TRUE)

what is the line that has the string error: (with the colon)?

bgoodri · October 17, 2019, 3:59am

What is

  Rv <- R.version
  GOOD <- file.path("Library", "Frameworks", "R.framework", "Versions", 
                     paste(Rv$major, substr(Rv$minor, 1, 1), sep = "."), 
                     "Resources", "lib", "libc++.1.dylib")
  GOOD

? Does that file really exist?

bigmanj · October 17, 2019, 4:01am

I think there are 2? One at the beginning and one at the end:

Error in compileCode(f, code, language = language, verbose = verbose) :
Compilation ERROR, function(s)/method(s) not created! In file included from file9327259324e.cpp:8:

and

Error in sink(type = “output”) : invalid connection

bigmanj · October 17, 2019, 4:04am

Oh, you mean in the actual stan model, sorry. The error does not specify a line. It says compilation error and then repeats the same string to show the file path where the file that did not compile is? I think.

bgoodri · October 17, 2019, 4:19am

Between those two. After about 500 lines of C++ code. After it says clang++ .... There will be error: …

bigmanj · October 17, 2019, 4:25am

Sorry, I’m not sure where to look for this–guessing not in the console.

Thanks!

torkar · October 17, 2019, 7:46am

@bgoodri, something is fishy here. It does not consistently work, if I run this
bar.R (1.4 KB) model.stan (2.1 KB)

Every now and then it samples correctly and doesn’t throw an error. Also if it throws an error then it’s not a given that fix_Mac() will fix it… Could someone with Catalina run the above 5-10 times (each time restarting RStudio and not saving the environment in-between?

I’d be most grateful if you could tell me if you also see inconsistent behavior in that it samples correctly every now and then!

Sorry @bgoodri, but there’s something more going on here :(

mansillo · October 17, 2019, 8:08am

@bgoodri & @torkar and on the sixth implementation:

SAMPLING FOR MODEL 'model' NOW (CHAIN 3).
Chain 3: 
Chain 3: Gradient evaluation took 2.2e-05 seconds
Chain 3: 1000 transitions using 10 leapfrog steps per transition would take 0.22 seconds.
Chain 3: Adjust your expectations accordingly!
Chain 3: 
Chain 3: 
Chain 3: Iteration:    1 / 2000 [  0%]  (Warmup)
[1] "Error in sampler$call_sampler(args_list[[i]]) : " "  c++ exception (unknown reason)"                
error occurred during calling the sampler; sampling not done

torkar · October 17, 2019, 10:05am

Shit, that’s what I was afraid of… Well, I don’t think @bgoodri will like this present when he wakes up…

janhannig · October 17, 2019, 11:03am

This model does not sample ever on catalina but works fine on the previous version

functions {
    real gpareto_lpdf(vector y, real ymin, real k, real sigma) {
      // generalised Pareto log pdf 
      int N = rows(y);
      real inv_k = inv(k);
      if (k<0 && max(y-ymin)/sigma > -inv_k)
        reject("k<0 and max(y-ymin)/sigma > -1/k; found k, sigma =", k, sigma)
      if (sigma<=0)
        reject("sigma<=0; found sigma =", sigma)
      if (fabs(k) > 1e-15)
        return -(1+inv_k)*sum(log1p((y-ymin) * (k/sigma))) -N*log(sigma);
      else
        return -sum(y-ymin)/sigma -N*log(sigma); // limit k->0
    }
    real gpareto_cdf(vector y, real ymin, real k, real sigma) {
      // generalised Pareto cdf
      real inv_k = inv(k);
      if (k<0 && max(y-ymin)/sigma > -inv_k)
        reject("k<0 and max(y-ymin)/sigma > -1/k; found k, sigma =", k, sigma)
      if (sigma<=0)
        reject("sigma<=0; found sigma =", sigma)
      if (fabs(k) > 1e-15)
        return exp(sum(log1m_exp((-inv_k)*(log1p((y-ymin) * (k/sigma))))));
      else
        return exp(sum(log1m_exp(-(y-ymin)/sigma))); // limit k->0
    }
    real gpareto_lcdf(vector y, real ymin, real k, real sigma) {
      // generalised Pareto log cdf
      real inv_k = inv(k);
      if (k<0 && max(y-ymin)/sigma > -inv_k)
        reject("k<0 and max(y-ymin)/sigma > -1/k; found k, sigma =", k, sigma)
      if (sigma<=0)
        reject("sigma<=0; found sigma =", sigma)
      if (fabs(k) > 1e-15)
        return sum(log1m_exp((-inv_k)*(log1p((y-ymin) * (k/sigma)))));
      else
        return sum(log1m_exp(-(y-ymin)/sigma)); // limit k->0
    }
    real gpareto_lccdf(vector y, real ymin, real k, real sigma) {
      // generalised Pareto log ccdf
      real inv_k = inv(k);
      if (k<0 && max(y-ymin)/sigma > -inv_k)
        reject("k<0 and max(y-ymin)/sigma > -1/k; found k, sigma =", k, sigma)
      if (sigma<=0)
        reject("sigma<=0; found sigma =", sigma)
      if (fabs(k) > 1e-15)
        return (-inv_k)*sum(log1p((y-ymin) * (k/sigma)));
      else
        return -sum(y-ymin)/sigma; // limit k->0
    }
    real gpareto_rng(real ymin, real k, real sigma) {
      // generalised Pareto rng
      if (sigma<=0)
        reject("sigma<=0; found sigma =", sigma)
      if (fabs(k) > 1e-15)
        return ymin + (uniform_rng(0,1)^-k -1) * sigma / k;
      else
        return ymin - sigma*log(uniform_rng(0,1)); // limit k->0
    }
   real gparetoJacP2L2(vector y, real ymin, real k, real sigma) {
       // L2 Jacobian for GPD with ymin known
      int N = rows(y);
      real inv_k = inv(k);
      real prejac1 = sum((y-ymin).*(y-ymin)) * sum((1+(k/sigma)*(y-ymin)).*(1+(k/sigma)*(y-ymin)).*log1p((k/sigma)*(y-ymin)).*log1p((k/sigma)*(y-ymin)))
                     -sum((y-ymin).*(1+(k/sigma)*(y-ymin)).*log1p((k/sigma)*(y-ymin)))*sum((y-ymin).*(1+(k/sigma)*(y-ymin)).*log1p((k/sigma)*(y-ymin)));
      real prejac2=sum((y-ymin).*(y-ymin))*sum((y-ymin).*(y-ymin).*(y-ymin).*(y-ymin))
               -sum((y-ymin).*(y-ymin).*(y-ymin))*sum((y-ymin).*(y-ymin).*(y-ymin));
      if (k<0 && max(y-ymin)/sigma > -inv_k)
          reject("k<0 and max(y-ymin)/sigma > -1/k; found k, sigma =", k, sigma)
      if (sigma<=0)
         reject("sigma<=0; found sigma =", sigma)
      if (fabs(k) > 1e-15)
         return -log(k*k)+0.5*log(prejac1);
      else
         return -log(2)-2*log(sigma)+0.5*log(prejac2); // limit k->0
      }
}
data {
  real ymin;
  int<lower=0> N;
  vector<lower=ymin>[N] y;
}
transformed data {
  real ymax = max(y);
}
parameters {
  real<lower=0> sigma; 
  //real k;
  real<lower=-sigma/(ymax-ymin)> k; 
}
model {
  y ~ gpareto(ymin, k, sigma);
  target += gparetoJacP2L2(y, ymin, k, sigma); //Fiducial Jacobian
}
generated quantities {
  real<lower=0> y_sim=gpareto_rng(ymin, k, sigma);
}

The code to run the program is

n=1000
x=-log(runif(n))

library(rstan)
source('FixMAC.R')

fitted_model = stan_model(file='pareto_gfi.stan') 
fix_Mac(fitted_model)

stan_data <- list(y=x, N = n, ymin = 0)

stan_fit = sampling(fitted_model, data = stan_data)
plot(stan_fit,show_density=TRUE)
summary(stan_fit)

torkar · October 17, 2019, 11:12am

Always provide a complete minimum working example incl. data. Add file + data to your post as a file and people can easily reproduce it :)

janhannig · October 17, 2019, 11:15am

Sorry - it was a few steps above. I edited my post.

janhannig · October 17, 2019, 11:15am

Rv ← R.version
GOOD ← file.path(“Library”, “Frameworks”, “R.framework”, “Versions”,

              paste(Rv$major, substr(Rv$minor, 1, 1), sep = "."),

              "Resources", "lib", "libc++.1.dylib")

GOOD
[1] “Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libc++.1.dylib”
Rv
_
platform x86_64-apple-darwin15.6.0
arch x86_64
os darwin15.6.0
system x86_64, darwin15.6.0
status
major 3
minor 6.1
year 2019
month 07
day 05
svn rev 76782
language R
version.string R version 3.6.1 (2019-07-05)
nickname Action of the Toes

bgoodri · October 17, 2019, 7:00pm

@coatless: Should something be added to LDFLAGS in Makevars with Macs so that it links against the libc++ provided by CRAN? It seems that the fix @kevinushey suggested does not always work.

coatless · October 17, 2019, 7:50pm

Perhaps the ~/.R/Makevars should have LDFLAGS set with

-L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib

Thus, the full file would be:

# clang: start
CFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
CCFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
CXXFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
CPPFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
LDFLAGS=-L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib
# clang: end

Under this configuration, I can successfully sample multiple times after running the fix_Mac() function given previously. c.f.

fix_Mac <- function(sm) {
  stopifnot(is(sm, "stanmodel"))
  dso_last_path <- sm@dso@.CXXDSOMISC$dso_last_path
  CLANG_DIR <- tail(n = 1, grep("clang[456789]", 
                                x = list.dirs("/usr/local", recursive = FALSE), 
                                value = TRUE))
  if (length(CLANG_DIR) == 0L) stop("no clang from CRAN found")
  LIBCPP <- file.path(CLANG_DIR, "lib", "libc++.1.dylib")
  if (!file.exists(LIBCPP)) stop("no unique libc++.1?.dylib found")
  Rv <- R.version
  GOOD <- file.path("/Library", "Frameworks", "R.framework", "Versions", 
                    paste(Rv$major, substr(Rv$minor, 1, 1), sep = "."), 
                    "Resources", "lib", "libc++.1.dylib")
  if (!file.exists(GOOD)) stop(paste(GOOD, "not found"))
  cmd <- paste(
    "install_name_tool",
    "-change",
    LIBCPP,
    GOOD,
    dso_last_path
  )
  system(cmd)
  dyn.unload(dso_last_path)
  dyn.load(dso_last_path)
  return(invisible(NULL))
}


stancode <- 'data {real y_mean;} parameters {real y;} model {y ~ normal(y_mean,1);}'
mod <- stan_model(model_code = stancode, verbose = TRUE)
fix_Mac(mod)

fit <- sampling(mod, data = list(y_mean = 0))
fit2 <- sampling(mod, data = list(y_mean = 5))

bgoodri · October 17, 2019, 7:57pm

OK. I was kind of hoping with the -L... that the fix_Mac() would not be necessary. Do you get the unknown exception thing if you compile and sample without calling fix_Mac in between?

bgoodri · October 17, 2019, 9:04pm

@kevinushey I am using someone else’s Mac that has Catalina and I can confirm that previously loading another package with compiled C++ code (such as readr) is sufficient to cause

library(readr) # or another package with compiled C++

Rcpp::sourceCpp(code = 
'
#include <Rcpp.h>
using namespace Rcpp; 

// [[Rcpp::export]]
int throw_exception() { 
  std::stringstream errmsg; errmsg << "this is an exception";
  throw std::domain_error(errmsg.str()); 
  return 0;
}
'
)

throw_exception()

to say

Error in thow_exception(): c++ exception (unknown reason)

instead of

Error in thow_exception(): this is an exception

However, if I don’t load any packages containing C++ previously, then I get the expected behavior. Is this a known issue? It doesn’t have anything directly to do with rstan code, but it may have a clue as to why we are having similar problems with dynamically loaded objects with C++ code generated by rstan.

This is using /usr/local/clang7/bin/clang++ and the ~/.R/Makevars file created by @coatless installer.

kevinushey · October 17, 2019, 9:36pm

Can we confirm that all of these packages are using the expected C++ standard libraries? Here’s a script that should dump this all out:


packages <- loadedNamespaces()
result <- lapply(packages, function(package) {
  
  dir <- system.file("libs", package = package)
  libs <- list.files(dir, pattern = "[.]so$", full.names = TRUE)
  if (length(libs) == 0)
    return(FALSE)
  
  output <- system(paste("otool -L", libs, "| grep -s libc++ || true"), intern = TRUE)
  if (length(output) == 0)
    return(FALSE)
  
  writeLines(paste0(package, ":"))
  writeLines(output)
  
})

As an example, I get some packages showing a mix, for some packages I installed from sources:

Rcpp:
	/Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
xml2:
	/Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
roxygen2:
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 800.6.0)
htmltools:
	/Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
digest:
	/Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
fs:
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)
testthat:
	/Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
stringi:
	/Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)

bgoodri · October 17, 2019, 9:46pm

When I do

library(readr)

followed by your function on that Catalina laptop I get

readr:
        /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)

Rcpp:
        /usr/lib/libc++1.1.dylib (compatibility version 1.0.0, current version 800.6.0)

After reinstalling Rcpp using /usr/local/clang7/bin/clang++l, it switched to

Rcpp:
        /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)

but now

Rcpp::sourceCpp(code = 
'
#include <Rcpp.h>
using namespace Rcpp; 

// [[Rcpp::export]]
int throw_exception() { 
  std::stringstream errmsg; errmsg << "this is an exception";
  throw std::domain_error(errmsg.str()); 
  return 0;
}
'
)

throw_exception()

yields

Error in thow_exception(): c++ exception (unknown reason)

irrespective of whether any other packages are loaded

Topic		Replies	Views
Update MacOS to Catalina or Mojave for least problems with Stan-based software? RStan installation	20	1742	December 13, 2020
Installing RStan on Mojave Mac OS X -missing step on C++ toolchain General	4	1029	February 20, 2019
Dealing with Mojave RStan rstan	5	2812	October 7, 2019
MacOS RStan	10	1890	November 2, 2022
Problem with C++ Compiler on Mac RStan	2	2381	October 27, 2018

Dealing with Catalina

Related topics