Rstan on NixOS

I have an install of RStan on NixOS. On first use I think it runs some checks on the toolchain and I get this error:

Trying to compile a simple C file
Running /nix/store/1dv1hi5sc3kqgyb52dlik8s7czfmls9b-R-3.6.1/lib/R/bin/R CMD \
  SHLIB foo.c
*** buffer overflow detected ***: /nix/store/1dv1hi5sc3kqgyb52dlik8s7czfmls9b-R-3.6.1/lib/R/bin/exec/R terminated
/nix/store/1dv1hi5sc3kqgyb52dlik8s7czfmls9b-R-3.6.1/lib/R/bin/SHLIB: line 4: 27400 Done                    echo 'tools:::.SHLIB()'
     27401 Aborted                 (core dumped) | R_DEFAULT_PACKAGES=NULL "${R_HOME}/bin/R" --no-restore --slave --no-site-file --no-init-file --args $@

foo.c is defined at

Running R CMD SHLIB foo.c at the command line does not report a buffer overflow error.

I think it must be to do with how the command is run inside rcmd_safe at because when I try this function on it’s own I get the same error.

The same thing happens when I replace rcmd_safe with rcmd and rcmd_copycat

Any ideas on what to do next would be much appreciated

1 Like

I would take that up with the pkgbuild people. I think you can trigger it just by doing

pkgbuild::has_build_tools(debug = TRUE)

A potential quick fix: I had this problem, but rstan works fine after switching to Rstudio.

1 Like

Unfortunately I have other problems loading Rstudio :-(

rstanarm isn’t building for me, but I can fit 8 schools with the following shell.nix:

{ pkgs ?
    import (
            "") {}


    Rstudio-with-my-packages =
            packages = with pkgs.rPackages; [


                # rstanarm - had trouble building



    pkgs.mkShell {
        buildInputs = with pkgs; [
            # libintl 
            # libiconv
1 Like

(Worked in a --pure shell.)

Ah, of course! Why didn’t I think of that?

One of the strengths of Nix and I just ignored it when trying to solve my problem!

The error does not actually stop it from running though. I have used it with brms and rethinking with some details here.

I think that brms uses precompiled stan models. The issue I am having is with the compilation step.

No, it compiles them at runtime. I am not that familiar with NixOS, but if you post the error messages I might be able to figure it out.

I get the *** buffer overflow detected *** error as well, although as @rgoswami said, it doesn’t prevent model fitting. (However, it is preventing me from using languageserver for lsp-mode for Emacs.)

Doing some digging, I’m pretty sure the issue has to do with environment variables being set to excessively long strings. In Nix, R doesn’t point to the R binary but instead a wrapped shell script that adds specified libraries in the Nix store to R_LIBS_SITE. It doesn’t necessarily do this in a smart way. In fact, the wrapped script I’m looking at includes duplicate paths, which is probably a bug.

Some invocations of child R processes work okay, for whatever reason. For example, system("R --vanilla -q -e 'ls()'") works fine. However, callr::r(function() 1 +2) (callr is used for languageserver, so it’s what I’m debugging right now) gets a buffer overflow, and I’ve noticed that it sets R_LIBS_SITE, R_LIBS, and R_LIBS_USER to be the same really long set of paths, which it exports to a temporary file that is presumably read by the child R process. Removing these duplicate environment variable settings prevents the buffer overflow, but as of yet I haven’t figured out a permanent fix for callr (or maybe for its dependency processx).

All right, I created a Nix overlay workaround. Basically, it creates a single R_LIBS_SITE directory, similar to symlinkJoin. This prevents environment variables from getting too long.

In a file named something like ~/.config/nixpkgs/utils/r-wrapper2.nix:

{ runCommand, R, makeWrapper, lndir, recommendedPackages, packages }:

runCommand ( + "-wrapper") {
  preferLocalBuild = true;
  allowSubstitutes = false;

  buildInputs = [R] ++ recommendedPackages ++ packages;

  nativeBuildInputs = [makeWrapper];

  # Make the list of recommended R packages accessible to other packages such as rpy2
  # (Same as in the original rWrapper)
  passthru = { inherit recommendedPackages; };
  # Wrap a site lib, similar to symlinkJoin but without propagating buildInputs.
  mkdir -p $out/library
  for lib in $(echo -n $R_LIBS_SITE | sed -e 's/:/\n/g'); do
    ${lndir}/bin/lndir -silent $lib $out/library/
  mkdir -p $out/bin
  cd ${R}/bin
  for exe in *; do
    makeWrapper ${R}/bin/$exe $out/bin/$exe \
      --prefix "R_LIBS_SITE" ":" "$out/library"

Then create an overlay in a file like ~/.config/nixpkgs/overlays/10-rwrapper2.nix:

self: super:
  rWrapper = super.callPackage ../utils/r-wrapper2.nix {
    # Copied from the original rWrapper in all-packages.nix
    # If you define this as rWrapper2, you can just use
    # "inherit (self.rWrapper) recommendedPackages;"
    recommendedPackages = with super.rPackages; [
      boot class cluster codetools foreign KernSmooth lattice MASS
      Matrix mgcv nlme nnet rpart spatial survival
    # Override this attribute to register additional libraries.
    packages = [];

You can call this new wrapper either rWrapper or something different like rWrapper2, depending on your preference.

Basically, the issue is that the R_LIBS_SITE environment variable accumulates all Nix propagatedBuildInputs, so it contains hundreds of paths. The R package callr duplicates these environment variables in some shell script files that are sourced by the child R script, probably just in case environment variables aren’t properly inherited by child processes for whatever reason. This is fine if R_LIBS_SITE is only a few directories long, as it is in most distributions, but it breaks when there are hundreds of long paths. This overlay makes it one path, containing symlinks.

The reason RStudio in Nix doesn’t face this issue is because the RStudio wrapper never redefines R_LIBS_SITE (because RStudio actually ignores it). Instead, the wrapper creates an .Rprofile file with the list of library paths.

This fix should probably be in nixpkgs proper, but I have to test that the symlinking used here doesn’t break anything.

Hope this helps.


I’m sure it helps somebody. Am I correct in inferring from what you said that I don’t need to change anything in rstan as a result?

1 Like

This is very helpful, thanks! Especially since without this rstan wasn’t working for me on unstable.

One additional observation is that if you declare too many packages in the override, running pkgbuild::has_build_tools(debug = TRUE) still results in a stack overflow error. I imagine this has to do with the fact that each package adds a long path. I don’t know what is the exact number up to which it works but for me I saw the error with around 30 packages but it was gone when I declared only rstan and brms.

Btw do you know how to install the overlay using nix-env? Because I tried (having both of your files in a dir where I also put a default.nix in which I import the overlay and override the packages and then doing nix-env -if ./default.nix) but then the R_SITE_LIBS variable is just empty and nothing works. I would be very grateful for a suggestion.