I get the *** buffer overflow detected *** error as well, although as @rgoswami said, it doesn’t prevent model fitting. (However, it is preventing me from using languageserver for lsp-mode for Emacs.)
Doing some digging, I’m pretty sure the issue has to do with environment variables being set to excessively long strings. In Nix, R doesn’t point to the R binary but instead a wrapped shell script that adds specified libraries in the Nix store to R_LIBS_SITE. It doesn’t necessarily do this in a smart way. In fact, the wrapped script I’m looking at includes duplicate paths, which is probably a bug.
Some invocations of child R processes work okay, for whatever reason. For example, system("R --vanilla -q -e 'ls()'") works fine. However, callr::r(function() 1 +2) (callr is used for languageserver, so it’s what I’m debugging right now) gets a buffer overflow, and I’ve noticed that it sets R_LIBS_SITE, R_LIBS, and R_LIBS_USER to be the same really long set of paths, which it exports to a temporary file that is presumably read by the child R process. Removing these duplicate environment variable settings prevents the buffer overflow, but as of yet I haven’t figured out a permanent fix for callr (or maybe for its dependency processx).