Cannot resolve the "cannot allocate vector of size … " error

Honestly, I think this might be due to some unnecessary Makevars.win configuration. The wiki is outdated.

Uninstall Rtools (all versions). Install Rtools 40. Rename your Makevars.win to something else, just to remove it from the picture for now. And try again. You may also need to rename .Renviron to something else as well.

First, a default configuration for Rtools 40 should not be C:/Rtools at all. It should be C:/rtools40.

Second, ensure after running Rtools40, that you open R and run: writeLines('PATH="${RTOOLS40_HOME}\\usr\\bin;${PATH}"', con = "~/.Renviron")

Third, if that still does not work, my guess is you have some lingering PATH problems, and a lingering Rtools directory.

You do not need a Makevars.win file with R 4.0 and Rtools40 to run Stan; it’s mainly useful for adding compiler optimizations, which should be done after ensuring that it works. All you should need, is R 4.0, Rtools40 (with defaults selected), and to run the writeLines('PATH="${RTOOLS40_HOME}\\usr\\bin;${PATH}"', con = "~/.Renviron") command in R. Then restart R, and it should compile the model.