Hi,
had the same problem as you since upgarding to R 4.0 and followed the wiki to install Rstan, and amend the Makevars.win files.
Still, I got error messages “cannot allocate…”
Turned out my code was also using the recommendation to avoid recompilation of unchanged Stan programs by calling
rstan::rstan_options(auto_write = TRUE)
However my stanmodel.rds were outdated. Once these were removed and models were recompiled, everything was fine.
Could that be part of your problem?
Cheers