Pkgbuild::compile_dll() on previously working rstantools-based package fails with "file too big" error

Your story is interesting; glad you solved the problem.

If I understand well, your package which was compiling within 15 minutes is now taking 15 hours ?
I encountered a similar problem with a rstantools-package when I moved from R 3.6 to R 4.0, see my post.
In fact, the problem was that the C++ toolchain was misconfigured. I followed the instructions for R4.0 in this vignette and I was then able to compile “as fast as before”.