Unfortunately that’s expected behaviour as the sample size increases.
When you call cores = 8, all it does is specify the number of MCMC chains that can be run in parallel. So if you specify cores = 8, but are using 4 chains, then you will only be using 4 cores. If you want to take advantage of extra CPU cores that you have available then you’ll need to update your model to use one of the parallelisation frameworks (reduce_sum or map_rect). Note that if you want to use reduce_sum you’ll need to run your models through cmdstanR, since that functionality is not yet available in RStan.
You don’t need to do anything to manually change how R uses memory, it will automatically use as much memory as it needs.
Run things under wsl.
To expand on Sebastians answer a little, we commonly see models running much faster under Linux than Windows (on the same hardware), there’s more information on that in this thread. However, you don’t need to switch to a new operating system to take advantage of this. With the Windows Subsystem for Linux (WSL) framework, you can operate a Linux environment from within your existing Windows OS. Running Stan under WSL has seen speeds comparable to native Linux.
The most user-friendly way to take advantage of this is through installing RStudio Server on your WSL install, which lets you interact with the WSL R installation through your web browser. The RStudio support pages have a great guide for setting this up which has worked really well for me: