Installing RStan on Mac Mohave

Perhaps try Stan with Mojave MacOS first? Then install.packages("rstan") again. There is more advice for Mojave here: Dealing with Mojave.