Ok, found it. Its actually not that tricky.
You need to start the Rtools bash shell like the Rtools guide on installin package dependencies says https://github.com/r-windows/docs/blob/master/rtools40.md
image from the guide:
Then write
pacman -Syu mingw-w64-x86_64-make
and Enter, then you will be prompted to install. Then close the window.
Make sure your path has both
C:\rtools40\usr\bin
C:\rtools40\mingw64\bin
Open shell (or restart Rstudio) and you should be good to go.
cc-ing those that might get windows and cmdstan related questions: @wds15 @mitzimorris @ahartikainen