I have FINALLY managed to fix the problem and get rstan working and, install packages from source. For the benefit of anyone who finds this thread and faces the same issues, here is what fixed the problem for me. Note, I am using Windows 11 and R version 4.3.0. I suspect this is specific to Windows and Rtools43.
While I am still guessing, it looks like my issue arose because there were lingering conflicting rtools configurations within my path that were stopping the correct version being used. Here is how I fixed it.
First, uninstall all versions of Rtools that you have installed.
Next, search for “system variables” in the search bar and click “Edit the system environment variables”. From there, go to “advanced” and click “Environment variables”.
You will see a whole load of user variables and system variables listed. Go through the list and delete ANY mention of Rtools in any of these places. Specifically, look in “Path”. Do take a lot of care here though, as I bet deleting the wrong thing here could cause all sorts of havoc. If you aren’t sure, double check before deleting anything. Also, make sure to only delete the individual lines that refer to Rtools - don’t delete the entire path/section.
Once you are certain all mentions of RTools has been deleted, click “OK” and close this window. Then, reinstall RTools43 and ensure you install it to the default location (C:\rtools43). Accept all defaults during the installation.
Go back to the Envirionment variables and check that you have a new “System variables” entry that is called: RTOOLS43_HOME and it should have a value of C:\rools43. If you have, great! Everything should now be working.
I hope this can help somebody in the future.