I started working more with AI to code, and found myself putting some basic work on making a custom chatgpt for coding in Stan. I know some have mixed feelings about these, but I felt this can help me make less mistakes, and get started quicker with fresh code. So, I just gave it 2.36 version instructions and allowed it to read the manuals. You can ask it to revise your code, or prompt to give you a head start on what you need. So far it helps me with changes that were made from previous versions, and get a good start when I need it. Basic stuff really, but I thought I might share this here in case it help others (you need an openai account, but should work with the free version as well):
I’d say most people are probably past arguing against the use of LLMs for coding in all contexts, so my suggestion would be one of the most boring things in statistics and LLMs: benchmarking. I have had mixed results when using LLMs assist with coding, and have doubts myself about how useful it is overall, so maybe (in a very bayesian fashion) applying it to common use cases – and compare it to vanilla chatGPT/Claude/Gemini/Copilot – and measuring performance (in whatever metrics may be relevant, at least at a first pass).
At this day and age, like using a template, using an LLM (short of vibe coding) to generate boilerplate code is probably just how things are done in the 20s, so having a community-oriented tool is better than not having one.
I generally agree with this. As someone who has worked on systems that integrate LLMs for a bit now, the number one lesson I’ve learned time and time again is that measurement is king. The particular human-computer interface of an LLM chatbot is especially prone to people having strong vibes-based perceptions of efficacy. Without some kind of metrics to fall back on, you really end up flying blind.