So Steve brought this up on github (I deleted my responses) and was talking about using AI generated code. I met some people that tried to use AI generated code to build a website, but couldn’t deploy because lack of programming experience (could not debug, it was a mystery).
But if we can use it to automate writing code, and test it thoroughly to take care of some issues, I don’t see why I shouldn’t embrace automation.
Admittedly, I am behind in the LLM game, I’ve been corporate so I wasn’t allowed to use LLM technologies on their codebase (I would be fired immediately), but wondering if anyone’s used this and compiled on a production code base in practice, and merged? It seems like it could be useful.
I have not actually tried to compile C++ generated code, and specify in practice, but it could help scale.
Granted, if there are issues, then you still have to be able to read compiler errors (SNIFAE). I think corporate environments people may be using it.
Just curious to hear peoples thoughts, because I can trust this community?
I am all for automation.
So like testing, using correct packages, and versions?
Sometimes I get responses using the Google AI that’s not totally correct, but mostly, so it looks like it could be generated, but need some corrections. I am hesitant, because sometimes debugging someone else’s code becomes more of a hassle than using someone else’s.
I believe the Stan Governing Body @SGB is in the process of working out guidelines for AI usage.
I had been using AI for just about everything these days. I used it for all of my plotting and data munging. I almost always used it as a code reviewer during development and before submitting a PR. I used it all the time to explain libraries I don’t understand and also to explain mathematical concepts I don’t understand (e.g., it’s really good at converting measure-theoretic notation to undergrad applied math notation and also checking algebraic derivations in LaTeX).
I cancelled my GPT subscription and am not sure what to do going forward now that all the tech companies feel like James Bond supervillains.
I use it a lot for things I don’t have much experience with but need done or as a reviewer. Another thing I use it for is prototyping. A lot of the code written for my binary Stan format proposal was written by Opus for example. But you can’t just blindly let it do it’s thing yet in my experience.
I stopped using it so much. I often feel repulsed by AI generated content. It’s like we’re living in a Philip K. Dick novel and my mind is revolting against the facade of meaningful dialog when, in fact, it’s intellectual junk food.
I recently started reading the Dune novels for the first time, and the fictional history of that story including a literal revolt against computers didn’t feel quite as alien as I imagine it would have to me at one point in time
That’s a very different application than writing code though. If I want to communicate something, I write it myself. But if I want to build something, I use tools and that can include an llm.
I love this conversation and look forward to seeing the guidelines the SCB proposes. I’m also looking forward to seeing how the use of AI is author-reported in publication.
I’ve used AI over the past year primarily for model framework. This is useful, but I do always feel a little over my head and wonder if the code returned as eloquent/efficient as it might be if not AI. My largest concern I think is the potential loss of hard-earned but perishable skills as one becomes more reliant on AI for code generation. Further, the incentive to be productive is strong, and so one may be more apt to accept quickly attained and functional code even if not complete understanding the mechanisms.
just my 2 cents. Again, really like hearing the experiences of others on this.
Me, too! Especially the image slop that everyone seems to be pasting into their presentations and web pages. But even the image generators can generate nice images if you put some work into it. I use them outside of work for roleplaying game content creation all the time. For instance, I made a bunch of great tokens of 1960s European cars for a historical Fast & Furious knockoff.
The dialogues are terrible. I just cancelled my GPT subscription in part because I hated everything OpenAI said, and partly because I really hate their new click-bait agent.
I’ve found the agents very useful when I’m struggling with a couple of awkward sentences. When given limited tasks like rearing a few sentences, they can be great. They’re also great at proofreading and findining inconsistencies in either code (i.e., doing code review), algebra (e.g., proofreading my LaTeX proofs), or text.
If you don’t like it just doing things for you, have it teach you how to do them. It’s an excellent math tutor because you can drill down on anything to just about any level.
For instance, I asked GPT to write a rank algorithm for a matrix (to convert values to ranks to implement ranked R-hat). I could have written this myself in 10 or 20 minutes, but the code I would’ve produced would have been terrible. It turned out the right answer was to call argsort twice. Apparently, this is a known algorithm, but I didn’t know and it and would have just done it the naive way. So if you pay attention to what it outputs and make sure to check everything thoroughly to the point where you understand everything it wrote, then I don’t think you need to worry about losing skill. In fact, I’ve found it made me better.
This is good to hear, and I’ll be happy to be wrong about this. Maybe as you suggest the key is more a change in workflow, perhaps thinking of AI as more of an intelligent but somewhat unreliable narrator and using the output as a prompt to dig deeper into the fundamentals.
But then in practice, for example, in biostatistics, there’s no known solution, and even solutions I’ve seen in papers on “unsolved problems,” they use methods that scale extremely poorly, for example combinatorial variable selection. Has any one tried to use this? Or does this have to be an entirely human task? I’ve looked at papers where I have worked on, and seen the output, and I can see where they put stuff out that’s not totally correct. But for automating code, sure.
Second question, what AI’s are we using to generate code? I have only asked ChatGTP 3-4 questions, I never thought about it.
And after using it, which specified AI models have been tested out, on, for example, complex finite elements simulations, of, a glass furnace for example?
Stan the models are pretty straight forward, meaning not very complex systems but helpful, for making desired inferences.
Moreover, isn’t agentic AI just a positive feedback loop (which can be prohibitive)? I can’t find anything I’m looking for on Google anymore. It’s more, additive interference than productivity. Any information assistance? This is driving me crazy.
It’s like search engines have become useless all of the sudden. I am a child of the internet era, but some agentic AI with an objective function that feeds itself is useless. I’d rather go back to the dewey decimal system and the library. It just interferes.
Alright, so Aki just fixed a PR with AI generated code. I don’t have a claude subscription yet, but I’m wondering what prompt he used? I haven’t experimented with this too much. I feel stupid!
and I may sound stupid, but these models have gotten a lot more complex, are we training separate dedicated LLMs on stan models, and then do we have a dedicated one for the math library? Because if I use the generic ones, they fudge some stuff. Some stuff not exactly correct. But I guess you have to expand the corpus to another desired set? That is actually valid? Am I sounding stupid? I am trying to
catch up here.
Alright on my linkedin I generated a GP regression model, and it spit out slop. I compared it with code that I wrote and built for testing. just like a bunch of nonsense. Yes, sorry I wasn’t looking into it properly. Whoops.