After watching the walnutpie video by @Bob_Carpenter I decided to spend some time with walnutpie. When I first saw it announced on Andrew’s Blog I commented that trying out muon as an alternative to adam could be interesting, as it just had started to make waves in the llm world. That was the entry-point for a 3-day dig into a number of areas of the entire Stan stack. Given the scope but single-source, I figured it might be useful to have a single point to discuss this effort, both on a content and process level.
Disclaimer: I don’t understand most of the math deeply and the code is entirely written by GLM 5.3. My contribution here is time spent on guiding the agents, pointing them at areas I thought might be promising, suggesting things to try and making vague, hand-wavy side notes that would sound smart in hindsight if they worked but otherwise were just proof of my shallow understanding. I do have a background in low-level/close-to-hardware programming which definitely helped.
I tried to make sure that the PRs contain everything to help those of you with a stronger math background and, in case you don’t want llm-generated code in any of the projects, to re-derive the solutions from the presented analyses.
I do not expect the PRs to simply get merged. I understand them more as Issues with added proposals.
`Stan’s AI Contribution Policy asks that upstream submissions be the work of an accountable human contributor, so I am keeping these on my forks as documented proposals
stan-math
#1 guard eigenvector adjoints against degenerate eigenvalue clusters. The reverse-mode adjoint divides by eigenvalue gaps. On exactly repeated eigenvalues the gradient is NaN. On rounding-degenerate spectra, such as a jittered GP kernel, it disagrees with finite differences by 30–50% in every build. On a kronecker-GP model the fix takes bulk-ESS from 29 to 411. JAX merged the same idea for forward mode in April.
#2 square() multiply instead of std::pow(x, 2). Under default -fmath-errno the compiler cannot remove the call, so it stays a ~105-instruction libm call in hot paths. That is worth −9–15% per gradient on GP models.
#3366 fused, packetized value+partials kernel bernoulli_logit_lpmf evaluates log1p eagerly for all N elements per gradient (Eigen’s nested Selects don’t short-circuit), making glibc __log1p the single largest symbol at 13.2% of program instructions on a hierarchical 2PL model. A fused value+partials kernel measures −22.8% Ir/gradient and −15.3% wall at AVX2+FMA.
walnutpie #7 — init guard: never start a chain at a non-finite-logp position. File inits fail fast: 0.16 s instead of 8.2 s of pinned, zero-ESS draws. Random inits get Stan’s rejection protocol as CLI policy.
walnutpie #8 — freeze clamp: a fallback instead of a macro_time must be in (0, inf) abort at the warmup freeze when the adapted step degenerates.
walnutpie #9 — find_reasonable_step improvements: momentum scale inverted against the sampler, fresh momentum per probe, asymmetric accept statistic. The probe returns the right step on the pinned cell, and the short-warmup blr class goes from bulk-ESS 5–9 (pinned) to 779.
# 335 compile_model cache ignores make_argscompile_model returns any cached <stem>_model.so next to the source without comparing the requested make_args, so building “with STAN_THREADS=True” against an already-built default pair silently hands back the default binary.
The cache returning the default binary is how you can believe you have STAN_THREADS builds while actually running default ones.
Update: Turns out that I missed the implementation of the stan ai policy and was only looking for root CONTRIBUTING.md files, thus missed the .github/ ones.
While we do not accept fully agentic solutions, if you have the time I encourage you to write all of these up as issues on the relevant repos (they don’t need to be long!)
At StanCon we had a contributor workshop and a hackathon, and some of these look like good “good first issue” opportunities
I probably should have realized this and brought it up when we were discussing the AI policy initially, but I realize now that I’m a bit confused by what we mean in the policy when we say “we do not accept fully automated or agentic submissions”. If @scholz spent time “guiding the agents” then to me that’s not the same as a fully automated submission (although I guess it depends on the degree of guidance and it’s not entirely clear to me from the OP to what degree the agents were guided). It’s one thing to say to a coding agent “go solve this problem” and then just submit the work the agent did without checking it or iterating on it. It’s another thing to guide the agent, check its work, have it try different approaches, etc. So perhaps we need more clarity in the policy to distinguish these scenarios?
Personally, I don’t have a problem with PRs to the R packages that are coded entirely by an agent so long as the human submitting it has really guided the agent in a productive way, helped it arrive at a high quality PR and can justify the choices made. Does the Stan AI policy prohibit such PRs? It’s not clear to me from the language. (I don’t know if the PRs discussed in this thread meet that standard, I haven’t checked, but it made me wonder about this.)
If the PR description appears to be entirely written by an LLM, I would take that as a pretty clear indicator, but it is ultimately unclear how to make that distinction.
I’ve personally been following an as-if type rule: If the submission looks like something a human would write themselves (either because they did, or they have their agent sufficiently prompted, or they audited the outputs well), it automatically clears that bar
First of all, for me, “fully agentic” means: prompting an agent and submitting the result without looking at the code or spending time to write a PR description. I agree that guiding an agent, checking its work, having it try different approaches, etc. is time consuming for a contributor. However, where I see problems:
Once a contributor says they don’t understand the math behind a PR, I struggle to see how they could design a testing framework that meaningfully checks it for correctness. That work then falls to the reviewer, who first has to work out what the PR is even trying to do before thinking up reasonable test cases. I don’t think that should be the reviewer’s main job.
Fully agentic PR descriptions are overly verbose and uninformative, leaving the reviewer with the task to understand what the PR is doing and creating a mental model. I find it very important that the PR description is written by a human and provides an understanding of what the changes in the PR are doing and how this is happening.
Finally, if nobody on the contributing side has read the code, they can’t judge whether the code is consistent with the existing code base. That question just gets transferred to the reviewer, who may or may not want to take on that maintenance burden.
That’s why I like @WardBrian’s stance: such contributions can still be very useful as issues, where the ideas and findings are shared with the community without committing anyone to reviewing unchecked code. As PRs I think they’re problematic.
Thanks @Florence_Bockting. I agree with all of that! I guess my doubt was mainly about whether we were considering “fully agentic” to include all code being written by an LLM even the user was very involved and understands everything (as in my hypothetical about PRs to the R packages). I think we’re on the same page.
Thanks @WardBrian, I think the as-if type rule is a good one
Thanks for everyone who engaged. I wanted to respond to some of the points that came up:
I misjudged how submissions of this scope usually run here and I am still somewhat uncertain of the desired form of contributions. The guidance I have found focuses on the creation process rather than the outcome. Especially the desired level of detail is something I’d ask for input on, as a full analysis is a lot to review, but a one-paragraph issues gets the findings rediscovered from scrath.
Something that could work for simpler things could be a short issues (claim, repro, numbers) and then a link to the full write-up from my fork for anyone who wants the derivation.That could be a fit for some of the correctness items I found, like the bernoulli_logit tail-sign fix, three --O1 wrong-code bugs, or constraint validators that accept out-of-domain input.
My commend about not having a deep understanding of the underlying math was initially ment more as a joke with a core of truth in there. While not entirely foreign to theory, I am a software engineer and not a mathematician, so my verification method would usually be exact output replication rather than re-deriving formulas (which was how I found an RNG whose stream depended on the system time rather than only the seed).
Given that, I have found a number of additional issues and possible improvements across the entire stan stack. Besides a lot of smaller fixes, some of the larger items with numbers:
stanli: fused-JIT regions (0.39× the instructions per gradient vs the current interpreter)
walnutpie: robustness guards that fix many chain-pinning issues I encountered (+57% aggregate bulk-ESS across the 21 posteriordb models vs stock)
math: adding gathered likelihood primitives (-17% to -65% on indexed-coefficient model classes) and batched arena records (eg. −18% total instructions on a 2PL model)
bridgestan: uniform AVX2/FMA builds for bridge + model objects (−11% to –21% wall time)
For scale: the full stanli fork stack vs the current CmdStan release peaks around 22× ESS/s on centered eight schools.
Given that some of these would be more involved than a small fix, I’m open for suggestions on how to move this forward.
I think the only incorrect level of information is not opening the issue, to be honest. The more concise the text/reproducer the better, but we can work with basically anything on a bug report. In terms of a time-to-patch metric, my guess is that you’d be better off with shorter issues and letting some findings get rediscovered than with a long issue which is more likely to be filed away as too complex during triage
In particular, I really hope you open issues about
soon, so we can try to fix them before our upcoming release!
For things that are bigger than a bug, the best way forward is probably through our design-docs repo, which gives a place to discuss larger changes and get consensus before any specific code is being reviewed
I think #1, #2, #3, #4 mentioned in the first post would be great to get to upcoming release, too, and all these are quite small PRs with tests included, so I think they would be feasible to include
I’m not sure I totally understand what you are asking for. Are you asking about what bug and feature reports should look like for Stan? If you can link the docs you found we may need to update them to make the process more clear. We should also probably update these templates for agents as well.
For feature requests or bugs you should always file an issue first. For bugs I find it helpful to have a few sentences (really only 3 or 4) explaining the issue and then a minimal reproducible example in code.
To be clear, I do a lot of agentic coding. I have been told through the grapevine I am the #2 token user at my office. If you point an agent at a repo and say, “Find issues!” It is going to find a ton of nonsensicle non-issues. If you ask the agent for an MRE it will stop reporting about 60% of the issues it would otherwise report. If you ask it to explain how likely it is for the ‘bug’ to arise that will remove another 20% of the issues. Human inspection of the MRE and bug path will remove another 15%. So out of 100 issues you end up with about 5 that may be real. And for reference my regular model choice is fable or sol 5.6 etc. .
So for me, especially with something an agent found, I would want some way that I can clearly read in a few sentences (no more than 4) to understand what the issue is along with code I can run in the repo’s testing framework. I think several of the issues you put up today are nice examples of those!
I’ve found with the most recent agents that, the more indecipherable their output, the more likely they are lying (especially fable). So making the agent clearly explain the issue is a good barrier. A few sentences + example gives me enough to estimate whether it is a real issue and then we can discuss how it should be fixed from there. Unless it is really a clear/obvious bug or easy to add feature I would not spend time beyond the initial issue.
I hope to track down the bug soon. I’m afraid to say that fixing the variance estimator bug you submitted the PR fix for did not fix the sampling problem. I was more hoping to find a problem like you found here,
where I’m generating momentum from the inverse mass instead of mass.
I hope not!
I think this is going to be a problem.
Here’s an example that I ran into. I was working on the User’s Guide chapter and used Claude to code up the Jacobian for a VAR(1) stationary parameterization. I have to say I can’t quite follow why Claude did what it did or why it was the right absolute Jacobian determinant. So in some sense, I couldn’t really “understand the math” just from reading it. But I do know what it needs to do and I could write the unit tests to make sure it was, for example, implementing the Jacobian determinants correctly by checking against a numerical solution. And I could inspect the function to make sure it was bijective without understanding why it’s coded the way it is.