I’ve been working with Nic (dev ops contractor) to improve the performance regression testing for develop and PRs and so that system has been a little unstable over the past week or two. But part of that instability seems to have been a legitimate performance regression of about 14% on at least this model https://github.com/stan-dev/example-models/blob/master/bugs_examples/vol2/schools/schools-4.stan that happened sometime after cmdstan commit eda879ea7fda18 (as in, that was a commit where performance was good and d09bd7db87dc83b838d51ee5f6c79434a3923fe9 is a commit where performance is 14% worse for that model and some of the other models in that directory.
Here is how you can try to reproduce this locally:
git clone --recursive https://github.com/stan-dev/performance-tests-cmdstan cd performance-tests-cmdstan ./compare-git-hashes.sh example-models/bugs_examples/vol2/schools/ develop eda879ea7fda18 false false
Does anyone have time to dig into this? Help would be appreciated in any or all of: reproducing the finding, finding out which commit caused this, figuring out why that commit caused it, and fixing it :D