Just wanted to give a heads-up on the plan for the next release - version 2.32
The proposed plan is:
Please let me know if you think any of these dates should be moved. We are pretty flexible this time around, barring any surprises, this should be a pretty minor release.
If you need a pull request reviewed or think a bug fix is a high priority for this release, please use this thread to report it.
Just want to double check: is 2.32 going ahead with removal of the deprecated array syntax? If so, it’d be worth thinking of this as a major release in terms of its impact on the community. e.g.
I’m hoping a bunch of things can get finished in Stan math
We should fix this normal_lcdf and normal_lccdf give infinite gradients for infinite inputs. · Issue #2881 · stan-dev/math · GitHub as well.
Let’s get the hypergeometric functions exposed in Stan b/c these are merged
I added a stanc3 issue Expose `hypergeometric_2F1` and `hypergeometric_3F2` · Issue #1285 · stan-dev/stanc3 · GitHub
I’m working on the PR which will formalize this now, but yes, that is currently the plan. For 2.32, the expiring deprecations will become errors, but will still be parseable and --auto-format’s --canonicalize=deprecations will recognize them and produce the updated syntax.
For 2.33 and on, they will not be recognized at all. It’s looking like Tuples will finally be able to land in 2.33, and they are really reliant on the new array syntax, so a delay in the deprecation schedule could start holding up features. If there are things we can do to make this less painful for the R packages I’d love to hear them
Tagging @paul.buerkner, as brms is by far the most prominent R package that depends on running Stan code via both cmdstan and CRAN’s rstan. Edit: the only strict dependency is on CRAN’s rstan, but running via cmdstan is important functionality.
If Paul feels good about the way forward, then I’m happy.
Is it planned that the auto-formatter will be able to digest the old Stan code with 2.33 and onwards?
I’d guess that as long as all this means is to switch on some parser option, then it’s manageable. In case the old syntax cannot be parsed at all with 2.33 going forward, then that could be an issue, I guess.
(great to read implicitly that tuples are going to land in 2023 presumably)
The plan is that in 2.33, the compiler can forget about the old syntax entirely. If the goal of deprecations is to allow the language and compiler code base to evolve, freezing the auto formatter would prevent us from actually deleting that old code.
Of course, if you have code from 2.19 or whatever version before 2.33, you can run the auto formatter from an “old” version (e.g. 2.31) as part of your update cycle and that will always work. It might be reasonable to provide a really striped down binary which only does migration of pre-2.33 code (e.g. not even compilation), since that binary would never really need to be updated.
For things I would like to see merged before the release:
get_dims and get_param_names: Add flags to get_dims and get_param_names by WardBrian · Pull Request #1245 · stan-dev/stanc3 · GitHub and Extend `get_dims` and `get_param_names` in model_base by WardBrian · Pull Request #3139 · stan-dev/stan · GitHub - @rok_cesnovar is reviewing them both currentlystansummary: Add argument to only output some parameters from `stansummary` by WardBrian · Pull Request #1143 · stan-dev/cmdstan · GitHub
Hey @spinkney, this sounds good! Is there anything else to do for me that helps finishing this PR until April?
I predict we get a lot of problems if RStan in CRAN is not supporting new array syntax when the old syntax stops working in a new release. It would be good to be certain that there is a new working CRAN RStan before removing the old array support. Ping @bgoodri @andrjohns
I don’t think so. I know @andrjohns is reviewing your PR but he’s swamped with Rstan cran stuff.
Before the freeze we need to either merge the PR which removes the deprecations or put together a new one which moves the removal versions back, so if we are deviating from the deprecation schedule it needs to be an active choice made soon. I’m not sure what the status of CRAN submission is, last I heard it was close (@andrjohns @hsbadr)
I believe 2.26 should be ready and there is only one package which will additionally fail under 2.31 (but they have a patch submitted which should be out to CRAN soon).
But @bgoodri is handling submission, so will let him clarify
I’ve also consolidated the current status of rstan downstream package failures/patches here: RStan & StanHeaders Downstream Patching Status · Issue #1053 · stan-dev/rstan · GitHub
There are now no additional failures under 2.31, so once StanHeaders 2.26 and rstan 2.26 are on CRAN, the upgrade process should be painless.
Given this (assuming a CRAN submission for 2.26 soon), one option could be to push the deprecation removals back one more version. Then rstan users will have roughly a release cycle to prepare - and we’ll have the same to prepare downstream packages.
On that note, would it be difficult to build a stanc.js where the deprecations are errors? Then I can run the reverse-dependency checks and see what patches are needed
EDIT: Never mind about the stanc.js, I didn’t think to just pull it from the PR
I recently did a check of our own CI to make sure none of the Stan or CmdStan test models would fail, so I have a Jenkins run with artifacts here: Artifacts of stanc3-test-binaries #25 : /bin [Jenkins]
Looks like the downstream failures are pretty extensive with the deprecations-as-errors, but I’ve suggested using the auto-formatter during the package installation as an interim solution: Auto-format Stan models during rstan_config()? · Issue #113 · stan-dev/rstantools · GitHub
That will only work for 1 version post-removal, so it’s not the best long-term solution. At some point the .stan files themselves will need to be updated on disk, not just during parsing.
Is it necessary to remove that functionality from the auto-formatter? It could be useful for users in the same way that it has been for deprecated syntax.
Not that I’m advocating for this as a long-term solution in any way - I wouldn’t want it to be in place for more than a version anyway, otherwise we’re promoting/enabling incorrect syntax
If the point of the deprecation policy was to improve the codebase, then yes it is necessary. The code the compiler is required to carry around to parse the old syntax is the same whether we’re auto-formatting it or compiling it down to C++, so we can’t actually move on until it is really gone.
We’re not going to be intentionally mean about this, so things which are “cheap” to update (e.g., turning multiply_log into lmultiply) might stick around longer or even forever, but we only guarantee it for one version.
If you know that the source code you’re dealing with is 100% old syntax, there’s no reason you couldn’t use a 2.31/2.32 compiler to do the update forever into the future, but if you’re not careful with enforcing the deprecation policy it’s easy to imagine a year from now someone will try to write code which uses e.g. a new function from 2.33 with the old array syntax, and suddenly there’s no compiler that exists which can handle both those things.