Monthly language dev meeting

Right. It’d be possible to code all of our constraints by hand using only the real type. So the question isn’t about adding expressiveness, but adding convenience. I like to ask myself if programs would be easier to write or easier to read with a feature. User-defined functions is a good example feature for comparison. They literally don’t add any expressiveness to the language (even mutual recursion can be unrolled into an iterative loop that gets inlined in place of any function call).

Right now, user-defined transforms get spread out over the parameters, transformed parameters (the actual transform) and model block (the Jacobian adjustment). So I’m wondering whether there’d be a way to encapsulate all of that in a way that’d be more understandable. For instance, compare this:

functions {
  vector my_simplex_constrain(vector x) { ... }
  real my_simplex_constrain_jacobian(vector x, vector y) { ... }
  vector my_simplex_unconstrain(vector y) { ... }
}
...
parameters {
  my_simplex[K] x;
}

vs. something like this with the same functions:

parameters {
  vector[K - 1] x_raw;
...
transformed parameters {
  vector[K] x = my_simplex_constrain(x_raw);
...
model {
  target += my_simplex_jacobian(x_raw, x);
}

There’s also the issue that this won’t work with initialization properly (where we use unconstrain to map user-supplied values to the unconstrained scale)—that’ll all have to be done by hand on the outside.

P.S. Rather than get dinged for more inappropriate behavior, I’ll leave it to you to decide where this kind of discussion goes on the forums. I was just trying to report the minutes of the meetings here, not open a larger discussion on language features. So feel free to move this or even delete it.

2 Likes

OK, let’s move the meeting up a week, as I’d like to be able to attend, too, and nobody else responded. So the next meeting will be:

  • 7 October 2021, 10 am NY Time (1 hr before the Stan meeting)

I’ll revise the top post to match.

2 Likes

I agree that convenience is important. The intro I gave was probably a distraction and I should have said something more like “Hey, while you are discussing constraints, could you also consider this? No pressure, but it would IMHO kind of help.” and just avoid the rest. Agree that meeting thread is usually not a good place for long discussions about design and those deserve their own topics. I think the meeting thread could be a good place for people watching from the sidelines to provide brief feedback/wishes/thanks/… as I don’t think those would work well elsewhere. But I think that the meeting organizers are free to setup a mechanism that works best for them for getting feedback from the community, so its mostly up to you.

Thanks again for running the meetings.

If anyone would like to get the Google invite for the meeting, please let me know. I’ll set up a standing invite for the second Tuesday of the month starting next month.

Please also let me know if there are agenda items for this meeting.

1 Like

Hey Bob, could you send me an invite?

Notes for Oct 2021 meeting

Design doc on deprecation

@WardBrian asked for volunteers to read his design doc on deprecation. If nobody expresses objections soon, we’ll go ahead and make this policy.

Type checking refactor

@WardBrian also asked for volunteers to review and think about a refactor of the type checker. The goal was to make it shorter and more readable. The PR reduces the size by 500 lines of code. The major changes are

  • the global symbol table is replaced with a functional map
  • the validation monad is removed
  • there are about 500 fewer lines of code

Hopefully, this will open the way for function overloading, tuple types, etc.

@WardBrian also volunteered to help update outstanding PRs for things like tuples and closures.

Function types in the langauge and the closure PR

@nhuurre has an outstanding PR for closures. I’d very much like to get this into the language. As it stands, it only implements closures, not lambdas. In order to have lambdas, we’ll need to introduce a syntax for functional types into the language. There’s a proposal in the closures design doc, but of course that’s not binding if there’s a better way to do it.

Apparently, functional types are already treated as first class objects in the type checker and the hang up is more on the parsing side. Closures are actually the more useful feature, but it’d be nice to get lambdas in that we can use in contexts where appropriate. I’m OK merging just the closures part of the closures design doc and not worrying about the lambdas yet.

TensorFlow back end

As things stand, there’s a lot of logic in stanc3 dedicated to generating TensorFlow back end code. It only supports a bare-bones set of a few handfuls of functions and is reportedly slower than the Stan math back end. The question came up as to whether we should continue to support it or move it out into its own module.

It’s not clear if anyone using it. This was mostly developed by @seantalts and @Adam_Haber, so it’d be interesting to hear from them on whether it’s still viable.

Even if it is being used, it was agreed that it’d be nice to move it out of the stanc3 repo so it’s not an ongoing drag on development. The consensus seemed to be that we try to break it off so that it could import stanc3 as a submodule for most of its functionality. Nobody thought it’d be worth fully factoring Stan math out of the parser, but it would be OK from the TensorFlow back end’s perspective with some extra stuff they didn’t need in the repo (there was general objection to splitting stanc3 into two repos for code generation and for parsing/type checking).

@Bob_Carpenter suggested all the cool kids were moving to JAX, so maybe we should target that instead of TensorFlow.

If we move things now, everything that works in 2.28 will continue to work going forward just by using the 2.28 release.

Transforms

We had a freewheeling discussion of variable transforms and how much users can be expected to know about underlying implementations. For example, should users be able to rely on <lower=0> constraints being implemented with log transforms or should that be considered an implementation detail? The conclusion seemed to be that it’s an implementation detail, but also critical for understanding how sampling was going to work. The Reference Manual does detail the transformations, but it’s not prescriptive in saying that they have to be of this form.

This is one of the motivations for user-defined constraints. For example, a user may prefer a soft-abs inverse transform to an exponential inverse transform for variables declared as <lower=0>.

We then realized that constraints and transforms are doing something different. It’s always been awkward for the doc and explaining Stan that constraints work to trigger transforms in the parameters block, but are just validation tests in other blocks. If we separate constraints and transforms, we could have constraints parameter variables act like other block variables. For instance, we might have

real<transform = softplus, lower = 0> a;
real<transform = exp, lower = 0> b;

This would open the gateway for us to have multiple forms of each transform for different computational purposes.

But this’d be a major breaking change in the language if <lower=0> doesn’t trigger a transform all by itself in a parameter’s declaration.

We decided that if users define potentially non-sensical transforms, one place to warn them would be with pedantic mode.

See you next meeting!

3 Likes

The November meeting will be Thursday 11 Nov, 2021. Let me know here or via email if you’re not on the list so far and would like a meeting invite. Thanks.

2 Likes

I sent out a standing Google Calendar invite with Zoom details. If you didn’t get an invite and would like to attend, please let me know and I’ll add you to the invite list.

There’s not going to be a language meeting today. Sorry about this—it’s my fault as I spaced after getting a booster shot yesterday. We’ll resume in January.

1 Like

The next language meeting is tomorrow, Thursday 12 January at 10 am NY time. It’s an open meeting and if you’d like to be on the Zoom invite list, let me know.

Here we are again. The next language meeting is this morning 10 am. Let me know if you’d like to be on the Google Calendar invite or if you’d like me to just send you a link to the Zoom for the meeting.

Summary of February 2022 Language Meeting

We spent this meeting talking with @mitzimorris and @spinkney about reporting project status. Sean’s updated the stan-dev org to use GitHub projects, which I understand are like the way Agile projects manage with index cards on a wall.

Sean thought we could do better in getting the word out about what’s going on in the project for a bunch of reasons: letting users know what’s coming, letting developers coordinating, informing the @SGB about what’s going on. He cited projects like Pyro’s use of Twitter, Medium blog posts, etc., to get the community excited.

We tried to work through what that would look like for a project our size and how we’d get the word out. Sean said starting with one feature is better than nothing, so I offered to write one feature up at the level he was asking for. Here goes.

Complex Number Support

The current status of Stan’s complex number support is that it works for complex scalars at the level of the language and math library. And at the level of arrays. We’ve added support for covariant typing, meaning that we can now assign int to real to complex, and that also works for arrays. We now need to add support for vectors and matrices, which requires work in the parser and code generator and also in the math library for polymorphic arithmetic (e.g., multiplying a real and complex matrix) and covariant typing. There is work to do on extending Eigen’s low-level BLAS level of functions to support. We could also use an actual complex-number based application for our user’s guide. After we get vectors and matrices, we want to add support for complex linear algebra (e.g., Schur decomposition and asymmetric eigendecomposition), and for fast Fourier transforms. Pretty much every single scalar and arithmetic funciton could be productively specialized for complex numbers. It’d be nice to think about the equivalent of var_matrix on the real side for the complex case. There is remaining work to do to generate actual complex numbers in the Python, R, and other interfaces rather than just returning real and complex components. There is a design document and issues and branches in progress in the language lib, the math library, and the interfaces.

Other language features

I also mentioned there are about 20 different features we could list this way that require work across the language, math, and interface repos. In no particular order:

  1. tuples
  2. closures
  3. lambdas
  4. 64-bit integers
  5. provide sample back end implementation in JAX
  6. ragged arrays
  7. sparse matrices
  8. regression syntax a la BRMS
  9. comprehensions a la Python (for efficient GPs and static var_matrix optimization)
  10. parallel loop constructs and more general map/reduce functions
  11. new simple types (sum-to-zero, probability, etc.)
  12. orthonormal matrix type for circular/hyperspherical/rotation stats (e.g., Stiefel and Grassmanian manifold types)
  13. vectorized truncation (this one’s relatively easy compared to the others)
  14. differential algebraic equation syntax and solver
  15. stacked/composed transforms, user-defined transforms
  16. user plug-ins at the C++ level
  17. user-defined gradients, including integration with ODE solvers
  18. integer and complex output types in language/interfaces
  19. expose transforms to users; allow Jacobian adjustment in transformed parameters
  20. more code transform and gen optimizations in compiler
9 Likes

Thank’s for putting the list together.

I’ve been working a lot on the complex container support in the language, so I’ve made a ‘project’ to track where things are at the moment. It would be worthwhile to create issues in stan-dev/math for the special functions (FFT, etc.) and function specialization we will need for complex matricies.

I’m curious how using a project will feel for this task, so it’s more of an experiment.

Here’s the link: https://github.com/orgs/stan-dev/projects/3

1 Like

I really like the way the projects page looks. I don’t know how you’d do this for much bigger projects with more complex interdependencies. I was expecting something that looked more like a PERT chart (graph of issue dependencies) and less of a progress report in terms of started/ongoing/finished.

I think it would be nicer if there was more of a way to express direct dependence (this issue is a prerequisite for this other issue, etc). But, for a big picture view I think it’s pretty nice. It also does some things semi-intelligently, such as if you close an issue or merge a PR it automatically gets moved to the ‘Done’ column etc.

Thanks @Bob_Carpenter for the nice summary and everyone in the meeting taking the time to discuss project status updates.

The goal will be to give updates during release cycles of a few features/projects. It can be very high level and concise (or more detailed if you prefer). The goal is to get people excited about Stan, about what’s being worked on, and the people to tag if you want more info on that project. We’ll also experiment with Github projects as a way to track things more consistently and a place anyone can go to see the status, see the current issues, ask questions, and contribute (hopefully!). Thanks @WardBrian for putting an example of projects together!

I spoke with @rok_cesnovar about adding a future/preview note in the 2.29 release notes. The complex number brief you gave will go in (thanks again!). I’m looking for one or two more, maybe tuples (lots of work has been done recently), varmat (it’s partially completed, what’s left?), pathfinder, closures, robust VI, and/or cmdstan I/O? I’m just calling out a few but any project can be added. I’ll tag some relevant developers @rybern @mitzimorris @WardBrian @wds15 @nhuurre @stevebronder @bbbales2 @andrjohns @Dashadower @hyunji.moon @yizhang (these names come to mind, if I left anyone off who contributed it was unintentionally).

I personally wouldn’t say much about tuples. I put in a lot of time yes, but it was primarily just getting Ryan’s PR back up to date with the current version of the compiler. Still a lot of work left there, more than a lot of the rest of that list I think

I think that still counts. It would be

  • Tuples PR is up to date with Stan 2.29 and work is progressing on adding this into Stan soon (or whatever release you’re targeting)

I hope this shows how brief updates can be.

1 Like

I’ve got three that are candidates:

  • Quantile functions
  • Packaging cmdstan for linux
  • User-defined gradients

Let me know if any those would be good to add and I’ll write up the brief notes

4 Likes

How could I forget qf! Yes that’s also a great project to track in GitHub projects. I think it would have issues for all the pFq stuff, and encompass some of what’s in @bgoodri quantile functions design doc

1 Like