Monthly language dev meeting

Great idea, I’d like to join

Just confirming we’ll be doing this tomorrow, 9 September at 10 am. Everyone who responded here saying they were interested should have a Google calendar invite.

I’d like to focus on the following two questions during the meeting.

Where are we?

I’d like to start with a survey of issues that people think are important to address in the short term, either because they’re bugs or blocking something else. This can be anything from the way the code is organized, how function argument testing works, to how it interacts with C++.

What I want out of this is a prioritized list of things that need to be fixed or improved.

Where are we going?

I’d like to collect a list of the big features we’d like to see going forward. By that I mean ones big enough that there should be a design doc.

I can go over my list below, but I’m really looking to hear from everyone about prioritization and other features.

  • closures
  • comprehensions
  • user-defined gradients
  • user-defined function overloading
  • tuples and structs
  • ragged arrays
  • sparse vectors and matrices
  • user-defined parallelization
  • plug-in external function calls
  • variadic functions like generalized sum()
  • inverse cdfs and inverse ccdfs
  • complex vector and matrix types

These I’d like to turn into some kind of roadmap.

4 Likes

Let me know if you’d like me to put you on the calendar invite for the zoom meetings going forward. The plan is to meet the second Thursday of every month at 10 am NYC time (the hour before the general Stan meeting). The math lib meeting is the third Thursday of every month in the same time slot.

I know @Niko has also said he’d like to get invites, so I’m just recording that here to remind myself.

I’ll be producing a summary of today’s meeting and announcing that here soon. Thanks to everyone who was able to join.

5 Likes

I can’t make our next scheduled language meeting (14 October) due to an all-day local meeting I have to attend for work. I suspect that’ll also take @WardBrian out.

So my question is whether you’d like to

a. have the meeting without me,
b. reschedule to a different day, or
c. skip this month.

The third Thursdays of each month are the math dev meeting, but we could meet on 7 October if that works for people.

@WardBrian just reminded me I never posted the notes from last meeting. Sorry about that. Feel free to edit or send me edits if I got things wrong.

Minutes for 9 September 2021

Status reports

Tuples

A high % of this project is done, but @rybern reports some residual issues:

  • nested containers with constraints
  • code gen is complicated
  • need to work out code for new matrix type
  • TO-DO: rewrite code gen (mostly done)
  • TO-DO: parser warning messages

Nested constraints

@WardBrian is working on a spec. There are lots of issues around offset/multiplier not being constraining and how much flexibility we want to allow in composing transforms. So far the only ones we can think of that are reasonable are offset/multiplier (applies first when unconstraining) and lower/upper bounds. And maybe ordered and positive_ordered types.

There’s an issue of whether to work toward user-defined constraints and Jacobian adjustments or whether to just keep our built-ins.

Closures

@rok_cesnovar reported these are almost done in both stanc3 and with the C++ code gen. This seems to be blocked in code review. There are a few deviations from the design doc that need to be included in the doc or at least listed before we write the reference manual and user’s guide doc. It needs larger-scale testing on both eh math library side (@nhuurre has a PR) and in stanc3 (ditto).

OpenCL optimizations

@rok_cesnovar reports these are just waiting on the varmint work to land in the math lib. It then needs a guide on how to add a simple signature.

Varmat

This is a huge efficiency gain from representing matrices as a matrix of values and a matrix of adjoints. It’s largely taking place in the math lib, but the language needs to figure out where it’s safe to use a varmat. @stevebronder has a work-in-progress PR for Stanc3 on this. He says it’s taking a very conservative, safety-first approach. This is running into the precompiled headers (PCH) issues during testing. @rok_cesnovar says all the files are tested, but it’s just Stan without CmdStan running end to end.

General comments

@avehtari suggested we should be more organized around reporting progress so users know which language features are coming. I really wish we had features coming out fast enough this was more of a problem! The issue came up of where to do that: (a) stanc3 wiki, (b) readme.md for stanc3, or (c) GitHub projects.

Documentation issue arising from RStan and CmdStan being at different versions. We need a way for users to easily access doc for the version they’re using and going forward, each new feature should be listed with the version it was introduced so everyone could read the most recent doc. This probably won’t be too much work to maintain, but it’d be a huge amount of work to do the first time.

@WardBrian brought up deprecation schedule and we had a long discussion around how it’ll make maintenance easier, how we can convert old code with the pretty printer, and how we should have a concrete schedule. Since the meeting, he’s turned this into a design doc that’s ready for comments.

@rybern pointed out that the main issue with overloading and supporting variadic functions is technical debt in the type checker, which handles user-defined and system functions differently. @WardBrian has been looking into that. The solution will probably involve more type variables and less explicit instantiation.

What is the syntax for structs going to look like? Perhaps this discussion is premature until we have tuples implemented. The implementation should just follow tuples. We probably don’t want ways to easily partially instantiate structs, though that discussion is ongoing.

How do we plug in external functions? @rok_cesnovar has a Stanc3 issue (#712) for this.

We generally preferred _qf to _icdf for inverse cdfs (aka quantile functions).

3 Likes

Thanks for sharing the minutes!

My 5 cents is that Stan is already flexible enough to let you code any necessary constraints - and nesting constraints in the order “built-in first, custom second” is also possible (if I understand the meaning of “nested constraints” correctly). So allowing “custom first, built-in second” - while definitely helpful in some cases - won’t IMHO bring a very big benefit. IMHO the only thing that’s not possible in the current version is to have a custom transform not apply the Jacobian correction when running optimizing (as you’ve already proposed some time ago: e.g., here and here and as I discussed here). So just wanted to +1 the idea while you are discussing constraints. It’s mildly related to some work I try to do with SBC for approximate algorithms, but it is not particularly blocking me on any specific project (and when truly needed, one can pass a boolean is_optimizing as data, it’s just not very elegant).

Thanks once again for all the work going into the language!

I vote to move the meeting up a week. I’d like to discuss the future of Stan2TFP and other non-C++ backends and how we’d like them to live/coexist in the stanc3 project

Right. It’d be possible to code all of our constraints by hand using only the real type. So the question isn’t about adding expressiveness, but adding convenience. I like to ask myself if programs would be easier to write or easier to read with a feature. User-defined functions is a good example feature for comparison. They literally don’t add any expressiveness to the language (even mutual recursion can be unrolled into an iterative loop that gets inlined in place of any function call).

Right now, user-defined transforms get spread out over the parameters, transformed parameters (the actual transform) and model block (the Jacobian adjustment). So I’m wondering whether there’d be a way to encapsulate all of that in a way that’d be more understandable. For instance, compare this:

functions {
  vector my_simplex_constrain(vector x) { ... }
  real my_simplex_constrain_jacobian(vector x, vector y) { ... }
  vector my_simplex_unconstrain(vector y) { ... }
}
...
parameters {
  my_simplex[K] x;
}

vs. something like this with the same functions:

parameters {
  vector[K - 1] x_raw;
...
transformed parameters {
  vector[K] x = my_simplex_constrain(x_raw);
...
model {
  target += my_simplex_jacobian(x_raw, x);
}

There’s also the issue that this won’t work with initialization properly (where we use unconstrain to map user-supplied values to the unconstrained scale)—that’ll all have to be done by hand on the outside.

P.S. Rather than get dinged for more inappropriate behavior, I’ll leave it to you to decide where this kind of discussion goes on the forums. I was just trying to report the minutes of the meetings here, not open a larger discussion on language features. So feel free to move this or even delete it.

2 Likes

OK, let’s move the meeting up a week, as I’d like to be able to attend, too, and nobody else responded. So the next meeting will be:

  • 7 October 2021, 10 am NY Time (1 hr before the Stan meeting)

I’ll revise the top post to match.

2 Likes

I agree that convenience is important. The intro I gave was probably a distraction and I should have said something more like “Hey, while you are discussing constraints, could you also consider this? No pressure, but it would IMHO kind of help.” and just avoid the rest. Agree that meeting thread is usually not a good place for long discussions about design and those deserve their own topics. I think the meeting thread could be a good place for people watching from the sidelines to provide brief feedback/wishes/thanks/… as I don’t think those would work well elsewhere. But I think that the meeting organizers are free to setup a mechanism that works best for them for getting feedback from the community, so its mostly up to you.

Thanks again for running the meetings.

If anyone would like to get the Google invite for the meeting, please let me know. I’ll set up a standing invite for the second Tuesday of the month starting next month.

Please also let me know if there are agenda items for this meeting.

1 Like

Hey Bob, could you send me an invite?

Notes for Oct 2021 meeting

Design doc on deprecation

@WardBrian asked for volunteers to read his design doc on deprecation. If nobody expresses objections soon, we’ll go ahead and make this policy.

Type checking refactor

@WardBrian also asked for volunteers to review and think about a refactor of the type checker. The goal was to make it shorter and more readable. The PR reduces the size by 500 lines of code. The major changes are

  • the global symbol table is replaced with a functional map
  • the validation monad is removed
  • there are about 500 fewer lines of code

Hopefully, this will open the way for function overloading, tuple types, etc.

@WardBrian also volunteered to help update outstanding PRs for things like tuples and closures.

Function types in the langauge and the closure PR

@nhuurre has an outstanding PR for closures. I’d very much like to get this into the language. As it stands, it only implements closures, not lambdas. In order to have lambdas, we’ll need to introduce a syntax for functional types into the language. There’s a proposal in the closures design doc, but of course that’s not binding if there’s a better way to do it.

Apparently, functional types are already treated as first class objects in the type checker and the hang up is more on the parsing side. Closures are actually the more useful feature, but it’d be nice to get lambdas in that we can use in contexts where appropriate. I’m OK merging just the closures part of the closures design doc and not worrying about the lambdas yet.

TensorFlow back end

As things stand, there’s a lot of logic in stanc3 dedicated to generating TensorFlow back end code. It only supports a bare-bones set of a few handfuls of functions and is reportedly slower than the Stan math back end. The question came up as to whether we should continue to support it or move it out into its own module.

It’s not clear if anyone using it. This was mostly developed by @seantalts and @Adam_Haber, so it’d be interesting to hear from them on whether it’s still viable.

Even if it is being used, it was agreed that it’d be nice to move it out of the stanc3 repo so it’s not an ongoing drag on development. The consensus seemed to be that we try to break it off so that it could import stanc3 as a submodule for most of its functionality. Nobody thought it’d be worth fully factoring Stan math out of the parser, but it would be OK from the TensorFlow back end’s perspective with some extra stuff they didn’t need in the repo (there was general objection to splitting stanc3 into two repos for code generation and for parsing/type checking).

@Bob_Carpenter suggested all the cool kids were moving to JAX, so maybe we should target that instead of TensorFlow.

If we move things now, everything that works in 2.28 will continue to work going forward just by using the 2.28 release.

Transforms

We had a freewheeling discussion of variable transforms and how much users can be expected to know about underlying implementations. For example, should users be able to rely on <lower=0> constraints being implemented with log transforms or should that be considered an implementation detail? The conclusion seemed to be that it’s an implementation detail, but also critical for understanding how sampling was going to work. The Reference Manual does detail the transformations, but it’s not prescriptive in saying that they have to be of this form.

This is one of the motivations for user-defined constraints. For example, a user may prefer a soft-abs inverse transform to an exponential inverse transform for variables declared as <lower=0>.

We then realized that constraints and transforms are doing something different. It’s always been awkward for the doc and explaining Stan that constraints work to trigger transforms in the parameters block, but are just validation tests in other blocks. If we separate constraints and transforms, we could have constraints parameter variables act like other block variables. For instance, we might have

real<transform = softplus, lower = 0> a;
real<transform = exp, lower = 0> b;

This would open the gateway for us to have multiple forms of each transform for different computational purposes.

But this’d be a major breaking change in the language if <lower=0> doesn’t trigger a transform all by itself in a parameter’s declaration.

We decided that if users define potentially non-sensical transforms, one place to warn them would be with pedantic mode.

See you next meeting!

3 Likes

The November meeting will be Thursday 11 Nov, 2021. Let me know here or via email if you’re not on the list so far and would like a meeting invite. Thanks.

2 Likes

I sent out a standing Google Calendar invite with Zoom details. If you didn’t get an invite and would like to attend, please let me know and I’ll add you to the invite list.

There’s not going to be a language meeting today. Sorry about this—it’s my fault as I spaced after getting a booster shot yesterday. We’ll resume in January.

1 Like

The next language meeting is tomorrow, Thursday 12 January at 10 am NY time. It’s an open meeting and if you’d like to be on the Zoom invite list, let me know.

Here we are again. The next language meeting is this morning 10 am. Let me know if you’d like to be on the Google Calendar invite or if you’d like me to just send you a link to the Zoom for the meeting.

Summary of February 2022 Language Meeting

We spent this meeting talking with @mitzimorris and @spinkney about reporting project status. Sean’s updated the stan-dev org to use GitHub projects, which I understand are like the way Agile projects manage with index cards on a wall.

Sean thought we could do better in getting the word out about what’s going on in the project for a bunch of reasons: letting users know what’s coming, letting developers coordinating, informing the @SGB about what’s going on. He cited projects like Pyro’s use of Twitter, Medium blog posts, etc., to get the community excited.

We tried to work through what that would look like for a project our size and how we’d get the word out. Sean said starting with one feature is better than nothing, so I offered to write one feature up at the level he was asking for. Here goes.

Complex Number Support

The current status of Stan’s complex number support is that it works for complex scalars at the level of the language and math library. And at the level of arrays. We’ve added support for covariant typing, meaning that we can now assign int to real to complex, and that also works for arrays. We now need to add support for vectors and matrices, which requires work in the parser and code generator and also in the math library for polymorphic arithmetic (e.g., multiplying a real and complex matrix) and covariant typing. There is work to do on extending Eigen’s low-level BLAS level of functions to support. We could also use an actual complex-number based application for our user’s guide. After we get vectors and matrices, we want to add support for complex linear algebra (e.g., Schur decomposition and asymmetric eigendecomposition), and for fast Fourier transforms. Pretty much every single scalar and arithmetic funciton could be productively specialized for complex numbers. It’d be nice to think about the equivalent of var_matrix on the real side for the complex case. There is remaining work to do to generate actual complex numbers in the Python, R, and other interfaces rather than just returning real and complex components. There is a design document and issues and branches in progress in the language lib, the math library, and the interfaces.

Other language features

I also mentioned there are about 20 different features we could list this way that require work across the language, math, and interface repos. In no particular order:

  1. tuples
  2. closures
  3. lambdas
  4. 64-bit integers
  5. provide sample back end implementation in JAX
  6. ragged arrays
  7. sparse matrices
  8. regression syntax a la BRMS
  9. comprehensions a la Python (for efficient GPs and static var_matrix optimization)
  10. parallel loop constructs and more general map/reduce functions
  11. new simple types (sum-to-zero, probability, etc.)
  12. orthonormal matrix type for circular/hyperspherical/rotation stats (e.g., Stiefel and Grassmanian manifold types)
  13. vectorized truncation (this one’s relatively easy compared to the others)
  14. differential algebraic equation syntax and solver
  15. stacked/composed transforms, user-defined transforms
  16. user plug-ins at the C++ level
  17. user-defined gradients, including integration with ODE solvers
  18. integer and complex output types in language/interfaces
  19. expose transforms to users; allow Jacobian adjustment in transformed parameters
  20. more code transform and gen optimizations in compiler
9 Likes

Thank’s for putting the list together.

I’ve been working a lot on the complex container support in the language, so I’ve made a ‘project’ to track where things are at the moment. It would be worthwhile to create issues in stan-dev/math for the special functions (FFT, etc.) and function specialization we will need for complex matricies.

I’m curious how using a project will feel for this task, so it’s more of an experiment.

Here’s the link: https://github.com/orgs/stan-dev/projects/3

1 Like