Addressing Stan speed claims in general

[Bob_Carpenter]

OK, I’m replying to everyone at once, so bear with me.

@mike-lawrence

I’m happy to help update examples if someone more knowledgeable with the examples repo points me to candidates that are likely inefficient.

Anything in the user’s guide or the example-models repo. That includes all the BUGS examples, all the Gelman and Hill examples, etc.

@imadmali

It was noted that this is often due to badly written models.

If this is true then it’s news to me, which imo does make this issue a PR/documentation problem. (Although I’m not the typical Stan user.)

Did you not realize that how the model is written makes a difference? How about how it’s parameterized? Did you try to find ways to make models go faster and not find the chapters on efficiency and reparameterization in the user’s guide?

Does anyone have any ideas on how we can get people to read something like an overview of Stan before writing. I find it hard because we have too much doc, not too little. For example, new dev doc is out of control.

@spinkney

If you take the time to read through the docs there’s a bunch of great material to help with optimization. This is awesome for those of us who take the time to read these. Many people don’t take the time and I don’t see how that’s going to change.

Glad someone saw them. But this also brings up the big problem we have—too much doc for anyone to get through. And it’s fragmented by language (R, Python, etc.) and by application area.

@imadmali

afaik linear regression with Spark on terabytes of data will outperform linear regression with Stan.

Yup. That’s a battle we’re not going to win. Nor are going to compete with TensorFlow or PyTorch on fitting neural network models. Or against custom genomics models for gene alignment.

@spinkney

It initially appears “hard” to learn and co-workers seem to gravitate to TensorFlow because they’re familiar with it. But the wealth of info, models and info from the forums makes this relatively “easy” to learn.

I think it depends on where you come from. If you’ve been using NumPy for years in Python, TensorFlow’s a lot more familiar than Stan. Whereas if you’ve been using BUGS or JAGS, Stan will be more familiar. If you come from C++/Java or other typed imperative languages, Stan will seem familiar, whereas those coming from dynamically typed languages like R or Python have more trouble.

@spinkney

Bijectors/automatic reparmeterization

Our transforms are “bijectors”. We just need to allow them to compose so we can non-center positive-constrained and probability-constrained hierarchical models.

Automatic reparameterization is very hard. It’s what the ML people call “normalizing flows”. it depends on aspects of the data and prior, not just the model structure.

People are working on automatic marginalization and automatic vectorization, but both are hard to do in general.

@maxbiostat

I don’t think anybody who knows a thing or two about computational statistics is out there saying “Stan sucks for [insert statistical task here], the guys at [insert dank new software package here] are really outshining them”.

Lots of people say things like that, and many of them are well credentialed in computational stats.

@maxbiostat

The way I, as a practicing statistician see it, Stan is general “statistical modelling”* language, which works really well for the majority of the models I care about, and I have a wide range of interests.

That was the goal in building it in the first place. Of course, we want it to be fast, too, which is part of working really well. But it’s not the only part.

@maxbiostat

One example would be a model which has discrete latent variables which can be written in marginalised form in Stan and “in full” in Nimble.

Monnahan, Thorson, and Brants did that for some ecology models w.r.t. JAGS. It turns out to be faster to marginalize in JAGS, too. But the real problem is that if you don’t marginalize, it’s very hard to fit someting like an HMM by sampling latent states.

@bbbales2

• How fast is Stan vs. anything else that does NUTS on hierarchical models/things in our wheelhouse
• What is the advantage of doing multi-threaded stuff in Stan
• What is the advantage of doing GPU stuff in Stan

Excellent questions and I’d add MPI to that list of parallelizations.

But then you have to break all this down. What about ODE models? Do we compare against dedicated packages like NONMEM and Monolix?

I think the SEO will come for free if we put a page up with good content. We have a lot of internet juice from the user base.

@breckbaldwin

We could chase down and ‘refute’ speed test claims.

That’s what @andrewgelman was suggesting to me.

@breckbaldwin

We should pull the poorly coded example models from documentation and the web although it may be too late.

I’m all in favor. Like a lot of these things, like publishing our own speed tess, we just need someone to do it. To do speed tests, we need to be able to do accuracy tests if they’re going to be anything other than log density and gradient speed evaluation, but even then we need arithmetic accuracy for a fair measurement (e.g., 32-bit vs. 64-bit).

I also hear the countersuggestion regularly that we just leave all the badly coded models in the example-models repo because they’re good for testing.

@breckbaldwin

Compiler improvements with Stan 3 may be a big help here since optimization can be done for poorly coded models.

I haven’t seen any activity on this in the last few months. @matthijs and @seantalts were leading the efforts around this.

@breckbaldwin

I suggest we consider hiring someone to guide this process. The SGB has resources, e.g. $, and the ability to seek out and engage a PR (Public Relations) professional.

What would such a person actually do? Are you thinking hiring a consultant about what to do or hiring someone to do the work? And would this also involve marketing, or just the PR side of things?