Old NUTS?

Hi!

It’s great to see that the bug in NUTS has been found so quickly.

I was wondering how hard it to would be to make the old sampler available as option to the users? I mean the NUTS v2.9.0 was apparently a very rock solid machinery.

So would it be very painful to keep that old NUTS around or is this a sensible thought?

Or if this is very hard, then maybe we should consider to change the design such that keeping old samplers around is easier in the future. I must say that I don’t know the inner workings of Stan around this and as such this is a very naive proposal possibly, but I thought its worthwhile to discuss.

Best,
Sebastian

It was not rock solid – it had a one out of the two bugs that the current sampler had. It also doesn’t have all of the same diagnostics. Really there’s no reason not to use the new sampler which uniformly improves over the older version.

yeah, I guess you are right. Also, given the dedication you put into this and the speed at which these bugs get fixed there is probably nothing better than just going with the latest.

I just thought it would be worthwhile to hear what others think.

Sebastian

You should fix the old NUTS and do comparisons. That’s
how NIPS papers are made. :-)

  • Bob

The old implementation is still kept around, just not exposed
to users. Now we have to update that, too. It’s why keeping
multiple things around is nontrivial amounts of work.

  • Bob

Well, the oldest version that I implemented. Matt’s original code wouldn’t be compatible with the way the code is currently organized.

It’s still code. It’s not that hard. Especially after the refactor gets in, it’ll be easy enough to reproduce.

Hi!

I would consider it as a plus if Stan could be easily switched to use an old nuts variant. The main use case would be much easier ways to roll back in a production system where one often cannot easily upgrade or downgrade some program due to other constraints.

However, I do not think that the old nuts implementation should be fixed if that requires extra backporting work. The old nuts should only get fixes if some shared code blocks got fixes and as such the fix for the old version is for free.

In short Stans ressources are limited, but a switch as from 2.9 to 2.10 was quite a big step and being able to swap the old stuff in would have been nice. However, this all must not add too much to our workload. Managing software versions can be done in production systems as well.

Sebastian

One reason I’m reluctant to have too many options for users
through the interfaces is

  • it clutters our doc and makes it harder to find the thing
    that you we recommend

  • we have to support them all going forward with doc and
    upgrades; it’s the usual quadratic growth problem of new
    features:

  • users are tempted to try them all and model comparison and
    model fit comparison is hard; this may be more a problem with
    computer scientists using machine learning packages than statisticians
    using MCMC

In terms of what the current status is:

  • the old old NUTS that Matt initial wrote doesn’t match our
    current interface standard, so that’d be a lot of work to resurrect

  • the new old NUTS that Michael wrote also had an issue with the
    tree building—don’t know how easy that would be to fix, but it’s
    non-trivial.

  • Bob

I meant to cite my post on Andrew’s blog for the reason
I’m reluctant to add lots of new features:

http://andrewgelman.com/2013/11/04/joel-on-shlemiel-the-painter-unknown-unknowns/

  • Bob

Ok, there are more gotchas which were not in my mind. Sounds like it was a good occasion to raise the topic, but we should stay with what we are currently doing - fix sampler bugs ASAP and say that the “NUTS” sampler is not a fixed target, but can change between releases (change for the better is the assumption here, of course).

Fine with me.

Sebastian

We’re not trying to keep things fixed between releases
other than for backward compatibility. Just keeping
the old NUTS fixed wouldn’t be enough to ensure replicability
if anything else changes (e.g., RNGs, which come in from
outside).

The bug’s fixed and trickling through the interfaces.
RStan’s already updated through CRAN and CmdStan has
been released. I think Allen just finished PyStan, but
don’t even know how to check if the new release is available.
The other interfaces just need the latest CmdStan, so they’rea
all up to date.

  • Bob