Do you have any RStan in Regression and Other Stories?
Only one little bit (the golf model). But we do have examples of rstanarm where we pull out posterior draws. At least, I think we have some such examples.
rstanarm will probably stay the same. If you do the golf thing using as.matrix() and / or as.data.frame() that will still return something 4000 by number of parameters.
1 Like
So as.matrix will put iterations as first index, but extract etc. will put iterations as last index?
In the future, I don’t think there will be an extract method, much less a stand-alone function. People can just do fit$theta instead of extract(fit, pars = "theta"). For the book, something like as.data.frame(fit)$theta would work if theta is a scalar or otherwise
mat <- as.matrix(fit)
theta <- mat[,startsWith(colnames(mat), prefix = "theta")]
Sorry for being super late… .but would threaded sampling of chains fit in here (with the goal to pool adaptation info) or need consideration? While less of a priority, as discussed, speculative HMC would be nice to keep around as potential feature eventually.
Another point:
The whole threading topic needs consideration in terms of how we manage their session. Right now threads are handled at the Stan Math level through STAN_NUM_THREADS - but I don’t think that should stay like this; or at least it should be thought through. Should we go all in with the TBB, then we can create task_arena which we assign a maximal core count. Each task_arena then executes some bigger chunk of work with the given resource limitation. Thus, this requires some session management which I would expect to be ideally part of stan services, I suppose. In short, this bit needs design.
This leads me to another point which I may have overlooked, but session management for the life-time of a chain could be worth considering. For example, the algebra solver wanted to save initials, the MPI map_rect needs to transfer data which is static or static data distribution in general. Right now, the MPI code and the GPU code have found their solutions around that, but it’s a common theme. This probably crosses the Stan-math / stan services boundary.
If some of the points need clarification, let me know. If this is out of scope for this, just dump it.
How much of this disucssion is about RStan and how much is more general? Is any of the extract discussion relevant for CmdStan? Is the JSON format going to be the same for R and Python and CmdStan? The default readers in R and Python expect different coding for NaN and inf.
This would be nice to build in. To make it useful, we’d need definitions for all the functions that current assume rectangular input like R-hat, n_eff, mean, etc.
I agree. Both of these cross the math library and the inference algorithms in Stan.
fit$theta is fine with me. I somehow thought that for these R objects, it was considered better to have another level of abstraction. Will fit$foo create problems if there are parameter names “foo” that overlap with existing objects?
Also, don’t forget:
• I’d like to be able to extract one draw. For example, extract_one_draw(fit, 82) would return a list with three elements corresponding to the 82nd simulation draw, thus a scalar for alpha, a vector of length 10 for beta, and a 2 x 3 x 4 array for theta.
• I’d like to be able to access summaries. For example, foo(fit, median) would return a list with three elements corresponding to the median of alpha, the pointwise median of beta (i.e., a vector of length 10), and the pointwise median of theta (i.e., a 2 x 3 x 4 array).It is not at all clear to me how much of this should be Rstan and how much more general.
I do think that the extract_one_draw feature and the extract summary feature will be generally important for Bayesian workflow.
Traditional statistical workflow is to take the data and dump out parameter estimates and standard errors. I think it’s an important part of Bayesian workflow, or I could say “probabilistic programming workflow” to be able to grab fitted parameters and be able to run with them.
@andrewgelman: extract_one_draw is not on the interface roadmap.
@seantalts: How are you handling suggestions like this before we have to make an up/down vote on the whole thing? If the extract-one-draw feature doesn’t get into the ratified interface roadmap, what does that do to its status?
What is the scope of the vote here? I’m just not sure what I’ll be voting on in a couple of ways. If something is not part of the roadmap, does that mean it won’t be considered?
There are headings for news like “New compiler integration” that don’t seem like they’re part of the roadmap. What’s the status of future roadmap topics or order of operations? What’s the status of “miscellaneous ideas”, as they don’t seem concrete enough to be used to make decisions going forward, nor do the “Stan language features”. Is a vote to ratify considered a vote for “Getting rid of arrays of real numbers from the language entirely to avoid confusion with vectors.” The latter’s listed under language features. Is blockless Stan definitely on the roadmap with a yes vote?
It just seems odd to me that we have things discussed down to the name of an argument in one place and something as vague as “named blocks or annotations”.
What does the injunction to make CmdStan for users mean? Does that mean it would be acceptable to fork a vesion of CmdStan to be used by interfaces?
It gets called out in the roadmap, but is there a decision on “some of the language interfaces have started to depend on source code within the Stan project rather than using only the exposed APIs,” Will interfaces calling things beyond the services to implement things one-off in their own interface still going to be OK in the future?
How much of this stanfit discussion pertains to the Python interfaces? It doesn’t seem at all relevant for CmdStan, which currently stores a chain as a file, so the indexing is chain -> iteration -> parameter -> value in CmdStan.
We don’t have anyone actively maintaining any of our modules as their primary job. What exactly are you looking for here?
It looks like ragged output is not on the roadmap. Does that mean every chain will continue to have to be the same size? What happens going forward if someone wants to add a feature like this that’s not on the roadmap?
2 Likes
One more thing. What’s the plan for changing user-facing interfaces? For instance, what is going to happen to the stan() command in R? It now uses (iter = 1000, warmup = 500) which will use 500 sampling and 500 warmup iterations. The roadmap calls for these to be (num_main_iters = 500, num_warmup_iters = 500).
Is there going to be a new function stan3() or something? Will there be an RStan release where all of iter, warmup, num_main_iters, num_warmup_iters are supported? If so, what happens to defaults in the case some mixed subset is provided?
Will there still be arguments buried in control = list(key = val, ...) format?
Also, thin is still scheduled for removal despite pleas to add it. I take it that means @seantalts wasn’t swayed by the discussion and the vote will include getting rid of thin.
I find this statement very confusing. Are StanAnalyze and StanFit new C++ libraries/programs? Or are they something else? Who is writing/maintaining them? Could we rename “Interface Package Architecture” to “New C++ Interfaces”. This would be clearer, I think.
1 Like
Hi–I think extract_one_draw() would be very helpful for probabilistic programming workflow. I guess it did not come up before because I associate Stan roadmaps with big-ticket items such as MPI, GPU, Laplace, blockless, ragged arrays, etc. I didn’t realize that little features such as extract_one_draw() would go in also.
A
I’ll add a section dealing with parallelism if @wds15 is committed to working on it! I think the minimum I might say is that we want to include or build software with an interface similar to the TBB and spec out how various parts of Stan can use parallel resources if available. I think we’re just at the design stage of this now.
You can think of this roadmap as covering just the interfaces and just the parts that achieved something like “rough consensus” and had some commitment to working on it, or (in my judgment) seemed like something we could advertise and possibly attract contributors to own.
To that end, anything not covered is not barred or disallowed and I don’t think there’s any reason to have a policy that e.g. @bgoodri couldn’t implement specific RStan-only features for @andrewgelman if they were both amenable.
I’ll add a note to clarify that - briefly, the services and the model class are the supported endpoints for interfaces. Anything else - file locations, names, etc can change without warning and should not be relied on. This is especially salient as we move to the new compiler hopefully next release (at least for CmdStan).
After posting this, Rok took a big sweep over CmdStan - closing outdated issues, checking that bugs were fixed, etc. I’m hoping we get someone who takes maintenance of CmdStan as a major priority. I think my hands are tied here for now, but the SGB is working on formalizing rights and responsibilities of the Tech Lead role.
I’m just now getting to all of the discussion that occurred over the weekend.
Re: backwards compatibility - this is all assuming a major version bump to RStan/PyStan/CmdStan 3.
I hope this addresses all of your questions; please let me know if you still have questions.
They are new non-C++ packages. I don’t think anyone agreed to work on them despite the design achieving rough consensus, so maybe I should drop them from the roadmap. I think possibly @jonah and @paul.buerkner were proponents of this?
They don’t have to :)
Just to clarify: by “go in,” I meant, “go in the roadmap.” I do think they should go into Stan.
There’s another issue that comes up a lot, which is that there are features that we want in all the interfaces, including R-hat, n_eff, quantiles, extract_one_draw(), etc. Maybe this means we need a “universal interface” or “basic interface” which includes all the features that are common to all interfaces? I have no idea.
Just quick clarification question:
Is extract_one_draw a function in R or Python that iterates through already sampled draws?
Or does it call Stan to sample a new draw? (I’m thinking of functionality like generators in Python -> next(gen) would return a new item from the gen generator, which could be either terminate at some point or give new items forever)
The idea behind separating StanFit from StanAnalyze is the users may be interested just in our convergence diagnostics and stuff (i.e. in the Stan services) but are not actually using Stan to fit their models. On the R side, we have also discussed a package that makes it easier to work with MCMC samples and transform them to various formats relevant in different applications. This package may or may not be identical to StanAnalyze. In any case, I think we should separate the Stan fitting specific stuff from the general diagnostics we have developed that also work for samples generated with software other than Stan.
4 Likes
Yes, although I am generally against having stand-alone functions. In the future, you should be able to do fit[42,] to get all the parameters on the 42nd main iteration in R.
I’m on board with this but one doubt occurs to me. What if someone gives a parameter in their Stan program the same name as one of names of the components in the stanfit object? That is, suppose the future stanfit objects have a component called foo and someone names a parameter foo. What would fit$foo do in that case?