I am working with @hyunji.moon, @Dashadower and Teemu Säilynoja (AFAIK not on Discourse) on a revamp of the API of the SBC package to bring the power of SBC to the masses. I would be super happy to get help with naming of the main components of the API. The question is:
What to call an object that describes specific model + algorithm + algorithm parameters except for the observed data
In other words, this object fully describes a function that takes in data and produces a set of posterior samples (with fixed size and number of parameters). E.g. for sampling with rstan it is a stanmodel object + list of additional parameters to sampling, for ADVI with cmdstanr it is a CmdStanModel object + list of additional parameters to variational, for sampling with brms it is the set of arguments to brm except for algorithm, …
I currently call it “backend” but I am not very happy about it - “backend” IMHO implies something reusable and general, while this object is typically very specific. I don’t like calling it “model” because it is model + additional information (and the same Stan model could give rise to many different “backends”).
Other candidate names we discussed are: “target” (as in “target of our investigation”), “engine”, “posterior simulator”, “fitter”.
“target” is too confusing, since we have target densities (not to mention that they get manipulated via target +=).
I like “engine” and “(posterior) simulator”. The first phrase that I came up with on my own is “routine” or “model routine”. Not sure whether or not that’s any better.
I would like to add that the combination of ‘fit’ object and algorithm is capable of constructing data and parameter joint space.
“Posterior simulator” seems to be what the combination of datasets and backed from your example code below is doing, under the hood of compute_results function.
results <- compute_results(datasets, backend)
What I am saying is the backend components you described include enough information to simulate prior samples and also data (there might be controversy on simulating data, but likelihood function is ingrained here in the stan code so the information itself is included although manual refactoring such as writing generator function is needed to actually get simulated data).
My image of the backend is “abstract joint (distribution) simulator” which could transform into a posterior simulator upon receiving dataset. Each dataset in your datasets from your code once combined with the backend makes different posterior simulator.
May I ask why you exclude ‘algorithm’? Was your intention just to use the default “sampling” (mcmc) instead of other options (“meanfield”, or “fullrank”)?
I think that’s incorrect - the backend never contains information on how to simulate non-modelled variables (e.g. covariate values in linear models), but a way to simulate/choose those is necessary to run SBC.
It indeed should be possible to use brms + variational inference with SBC. However, I currently by design try to keep HMC and ADVI as separate backends (they produce different values, have different diagnostics to check, …). So at some point there would probably be a brms_ADVI_SBC_backend or something like this. Or we’ll realize that maybe having the same backend be someimes capable of expressing very different algorithms actually isn’t a problem. I admit I am not completely sure about this choice.
Because a) not all “backends” I hope to eventually support produce executables (e.g. inlabru, MCMCglmm) and b) a single Stan executable can still correspond to multiple backends that differ in the algorithm used (HMC/ADVI + variants) or sampler options (e.g. adapt_delta).
Just to be clear, the object I am trying to name does not produce data - it takes a dataset (generated previously) and produces posterior samples. In fact, there is a generator object in the API - here’s a diagram that hopefully makes the outline a bit more clear:
How about optimizer? Sampler is another option but considering the future addition of algorithms such as variational inference or closed-form inference engines without stochastic components, the former seems to be more inclusive.
If I may add some notes on “can sampling be viewed as optimization?”, I wondered this question myself and have logged the quest here (Bayesian calibration (1) Sec.1.2) with references (the answer is yes!).
@jhuggins introduced me to these sampling ~ optimization concepts (he also maintains variational inference package, viabel which could play the role of one module for backend from Martin’s diagram. I wonder what his opinion on whether backend could be viewed as optimizer is. May I ask :)?
So it feels like there is some support for “engine”. I also know Shi actually liked “backend”, no big support for any of the other options, some having good reasons to not be chosen. So it is a bit split between “engine” and “backend”. I admit that after having “backend” in the code for a while, I feel less averse to it. Will wait a couple days if anyone has any other impressions/arguments and make a final decision.
I think “optimizer” is a bit dangerous as I would gues most people have a relatively narrow conception of what “optmizer” means and would expect it to no cover MCMC or even ADVI. I certainly wasn’t aware of any connection between optimizing and sampling.
Certainly - would love people to plug in their packages. The necessary wrapper code is quite small and a tutorial on how to this will definitely come with the package in some near future.
Since the object in question is composed of a model and an algorithm+parameters (= engine) i suggest to use a term that describes its components: “model engine”.
Maybe if you don’t have a word for the whole, just combining the words of the parts is a generally sensible way of doing it?
@andrewgelman and @avehtari introduced HMC and variational inference as “1.5. Generic computation algorithms” in their paper on great statistical ideas of the past 50 years. Might the word “generic” explain backend’s superb computing (optimizing) ability to sample from joint distribution without explicit “information on how to simulate non-modelled variables”?
I wish to carefully ask for feedback on using the word estimator instead of backend. For two reasons.
First, while communicating modelers outside Stan community, I felt estimator is easier to communicate (which I am open to any objection!). The context was this paper which uses Stan’s HMC estimator but generator is defined using system dynamics simulation language in an implicit manner. Authors of the paper developed a package which translates the simulation language to differential equation functions in stan (tagging @charlesm93 who might be interested and can give good advice). The paper lists two justification for using Stan: faster estimation and visual diagnostics. SBC, however, is not included in their workflow which I am in the process of proposing. For instance, diagnostics section from the notebook in appendix is a good place to start.
Second, definition of suffix -or is “one that does a (specified) thing”. generator and estimator pair exists in both mathematical and program version. Data generating process for generator, computational algorithm for estimator will be the mathematical version while their implementation such as generator d For me, backend is program version of estimator and hence does not correspond well with generator compared to estimator.
In summary, I wish to get feedback on
i) the following frame, with example in the table
ii) using generator and estimator in a narrow sense (only for program)
In general, mathematical version of generator would be convolution of prior and likelihood while estimator may be the form of \begin{aligned} \hat{\nu}(\theta) &=\arg \min _{\nu} L(\nu ; \pi, x) \\ &=\frac{\exp \{-l(\theta, x)\} \pi(\theta)}{\int \exp \{-l(\theta, x)\} \pi(\mathrm{d} \theta)} \end{aligned} suggested in this paper which introduce variational formulation of Bayesian update.
Should we be thinking one level more concrete than the above?
Is this a model or is it a specific instantiation of a model, such as a Stan program or a BUGS program? I don’t understand how you can say anything about a model without picking a specific instantiation.
Is a centered and non-centered parameterization considered the same model? They expose different variables to sample, but the aggregate of all transformed parameters has the same distribution.
Is it an algorithm or a specific algorithm implementation? There are lots of different implementations of HMC that have different behavior w.r.t. things like arithmetic stability (e.g., how do you multiply inverse mass matrix by a vector can be done more or less stably?).
I find it can help to work backward. Is the net result something you want to view as a function from data to something that learns a sequence of draws? I’m very prosaic and would name that something like “data2draws”.
