Sorry I haven’t been too clear with the goals. I’ll try to outline some of the problems that I see and that this proposal addresses.
Problem: it’s currently very hard to separate the sampler parameters, sample parameters, and the constrained parameters
- The use case is in showing sampler diagnostics and showing estimates of parameters (like in ShinyStan)
- This is currently being done by parsing the names of the output, searching for trailing double underscore, and asserting that those are not parameters
- Note: the goal is not to intentionally separate them out to ignore the sampler parameters or the sample parameters from the draws. It’s so we can manipulate them easily; the sampler and sample parameters, for a given Stan version, will be the same across Stan programs whereas the number of unconstrained parameters are not.
Problem: to output unconstrained parameters, it also outputs the sampler parameters and sample parameters
- Since everything is output at once, to get the “diagnostic” output, we duplicate a lot of output
Problem: there is no mechanism to output the constrained parameters separate from transformed parameters or generated quantities or unconstrained parameters
Problem: it’s hard, in implementation, to filter parameters
- RStan and PyStan both have the ability to filter parameters. This would make the logic a lot easier.
- Allen: the system bus makes this very inefficient.
Problem (design): the algorithm code currently knows about how to output constrained parameters by taking the unconstrained parameters and pushing it through the transforms
- The sampler should just operate on the unconstrained parameters and just use a templated model template that exposes a
log_prob() function. Instead, the sampler itself calls the transform code taking unconstrained parameters to constrained parameters making it a little more difficult using the algorithm code with another C++ model template not generated by Stan.
- All that logic would just be moved from the sampler code to the services layer, which conceptually is the right place for these things to be happening.
Problem (design): changing sampler and sampler parameters change all of the output and there’s no decoupling between that and the unconstrained, constrained, transformed parameters and the generated quantities
- The current output is just one
std::vector<double> per iteration. Adding a sampler parameter actually affects everything that uses the output. By splitting the output into separate chunks, we can independently update parts. Ideally, the parts that handle the unconstrained parameters, constrained parameters, transformed parameters, and generated quantities don’t have to change often.
Problem: we promote ints to doubles for the sampler parameters (and also generated quantities)
- We can fix this for the sampler parameters. Can’t do it easily for generated quantities.
Regarding thinning: I’m trying to think of @bgoodri’s use case with the accumulators. I think the cleanest thing is to always pass the sampler parameters, sample parameters, unconstrained parameters, constrained parameters at each iteration and allow the writer handle thinning however it wants. For the use case where it goes into memory and is thinned, the writer just needs to keep the thinned samples and do nothing with the rest. Now, if there’s an accumulator for means / variances / whatever, I don’t know if that should work on every iteration or just the thinned iterations. If that were to move back into the service method, which is reasonable, do we only pass back the values every time it’s supposed to be saved? If so, there’s really no way to have an accumulator for every iteration.
Regarding a system bus: I’m not opposed to that as the underlying implementation, but it doesn’t help solve a lot of these problems. There’s just no efficient way to selectively save parameter. In order to do that, you need context to figure out what the message means, which means inspecting the message. Then doing something to break apart the message, then filtering, then reconstructing it. If you want that to be the implementation for httpStan, I’m happy to help.
Did that help at all? If there’s a better way to communicate this, please let me know what would help (maybe an example) and I’ll write it in that form.