Great questions!

I’ll start with what we currently work with:

- in sampling,
`lp__`

is the log joint probability distribution evaluated at the unconstrained parameter values including the Jacobian term and up to an additive constant (proportionality constant on the log scale).
- in optimization,
`lp__`

is the value of the objective function, which is the log joint probability distribution evaluated at the unconstrained parameter values up to an additive constant. There is no Jacobian term. On any model with a non-trivial transform, if you took the same unconstrained parameters, you would get different values under sampling and optimization.
- in ADVI,
`lp__`

is always 0. It’s unused and in the inference portion of the algorithm, it would probably make sense we to replace that with the elbo or something else. Post-inference algorithm, ADVI also draws Monte Carlo samples from the variational solution. I’m pretty sure the `lp__`

value isn’t computed here.

I’m thinking ahead. We’ve already seen it with ADVI, but there may be new algorithms in the future that don’t report `lp__`

. The all have to report something, but they should report what they need instead of trying to conform to how sampling deals with it. And… we’ve overloaded the meaning of `lp__`

across sampling and optimization. (I’m not sure what `fixed_param`

returns for the `lp__`

value. It could report a number of things including 0.) Also, with Michael’s improvements to the algorithms, there might be additional things that are reported. Perhaps not at the sampling algorithm level, but definite within algorithms.

Our current design is hindering us from moving forward quickly with updates to sampling algorithms. We’ve had issues with being able to move quickly here because we’re putting these things together: `lp__`

, sampler parameters (`tree_depth__`

, etc) and the actual values. If we had those three things separately, it would be easier to report additional fields (like `energy__`

or something new!) without having it affect the other two things which really shouldn’t change at all.