Jacobian of softmax tansformation of a (n-1 degrees of freedom) unbounded parameter

Modeling
28

Let’s be careful to separate out to similar but distinct concepts:

Transformations map points in one space to points in another, in this case an unconstrained space to a constrained space. For example in Stan we use the stick breaking transformation to map a K complex to K - 1 unconstrained real variables.

A pushforward distribution quantifies how a transformation warps a probability distribution over the input space (or representations of a probability distribution like samples or a probability density function) to give a corresponding probability distribution over the output space. For example the transformation

y_{k} = \frac{ x_{k} }{ \sum_{k' = 1}^{K} x_{k'} }

pushes forward the density function over the input space

\pi(x_{1}, \ldots, x_{K}) = \prod_{k = 1}^{K} \text{gamma}(x_{k}; 1, 1)

to a uniform probability density function over the output space,

\pi(y_{1}, \ldots, y_{K}) \propto \text{constant}.

Critically transformations can be defined in the context of input and output spaces alone. In other words they can be applied to any Stan program. A pushfoward distribution, however, has to be defined in the context of an input space, an output space, and a probability distribution on the input space to be well defined.

This is all to say that I have seen the y_{k} = x_{k}/ \sum_{k' = 1}^{K} x_{k'} transformation used in statistical applications before, but only in the particular instance of pushing forward a product of gammas density function into a Dirichlet density function. In other words it’s used just as a way of implementing a Dirichlet density function over a simplex, but not more general models over simplifies. That said it fundamentally suffers from the identifiability issue discussed above so even though it’s mathematically appropriate it’s terrible in practice without some additional heuristics to temper the identifiability.

If we’re talking about general statistics or probabilistic programming applications, however, then I haven’t seen y_{k} = x_{k}/ \sum_{k' = 1}^{K} x_{k'} used to implement simplices. All of the applications I have encountered have used the stick breaking construction or constructions equivalent to it (such as the hypersphere transformation I introduced way back in the day, [1010.3436] Cruising The Simplex: Hamiltonian Monte Carlo and the Dirichlet Distribution).

1 Like