Softmax(matrix)

#1

Is there a way to do softmax(matrix), not just softmax(vector)?

I am trying to compute the logit probabilities a hierarchical multinomial regression from a NxK matrix. Ideally, I would like to vectorize without looping thru N. But when I get to the softmax step, it would not take a matrix. Is there a way to handle this?

Thanks.

#2

The softmax function is not defined for matrices or row_vectors for that matter. You would have to do a loop, perhaps over the transpose of a NxK matrix.

#3

Hmm…Is there a way to use the multinomial distribution without the softmax command? I am computing the utilities for each of the alternatives. Then I just apply y[n]~multinomial(softmax(utilities)). Any suggestion would be appreciated.

#4

There is categorical_logit but no multinomial_logit. If you look at the code for the former, it would be easy to adapt to the multinomial case.

#5

Can categorical_logit give vectorized samples in the output? It is not clear from the manual.

Also, how would I adapt it to make multinomial_logit? Can you provide the source code, if I want to create it myself?

Thanks.

#6

I am not sure what you mean, but I assume you are asking whether categorical_logit_lpmf can input an integer array of outcomes over the observations, in which case the answer is yes.

#7

Sorry. This is what I meant to ask:

y~normal(mu, sigma);
// y can be be real or a vector or a row_vector; mu and sigma are of the same type with the same dimensions.

Is the following allowed?

y~categorical_logit(x);
// y is a vector with length N; x is a NxK matrix, where K is the number of alternatives

#8

y can be an integer array (not a vector) of size N but x must be a simplex. There is a PR to do something closer to what you are looking for:

#9

I think he means something like:

int[,] y = softmax(matrix);

Bottleneck is (I think) the toggle of max. calculation and arithmetic calculation in a loop of log_sum_exp's. That’s why I was hoping of GPU support of later then we could speed wise advance with softmax too.