Speeding up hierarchical models

[stevebronder]

@bbbales2

And this corresponds to peeling off a term: αi[n],j[n]\alpha_{i[n], j[n]} . In this case, alpha is a matrix and has two indices, so there are two m terms.

vector hierarchy(int[,] idxs,
               vector z, matrix alpha, int m1, int m2,
               args...) {
  return alpha[idxs[m]] + hierarchy(idxs, args...);
}

I think you are missing m1, m2 here

@bbbales2

We might not be able to get away with using floats in our calculations, but I don’t see why we couldn’t get away with making all our covariates (X and all the zs) floats. This would make our matrix-vector products twice as fast, because we are reading half the memory, and also lower the memory needs for the hierarchical slopes.

In the proposal for static matrices one of the benefits I footnote is that making vari_type<T> also allows for vari_type<float>.

@bbbales2

We could also make integers in the data block shorter versions if the limits are statically defined. Probably most hierarchical models we will work with in the short term have less than 65536 groups. In this case, all the group indices could be uint16s, which would cut the memory requirements for reading the indices by four.

I think we need to start allowing for attributes on types in the stan language. For instance with sparse, complex, and static on the horizon (however distant) the combinations of explicit matrix types would explode. If we had attributes on our base types we could have

static short matrix[N, M] A;
short int idx[M, N];
static long matrix[N, M] B;

short would be float, long is long double, and short for int can be uint16.

@bbbales2

So when we read XX (which is expensive), we currently also write an XX (which is also expensive) and then during the backwards pass we use that alternative XX . We can save the write if we recognize somehow that XX isn’t going out of scope. The same thing goes for the indices in the hierarchy portion of the model (though these indices wouldn’t exist in the base for-loop autodiff version).

I wonder if there would be some dirty trick we could do with like a second stack that holds just data. Then the first time that data is allocated we keep it on some stack and reference it then on.

@wds15

Isn‘t this where perfect forwarding aka universal references would do the trick? Ping @Stevo15025

Possibly, it would allow for b to be moved into the struct that holds the chain method. So it would be allocated once and forwarded along until it came to it’s final resting place in the struct that holds the chain method. That’s useful for arrays