Measuring and comparing computational performance in Stan with different compilation alternatives. Using reduce_sum does not bring any advantage?

General
18

Okay so I finally had some time for doing this.

I have tested 13 different STAN implementations of the same things (some of them I knew were gonna be inefficient based on our discussion but would like to know how much).

I have tested multithreading using openmpi vs nonopenmpi and 1 thread vs 10 threads. Also the comparison includes cpu vs gpu.

Surprisingly (and I guess it is due to memory transfer) the rank of all the tests I did over the 13 models is as follows:

  1. Pass W and X as a shared argument to reduce_sum, and slice over Y; then use Categorical_GLM function indexing columns of matrix X and then transposing running on CPU, 10 threads and openmpi

  2. Same as above but indexing rows of X directly.

3,4) Same as 1 and 2 but without using openmpi (the difference of using and not using openmpi was not very high)

  1. Running a single thread (i.e no openmpi and no stan threads) and using GPU computation over categorical GLM directly (as suggested by you).

  2. Same as 1,2 but computation is run in GPU, i.e each thread from the reduce_sum evaluates the Categorical GLM on a GPU.

The dataset had C = 10 classes and 10 features, and a total of 55K datapoints was used to evaluate the likelihood. I used a 2080RTX TI Nvidia GPU. The times where computed by running the program 5 times and then average results

Not sure if it would be interesting to have all the code and the excell files with the runtimes in github or in a place where stan users could check it?