Hello all. After implementing the within-chain parallelization with reduce_sum in cmdstanpy (with your help), I reached a dead end. My model is pretty complicated (attached) and I fit a lot of data, and even if I run it on many cores (now 25) it takes unreasonably long time to fit (it’s stuck in wa…

That’s huge! Did you measure your speed ups on smaller problem sizes first? Do you have a sense for how long this will take with a single core?

Yes it is :) I was building it during the last few months. I was running it on small N of 8-10 on a similar number of cores and it finished within 20 hours. N=20 on 20 cores scaled up to 40 hours. And these are just small portions of my data. Single core processing of N=15 took ~70 hours for the fas…

Usually vectorization is super important. If that does not help, then things are odd to me. Anyway, you should try to slice parameters used per group; so if your model is a hierachical one, then slice all the parameters needed per group parameters in the shared arguments list is very expensive, s…

[image] wds15: slice parameters used per group; so if your model is a hierachical one, then slice all the parameters needed per group You mean I should pass all the arrays I want to slice before I pass grainsize in the partial_sum?

[image] wds15: Other than that make sure the model runs efficient without reduce_sum first, then go to reduce_sum… I hope that is what happened in your case. Based on the tests I run, the serial version would take more than a month for the full dataset. I am not sure what counts as efficient …

[image] wds15: Usually vectorization is super important. If that does not help, then things are odd to me I guess not always. [image] Multithreading with pystan3 Loops in Stan are C++ loops so sometimes vectorization doesn’t help.

This version with a for loop: for(i in 1:N) { y[i] ~ normal(mu[i], sigma); } is massively slower as opposed to y ~ normal(mu, sigma); Now, building up a large mu vector using a for loop is ok, but definitely not calling out by element to the lpdfs.

I vectorized these loops where possible in the partial_sum function, but didn’t see significant speedup. This was odd to me, but I left it vectorized.

[image] wds15: slice parameters used per group; so if your model is a hierachical one, then slice all the parameters needed per group you mean something like this? for (w in 1:W) { target += reduce_sum(partial_sum, parameters1[,w], parameters2[,w], grainsize, data1[,w], data2[,w]) } I …

No. Never call many reduce sum like this if you can avoid it. Can u annotate your model to see where the 25k params and all that data is? Somehow easier to read model would be nicer.

Model with reduce_sum takes too long

Modeling

nerpa August 8, 2020, 4:34pm 8

I guess not always.

Topic		Replies	Views
Can't get reduce_sum to help with model runtime Modeling	7	448	December 19, 2023
Help with reduce_sum Modeling	32	1627	August 4, 2020
Reduce_sum results in much slower run times, even for large datasets Algorithms paralellization	6	1563	March 17, 2022
Trying within-chain parallelization with reduce_sum increases runtime a lot Modeling paralellization	6	1037	August 2, 2021
Reduce_sum parallelisation issue Modeling cmdstanr , multivariate-normal	12	1132	February 24, 2022

Model with reduce_sum takes too long

Related topics