Torsten 0.87 features dynamic load balancing for MPI integrators, complementing existing static load balancing scheme. The dynamic load balancing benefits use-cases when the computation load is uneven among the population. As an example, in a typical population PK-PD trial, various treatment arms likely involve different dosing period, so that the numerical integrator will be having different t_{\text{end}} for different subjects. In this scenario, static load balancing leads to suboptimal performance. Similar scenario arise when during parameter inference certain subject’s ODE become stiff (as proposed parameter changes the behavior of the ODE) and the others’ are non-stiff. The comparison below is based on integrating 128 subjects consisting uneven loads.
Nice! Can you link to the dynamic load balancer pr / commit? Did you write your own or use tbb?
Really nice!
Could you post another comparison with static-randomOrder? What I mean by that is to take your data set with unequal computational load per subject and then just reshuffle the order of the subjects randomly. When you do that then the static scheduling should still be OKish… at least this is what I always was hoping for when developing map_rect.
(this is not to say that scheduling is not needed, no…random recording of the data should not be a requirement to make things fast)
There’s no PR as it’s in Torsten, a forked Stan that I’ve been working on.
This is for distributed computing so not thread-based(not that I cannot take advantage of @wds15 's work in the future).
Yes, planned after stancon.
static & dynamic balancing are orthogonal, as they target different applications(Also, Torsten’s static scheme is not based on master-slave pattern). So without evidence I’d say static scheme will be OKish until it’s not, and that’s when dynamic balance scheme kicks in. Also note Torsten’s doesn’t use “sharding” so dynamic scheme has extra benefit.