One issue that I think use some more discussion is tuning the parallelization in reduce_sum. More and more users are exploiting cloud resources and the common intuition based on all threads being on one machine, or at least being on one machine with few contested resources, often doesn’t translate particularly well. While there won’t be a general solution by any means I think it would help to set better expectations by discussing the circumstances when reduce_sum might actually offer practical speed ups.
2 Likes