MPI Design Discussion

[Bob_Carpenter]

@wds15

the final output is a ragged vector.

That won’t work with Stan—we don’t have a ragged vector type yet.

@wds15

should be able to support functors efficiently which are static

That’ll cover our current functions. It won’t allow lambdas, though, which produce what is essentially a functor.

@wds15

map_rect should compile without any MPI stuff if you do not define STAN_HAS_MPI.

Great. I must have misunderstood what I was reading.

@wds15

to split the map/reduce operation such that we essentially have two steps

That sounds good to me.

@wds15

the first reduce step is fully ignorant of MPI

It’s great you coded it that way.

@wds15

This is the reason why the map_rect_serial uses map_rect_reduce as reduce step and it uses map_rect_combine as combine step. Both are MPI free. On the other hand, the map_rect_mpi uses map_rect_reduce (the same as before, no difference) and mpi_map_rect_combine. the mpi_map_rect_combine internally does the combine step by first collecting things at the root and then just using map_rect_combine.

OK, I think that answers my question above, which I’ll pull down here just for confirmation.

@wds15

the map_rect_serial implementation , the one here, is not parallelized, but it does work exactly the same way as the MPI implementation. I do not understand why it is a bad idea to do like this. After all, this implementation cuts down a larger problem I gave it from 17h down to 10h on a single-core run.

By “works exactly the same way”, does it depend on MPI and just run multiple processes on one core? Or do you just mean that it reduces the autodiff tree by doing nested autodiff and producing smaller expression graphs overall? If it’s the latter, that’s fantastic. Even if it’s the former, that’s good to know and would be good to know how it scales. 70% faster code is fantastic and it’d be nice to exploit this trick elsewhere.