Is there an example in the stan math library which illustrates the pre-allocation design pattern? I have looked at code for calculating the variance and I think this goes in the direction of what you have in mind. Although, you want to go one step further and link parents to childs already in advance as I understand you. Once the real function values and Jacobians stream in they are filled into the right places... if I got you right.
The trick will be to make the deserialization process work exactly this way. I think then it will run asynchronously, although I have to check the boost mpi implementation here.
I have invested some thought into how this goes in the language, but this is far from user-friendly what I have. Essentially the design follows the ODE pattern. So the user defines a function with the signature
real mpi_function(real theta, real x_r, int x_i)
which does the work for each task. The user first has to call the function
int setup_mpi_function(int M, real[,] X_r, int[,] X_i)
which behaves differently on the root node and on the workers. On the root node this function does not do much except to calculate the work chunk assignment, but on the worker node this function goes essentially into an infinite loop and it waits for sets of parameters to receive from the root for which to calculate the function and the Jacobian. The setup function is intended to be called inside the transformed data block. I think in a proper Stan implementation we could call this function after the transformed data block.
Inside the model the user then calls a function with the signature
vector run_mpi_function(real[,] Theta, int chunks, real[,] X_r, int[,] X_i)
The chunks are from the setup function, Theta are all the parameters, X_r/X_i is real and integer data. These data-structures define per row one job (a ragged array and possibly tuples would be nicer here).
This way of coding works, is similar to the ODE pattern, but maybe not super user friendly.
If you want I can throw my research code on a branch on cmdstan so that you can have a look if you like.