Parallelizing the sampler (not the model)

[Bob_Carpenter]

@betanalpha

There’s a decent amount of allocation in the dynamic trajectory evaluation, but the actual memory footprint at any give time is quite small. The cost of those memory allocations is unlikely to substantial relative to the gradient evaluations themselves,

We were just discussing this in another thread starting with this comment from @Red-Portal.

Memory pressure becomes a much bigger deal when parallelizing, as typically only a relatively tiny L1 cache is on-core and the L2, L3 cache and RAM are shared. There’s also the problem of memory locality with a bunch of ad-hoc allocations. Allocating everything together in contiguous memory could be a big saving. This is something I really messed up in coding Stan arrays as C++ std::vector as we have this problem all over the place.