I read here that within-chain parallelization is not supported in Stan due to the mentioned reasons. However, when I run my inferences in Stan I often notice that multiple different chains are getting processed concurrently. Does this mean Stan supports parallelization between chains? If yes, is this done automatically, or does the user need to add additional arguments during building the model or compiling the model?
I’m asking because I’d like to parallelize my code among the chains if possible. I’m running on a cluster, which is based on Slurm. I’m essentially wondering if I can be as relaxed as simply issuing the Slurm flag --cpus-per-tasks=10 and have Stan take care of the parallelizng the chains, or I need to do something more involved.
I think this is somewhat outdated. IIRC, map_rect handles within-chain parallelism.
It’s not so much parallelism between chains, since each chain is independent of the others. There’s no communication between the chains.
It’s a little more involved, but not by that much. If you aren’t bothering with map_rect, then you can tell your Stan implementation to create as many chains as you have cores available. That may involve, say, passing $SLURM_CPUS_ON_NODE to Stan somehow, which will depend on which implementation you use, e.g. RStan, PyStan, CmdStan, etc.
Yeah of course! But I don’t plan to alter that value often. So, for my own edification, Stan automatically recognizes the CPU resources available to it and distributes independent chain processes across them, with a 1-to-1 correspondence? That’s quite amazing!
I think it’s actually the underlying OS that does that. AFAIK, the Stan implementations RStan and PyStan can put each chain on its own thread, and it’s the OS’s responsibility to figure out what threads go on what CPU cores. Typically, though, there will be 1 chain per core in practice.
I’m not sure, since I have no experience with slurm, myself. I presume that it’s similar to other cluster schedulers I’ve used, i.e., I write a job script that uses special comments and environment variables and is submitted to the scheduler via some command-line utility.
I’m not sure why this got flagged, but I’m guessing it’s by accident. It shows up in my to-review and I’m not sure whether giving a thumbs-up agrees with flag or a thumbs-down agrees with flag, so I’m afraid to touch it.