Hi, When I try to run a stan file in parallel with joblib/multiprocessing through pystan, the sampling hangs at 0%. The same type of code runs if I remove for loops from the stan code. I have run multiple types of stan programs in parallel with joblib where there is no for loop and they run perfectly fine, but a simple program with for loops in stan code causes program to hang in parallel. If i set n_jobs=1 , same code runs fine without any issues.
For pystan 3 I’m not sure if multiprocessing / joblib is the way to go given that all of them call probably the same httpstan server. Have you tried threading?
I am using pystan3 for this . I have not used threading until now . I have a CPU intensive module which also utilizes pystan. Threading might not be able to deal with it. Let me try threading too. Can you suggest some other tool instead of pystan to use multiprocessing . Can cmdstanpy work ?
I have started using asyncio instead of multiprocessing , it is working code runs without breaking but it is extremely slow, simple for loop based execution is much faster than multithreaded approach. May be its due to high intensive CPU tasks.
Careful: asyncio is not parallel execution – it runs in a single thread with a single interpreter. It is more like a complicated version of control flow.
Yes, I have observed asynchio is too slow if its a CPU intensive task. What I believe with the above case is that, when stan processes for loop it makes multiple threads, thus when multiprocessing module is run where the child programme is itself calling multiple threads then the program execution stops
down. I had observed similar thing on another project. In that project I had disabled all threadings and paralleization inside an already parallelized function. Is there a way I can completely disable any kind of threading inside the stan.sample() function. I strongly believe error is arising from some where here.
Yes, exactly, I have run many complicated stan models , they donot have for loop and they run fine. For a comparision in the same code I run 20 concurrent complicated stan models with joblib and they run perfectly fine without any errors. They donot have for loop in them though. But in this stan code with for loop I am facing this error. I have reduced n_jobs=2 in joblib and still teh same issue. I will try your suggestion now too. Unfortunately, due to some policies I am unable to share my code . It might have been helpful, but I use same piece of code with different stan models(non for loop stan) and they run perfectly fine.
