# Rstan with TBB

**URL:** <https://discourse.mc-stan.org/t/rstan-with-tbb/13122>\
**Category:** Interfaces\
**Created:** [February 14, 2020, 3:03pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122 "2020-02-14T15:03:12Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![inferrator](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/inferrator/32/7140_2.png) [@inferrator](https://discourse.mc-stan.org/u/inferrator)\
**Post date:** [February 14, 2020, 3:03pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/1 "2020-02-14T15:03:12Z")

</div>

Dear Stan community:  
It’s been a while since I played with this. I created a stan model using map\_rect, and I’d like to run it using several cores via TBB. I haven’t tried this with cmdstan yet, but I would prefer to use rstan for seamless model tuning. Details below. Am I doing something wrong, or is this just not supported yet in rstan?

Operating System: Clear Linux  
Interface Version: 2.19.3  
Compiler/Toolkit: g++ & clang++

_devpkg-tbb_ is installed. I have also tried putting the tbb/include files into rstan/include, same outcome.

Makevars: (tried both g++ and clang++)

```
CXX=g++
CXX14FLAGS += -DSTAN_THREADS
CXX14FLAGS += -pthread
CXX14FLAGS += -fPIC

```

Error when compiling with `stan_model` in R:

```
Error in dyn.load(libLFile) : 
  unable to load shared object '/tmp/Rtmp1K845W/file6dc360d72dc.so':
  /tmp/Rtmp1K845W/file6dc360d72dc.so: undefined symbol: _ZTIN3tbb4taskE
Error in sink(type = "output") : invalid connection

```

---

<div class="post-metadata">

**Author:** ![bgoodri](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bgoodri/32/4451_2.png) [@bgoodri](https://discourse.mc-stan.org/u/bgoodri)\
**Post date:** [February 14, 2020, 3:10pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/2 "2020-02-14T15:10:34Z")

</div>

Does the `psapply` example in

[https://cran.r-project.org/web/packages/StanHeaders/vignettes/stanmath.html](https://cran.r-project.org/web/packages/StanHeaders/vignettes/stanmath.html)

work?

---

<div class="post-metadata">

**Author:** ![inferrator](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/inferrator/32/7140_2.png) [@inferrator](https://discourse.mc-stan.org/u/inferrator)\
**Post date:** [February 14, 2020, 3:27pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/3 "2020-02-14T15:27:45Z")

</div>

No. Lots of warnings, and the same error message (undefined symbol: \_ZTIN3tbb4taskE).

It only compiles if I remove the Makevars file / omit the `Sys.setenv(PKG_CXXFLAGS = "-DSTAN_THREADS")` setting

---

<div class="post-metadata">

**Author:** ![wds15](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wds15/32/908_2.png) [@wds15](https://discourse.mc-stan.org/u/wds15)\
**Post date:** [February 14, 2020, 3:49pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/4 "2020-02-14T15:49:30Z")

</div>

Is `cmdstanr` an option for you? Head over to the stan GitHub where you can find that R package which makes running stan models from R with cmdstan easy.

---

<div class="post-metadata">

**Author:** ![inferrator](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/inferrator/32/7140_2.png) [@inferrator](https://discourse.mc-stan.org/u/inferrator)\
**Post date:** [February 14, 2020, 3:58pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/5 "2020-02-14T15:58:49Z")

</div>

I’ll take a look.  
Still, it would be nice to get it work in rstan, too.

---

<div class="post-metadata">

**Author:** ![bgoodri](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bgoodri/32/4451_2.png) [@bgoodri](https://discourse.mc-stan.org/u/bgoodri)\
**Post date:** [February 14, 2020, 4:44pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/6 "2020-02-14T16:44:20Z")

</div>

It does work in rstan, just not for you apparently. Are you using the TBB from the RcppParallel package or from a package manager?

---

<div class="post-metadata">

**Author:** ![inferrator](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/inferrator/32/7140_2.png) [@inferrator](https://discourse.mc-stan.org/u/inferrator)\
**Post date:** [February 14, 2020, 4:46pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/7 "2020-02-14T16:46:11Z")

</div>

I have tried it two ways,  
(1) I downloaded TBB from intel, and copied include dir into the rstan include directory  
(2) removing (1), I used TBB from clear’s package manager

---

<div class="post-metadata">

**Author:** ![bgoodri](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bgoodri/32/4451_2.png) [@bgoodri](https://discourse.mc-stan.org/u/bgoodri)\
**Post date:** [February 14, 2020, 4:56pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/8 "2020-02-14T16:56:26Z")

</div>

I forgot that for rstan 2.19.x, `map_rect` is implemented with just C++11 threads, rather than TBB. So, you don’t need any TBB stuff to get that to work.

For rstan 2.21.x — which is on GitHub but was not accepted by CRAN — you do need the RcppParallel package but you don’t need to be including TBB sources in rstan sources or anything like that.

---

<div class="post-metadata">

**Author:** ![inferrator](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/inferrator/32/7140_2.png) [@inferrator](https://discourse.mc-stan.org/u/inferrator)\
**Post date:** [February 14, 2020, 4:58pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/9 "2020-02-14T16:58:47Z")

</div>

It ran once. Then created the make/local file, and it compiled fine.  
However, now I have some cmdstanr issue, it throws this:

```
Error in gsub(addnlpat, "\\1\n", str) : 
  invalid regular expression '(.{1,256})(\s|$)', reason 'Invalid contents of {}'

```

Which I believe is an unrelated issue with cmdstanr.

---

<div class="post-metadata">

**Author:** ![inferrator](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/inferrator/32/7140_2.png) [@inferrator](https://discourse.mc-stan.org/u/inferrator)\
**Post date:** [February 14, 2020, 5:00pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/10 "2020-02-14T17:00:15Z")

</div>

So on rstan 2.19.2/3 it should multi-thread out of the box without changing the makevars file?

Also, I should really switch to the development version, I guess. The v8 dependency is nasty, though.

---

<div class="post-metadata">

**Author:** ![bbbales2](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bbbales2/32/77_2.png) [@bbbales2](https://discourse.mc-stan.org/u/bbbales2)\
**Post date:** [February 14, 2020, 5:24pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/11 "2020-02-14T17:24:46Z")

</div>

> [@inferrator](#):
>
> However, now I have some cmdstanr issue, it throws this:

Any chance you could make an issue over here: [Issues · stan-dev/cmdstanr · GitHub](https://github.com/stan-dev/cmdstanr/issues) and post a model/data/R script that lead to this?

---

<div class="post-metadata">

**Author:** ![inferrator](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/inferrator/32/7140_2.png) [@inferrator](https://discourse.mc-stan.org/u/inferrator)\
**Post date:** [February 14, 2020, 6:17pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/12 "2020-02-14T18:17:55Z")

</div>

I’d love to, but now it works again. I have no idea what it was. Maybe something in the .stan file, that still let it compile but didn’t let it run? I’ll keep an eye on it.

Is there a way I can look at the cpp file with cmdstanr? I’d like to know if the DSTAN\_THREADS flag worked. I still don’t see the sampler use multiple cores.

---

<div class="post-metadata">

**Author:** ![bgoodri](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bgoodri/32/4451_2.png) [@bgoodri](https://discourse.mc-stan.org/u/bgoodri)\
**Post date:** [February 14, 2020, 6:23pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/13 "2020-02-14T18:23:11Z")

</div>

> [@inferrator](#):
>
> So on rstan 2.19.2/3 it should multi-thread out of the box without changing the makevars file?

You should only need to add `-DSTAN_THREADS -pthread` to the compilation flags and then at runtime set the environmental variable `STAN_NUM_THREADS`.

> [@inferrator](#):
>
> Also, I should really switch to the development version, I guess. The v8 dependency is nasty, though.

Well, the C++ version of `stanc` will go away some day (hopefully soon), but installing the V8 R package should suffice if you have the underlying library installed from the package manager. Having a dependency on the OCaml libraries would be more onerous than JavaScript from R’s perspective.

---

<div class="post-metadata">

**Author:** ![bbbales2](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bbbales2/32/77_2.png) [@bbbales2](https://discourse.mc-stan.org/u/bbbales2)\
**Post date:** [February 14, 2020, 6:39pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/14 "2020-02-14T18:39:32Z")

</div>

Try compiling with `threads = TRUE` in the cmdstanr compile function ([https://mc-stan.org/cmdstanr/reference/model-method-compile.html](https://mc-stan.org/cmdstanr/reference/model-method-compile.html)).

You also need to set the number of threads you’re using with `set_num_threads` ([https://mc-stan.org/cmdstanr/reference/stan\_threads.html](https://mc-stan.org/cmdstanr/reference/stan_threads.html)).

---

<div class="post-metadata">

**Author:** ![inferrator](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/inferrator/32/7140_2.png) [@inferrator](https://discourse.mc-stan.org/u/inferrator)\
**Post date:** [February 14, 2020, 7:21pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/15 "2020-02-14T19:21:02Z")

</div>

Thanks. `num_threads` was correct, but I recompiled with `threads=TRUE`. Still, not using the cores well. It could be my model. I want to look at different cuts to create the shards. However, is there another way to verify that it’s using all the cores? When running, it says ‘Running MCMC with 1 chain(s) on 12 core(s)…’, but is that definitely correct?

---

<div class="post-metadata">

**Author:** ![bbbales2](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/bbbales2/32/77_2.png) [@bbbales2](https://discourse.mc-stan.org/u/bbbales2)\
**Post date:** [February 14, 2020, 7:33pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/16 "2020-02-14T19:33:38Z")

</div>

> [@inferrator](#):
>
> When running, it says ‘Running MCMC with 1 chain(s) on 12 core(s)…’, but is that definitely correct?

That sounds right.

@wds15 anything special for map\_rect?

---

<div class="post-metadata">

**Author:** ![inferrator](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/inferrator/32/7140_2.png) [@inferrator](https://discourse.mc-stan.org/u/inferrator)\
**Post date:** [February 14, 2020, 7:58pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/17 "2020-02-14T19:58:57Z")

</div>

Thanks so much. I found one issue in my model file, and now it’s working much faster. It’s still not producing 100% load on 12 cores, but pretty high load on 8 of them. So that’s pretty nice already. Now I can do more tweaking and tuning on my model.

(all using cmdstanr, which is nice btw)

---

<div class="post-metadata">

**Author:** ![wds15](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/wds15/32/908_2.png) [@wds15](https://discourse.mc-stan.org/u/wds15)\
**Post date:** [February 14, 2020, 10:14pm UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/18 "2020-02-14T22:14:56Z")

</div>

The tbb version of map rect since version 2.21 is a lot faster than previous ones. This is only available with cmdstan at the moment.

---

<div class="post-metadata">

**Author:** ![inferrator](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.mc-stan.org/inferrator/32/7140_2.png) [@inferrator](https://discourse.mc-stan.org/u/inferrator)\
**Post date:** [February 15, 2020, 12:35am UTC](https://discourse.mc-stan.org/t/rstan-with-tbb/13122/19 "2020-02-15T00:35:15Z")

</div>

It works quite well, and `cmdstanr` works like a charm. I think I want to try MPI next, since I have a number of cores to keep busy.  
I am not sure what to mark as a solution. For me the real solution is to switch to cmdstan, since it actually uses TBB. Strictly speaking, the question was about rstan, so either using the development version or just using threads would be the answer. Either way, thanks everyone.