Supporting CmdStan installation via conda

This was discussed today during the Stan developer meeting, and this thread serves to document the process and try to find other users who can help.

Some discussion is also visible here, along with the existing attempt: FR: Support complete installation via conda · Issue #395 · stan-dev/cmdstanpy · GitHub

Summary: We would like to simplify the installation of CmdStan for CmdStanPy. In particular, we would like to use the conda package manager to install CmdStan automatically when CmdStanPy is installed.

Why? Because conda can handle the additional requirements for compilers and other unix tools automatically. This means Windows users for example won’t need to separately install RTools.

Current Status:
We have various attempts which succeed in building cmdstan in the docker test images conda-forge uses. The main problem we’ve encountered is that the pre-compiled stanc3 seems to segfault when called. A combination of infamiliarity with conda recipes and with the specifics of the cmdstan installation make debugging this rather difficult.

3 Likes

@bparbhu I’d love to chat sometime about your similar effort. The most recent road block we’ve encountered seems to be related to some of the c++ libraries stan math uses (see this comment). Someone more experienced with the compilation here would probably have a much easier time making sense of this

1 Like

HI @WardBrian,

Happy to discuss anytime on Wednesday this week. If you want I can set up some facetime for us to talk more about what we’re both doing. The route I’m taking is more of a graphical user install using a combination of NSIS & PowerShell for windows, Bash scripting for Linux, and most likely some other route for mac installs. My email is brian.parbhu@gmail.com, I’ll send out an invite for Wednesday just let me what time works best for you and also be sure to send me your email.

Thanks again,

-Brian

In terms of the tbb library, I think that is in relation to the tbb/intel math library that CmdStan can use as well. Though, I’m not too sure about the specifics of what is going on there. Though, when installing CmdStan, we do make sure to run an install command around that library. So it would be something like make install-tbb or mingw32-make install-tbb on Windows 10.

Installation via conda is now passing all the tests on the conda-forge CI platform. Hopefully they can complete review and ship the package very soon

1 Like

For those following along at home, cmdstan is is now available on conda-forge on Windows, Linux, and MacOS.

In the future this should be able to support non-x86 architectures once we have stanc3 builds for them, which I know is being worked on. We should also be able to support MPI builds via conda.

7 Likes

Now we just need a cmdstanr package in conda forge and we are good to go! :)

I know nothing about R packaging, but I have learned quite a bit about Conda and conda-forge doing this. If anyone wants to try to add cmdstanr to conda-forge I’m happy to help/be a resource

It turns out that it is very easy to mirror a CRAN package on conda-forge (see GitHub - bgruening/conda_r_skeleton_helper: Cleaning up Conda r-packages), but CmdStanR is not yet on CRAN. @jonah - is that still being persued?

Cool, yeah we still want to get cmdstanr on CRAN. @rok_cesnovar and I should probably have a chat about which issues we absolutely need to close before then. We have a 1.0 milestone but I’m not sure if that has to be exactly equivalent to the CRAN release milestone.

1 Like

Just flagging that the non-x86 stanc3 building has been merged and the binaries are available in the latest nightly: Release nightly: Merge pull request #936 from andrjohns/feature/multiarch-binaries-2 · stan-dev/stanc3 · GitHub

I’m guessing it’s not conda-compatible until they’re built for releases, but could be good for testing at least

2 Likes

For it to be available on conda, cmdstan needs to package them in a release on github. I noticed that the linux-arm build already has been, so I can try to learn more about the conda-forge architecture settings using that

Oh right. I’m working on the cmdstan integration at the moment, so will let you know

Great! Are there plans for similar support for native builds for M1 macs? (Not sure if OCaml even supports this, we’d probably need a newer version than we have pinned)

I think the platforms Conda supports are:

win64
mac64
mac-arm64
linux-64
linux-aarch64 (arm)
linux-ppc64el

So we’re currently at 4/6, with the fifth already done on the stanc3 side

Native builds for M1 are a bit tricky since there aren’t any M1 Github runners we can use to build the binaries, and there isn’t the incentive to use an EC2 instance (like we previously did for linux-aarch64) since the Stanc3 binaries will run through Rosetta.

One approach that I was thinking of investigating is using x86 OSX runners to build Universal binaries: Apple Developer Documentation, but I have no idea what that would entail for our build process at the moment