Request for comments: [WIP] Stan CMake-ified Monorepo

@seantalts

Hi,

I have an alpha quality git repo here tracking Stan 2.20. It uses CMake to automatically find and configure all dependencies. I’m looking for feedback from everybody, but especially from Mac and MPI users since I haven’t tested those code-paths yet.

As you see from the README, I’m still tracking down and fixing things. However I would welcome comments from other developers on any opinions they might have, especially now where I’m at a point where it’s mostly functional.

From a developer perspective you shouldn’t see much difference except that run_tests.py is now CTest. However, there are a bunch of changes, probably too many to list in a short post here. So I encourage people to take a look and ask questions in this thread, and I will do my best to answer.

I’ve also set up a dashboard for CI results with CTest as an experiment:
https://my.cdash.org/index.php?project=stan_monorepo

I am in no way proposing anything that would affect the current Jenkins workflow. It’s just functionality you get for free by using CMake.

4 Likes

This is so awesome! Will dig into it later today, but very excited.

How easy is it to hook up test coverage tools to this now?

Depends on the tool. For things like CPPCHECK, or CPPLINT, CMake has dedicated support for inserting it into the build.

See: https://blog.kitware.com/static-checks-with-cmake-cdash-iwyu-clang-tidy-lwyu-cpplint-and-cppcheck/

As for hooking up Jenkins itself, your Jenkins scripts can hook into CTest directly.

1 Like

This looks great. Some questions

  1. Are all of the existing tests running now? How are we handling distribution tests? (We were thinking they might be run separately - assuming that’s how we’re doing it, but wondering how to run them).
  2. Do you know what’s causing the 84 failures?
  3. Would you mind going a little into the design choices you made here? Around the directory structures for src and test, how the cmake builds are designed to work, that kind of thing?

Thanks!

  1. All the tests are running, except for CmdStan; I’ve had some issues with 2.20 and the static crtp initialization. Since I was previously compiling multiple models into a fat unit test binary. I need to spend some time looking into this.

Distribution tests are handled by setting the appropriate variable at configure time, e.g.
cmake .. -G"Ninja" -DSTAN_BUILD_RVECTESTS=TRUE -DCMAKE_BUILD_TYPE=Release && ninja && ctest --output-on-failure -j<n_processors>. It should work, although I haven’t tested it in a few weeks, since I have to take special measures to run it due to how much disk space the artifacts take up.

  1. Half the failures are from the OpenCL tests failing due to an issue in 2.20 resolved here: https://github.com/stan-dev/math/issues/1297 - I just haven’t got around to rebasing again because I’m a bit wary of chasing ^HEAD until I’ve fixed everything else. If you look in the dashboard report, some of them may not be suffixed with CL, but indeed they are part of the OpenCL unit testing. Probably I ought to add a suffix to all tests that are part of the OpenCL tests. It’ll be kind of ugly though since the GTest name for the test will be “Foo.BarClCl”; a lot of the OpenCl tests call out to base Stan Math tests in cholesky... etc.

The other half, I need to spend quality time with the codebase and debug. Pull requests welcome :).

  1. Directory structures in src: I’m a little alarmed that you’re surprised by the directory structures, since this arrangement is what I thought was desired by you and @Bob_Carpenter. Can you explain how it’s different from what you expected?

Directory structures in tests: I pretty much just copied and flattened the existing test directories in Stan/Math/CmdStan.

How the CMake builds are supposed to work:

Well, hopefully, as in the Quick Start I wrote in the readme, building should look like:

  • git clone <repo_url> && cd <repo_url>
  • mkdir build && cd build
  • conan install .. --build=missing
  • cmake .. && cmake --build . -j<n_procs>

I’ve created virtual targets for the different configurations of Stan Math (Stan, Stan-with-opencl, Stan-with-mpi, Stan-with-multithreading) and you pick and choose the one you want in the build file if you’re a downstream user using CMake. Otherwise if you’re not you manually set all the header options as you did before.

As an aside, this means I generate different stanc for base Stan and stanc-with-mpi since there are MPI dependent ifdefs in the source.

All this should be pretty seamless for users since I’ve written build logic to generate these virtual targets if and only if the appropriate dependencies have been found in the system.

This also means the CI builder doesn’t really need to know anything about configuration - if the dependencies are present, the binaries/tests are automatically compiled and registered with the CTest test runner. If they aren’t, nothing happens.

  1. Using a package manager

So I guess one big difference is that I’ve included a call to conan in the quick start build steps. My goal while doing this was to make it possible to build Stan without using hard-coded dependencies in the build tree. Instead it’s managed via CMake find_package calls. Then whoever is using Stan and packaging it can seamlessly require the use of their dependencies without modifying the build script - all they have to do is make sure their path is set correctly. For example, when I am compiling in MSYS2, I don’t even need Conan since I have Sundials installed in /usr, and everything else has a system package. Still, we want to support one-command build so calling out to a cross-platform package manager seemed to be the easiest way to support that.

Also, Conan as a package manager platform allows binary artifacts to be installed side-by-side as build-requirements, so you can do stuff like distribute stanc3 binaries or Eigen-with-BLAS support in one command. I haven’t documented how to do this but on my Linux environment I use it to install CMake into a virtual environment since the system CMake is sort of ancient.

  1. Packing things into fat test binaries.

One other change is that instead of building individual executables for each source file, I compile them into .obj and link them together with a single GTest main.

This is mostly for developer ease of use. The CMake ecosystem is not designed to handle mass amounts of small binaries. Each target gets an individual entry in the CMake build graph. Each time a build is triggered CMake traverses the entire graph in order to check for changes in dependencies. This is good when the number of build targets is small compared to the number of sources. CMake automatically only rebuilds dependencies that change. When testing small changes in the code base, CMake configuration time can dominate compilation and link time, even in Stan is which full of large, repetitive template instantiations. It also makes the project must more friendly to IDE users who must browse through all targets individually in tiny dropdown menus. Basically a compile target is much more heavyweight in CMake than a Makefile target; each one individually has or has the capability of having its own compile flags, linker flags, etc.

If you wish to run an individual test, CTest supports filtering tests by name, or by label (although I haven’t used that feature yet in the codebase, it exists). There’s also running tests by binary - in Stan Math, in order to cut down linking times I’ve separated out the unit tests into smaller groupings (see CMakeLists for details). I’m not against multiple binaries, just that if there’s N sources log(N) test binaries is the way to go. I realize that one big binary has its own issues with link time. I recommend everybody working on Stan use the LLD linker, Ninja, and ccache while doing incremental building and testing.

1 Like

Just a few quick questions:

  • We used to patch boost whenever needed - and there are possibly still patches from us in there. Can we still do that and override a few boost files which have bugs?
  • How would an install of the Intel Tbb work with Conan? And would that Intel Tbb install work on windows?
  • How does Conan interact with RStan / PyStan?
  • Is CMake now a hard requirement on all platforms?
  • Would it be possible to create with CMake a set of makefiles and distribute that instead? Then users would still just need make instead of CMake.
  • Do the MPI builds use the rpath feature to hard-code the absolute path of the dynamic libraries into the target binary?
  • Is cpplint and all the Stan-math formalities now put onto the entire source (cpplint, auto formatting of code, dependency checks, …)?
  • How is the old make/local way of configuration being replaced?
  • Can you recommend some good doc of CMake and Conan?

Looks like things are getting a lot easier.

If some of these questions are answered in your readme, then a RT(*)M answer is just fine.

Sebastian

  1. Yes. This is seamless if you use Conan - it is how I provide Sundials with printf stripped for CRAN compatibility. If you do not want to use Conan then the workaround would be for us to instruct users to curl -L "https://dl.bintray.com/<stan_org>/stan/<our_modified_boost_bz2> then call cmake -DBoost_ROOT_DIR="path_to_boost_sources". Or, as a last resort, it is possible for me to write a download script in CMake to download and patch the boost sources; it has a curl equivalent inside.
  2. You put TBB as a dependency in conanfile.py. The package is located at https://github.com/conan-community/conan-tbb. It should work for all platforms if the package maintainer has done their job. If they haven’t, then we must send a patch upstream. It’s all open-source and run by volunteers. We are not stuck if they don’t accept our patch - any open source project can freely host things, so we always have the option of forking and uploading our own stuff. Then inside our own Stan repository we write a FindIntelTBB.cmake file (or more likely copy something from https://github.com/wjakob/tbb).
  3. Don’t know. I was hoping to get feedback from RStan/PyStan people.
  4. Yes. I know many systems have old versions of CMake, or no CMake at all. I hope it is not too troublesome to require that such users 1) download a binary, 2) use Conan to install CMake or 3) use pip to install CMake https://pypi.org/project/cmake/.
  5. Theoretically possible, but a bad idea since the CMake generated makefile is difficult to understand by non-technical users. I am hoping these people (probably unfortunate RTools 3.5 users) will just use Conan which is available as a python package. With a package manager, they can install any binary dependency like CMake, or (if we bothered to package it) RTools itself, an arbitrary MinGW toolchain, or even MSYS2 itself. Note: if we really wanted to make things simple for users, I can generate an actual self-extracting installer program for binaries based on the CMake configuration.
  6. I need feedback from MPI users, I am not even sure the MPI parts work right now since I’ve only been focused on making sure the OpenCL, multithreaded, and base configurations work.
  7. Not right now, although I could require that such things be run. In CMake to use them is just a configuration variable away, please see my above post.
  8. CMake replaces it all.
  9. Conan: https://docs.conan.io/en/latest/introduction.html - unfortunately the official docs are the best you will be able to get.
    For CMake, ranked in order of usefulness:
4 Likes

Very cool. Does this imply that without -DSTAN_BUILD_RVECTESTS=TRUE it would build the rest of the distribution tests? We have two modes of running the distribution tests - one that also tests all possible row vector instantiations and one that doesn’t. We don’t expect folks to run either of those locally, but we’d like to be able to run both ways on CI.

It’s almost exactly the same, I was just wondering about why the stan dir in src/stan/* and why test/math_unit instead of test/math/unit. I wasn’t sure if either or both of those made something with cmake easier.

stanc actually changes with MPI enabled? I thought just CmdStan and Math used that flag…

Sounds awesome :)

re: conan- isn’t there a way to have CMake download the required source if it’s not found? It’d be slightly nice to keep the number of new dependencies low here.

This makes sense to me. How long does it take to recompile and run a specific test? That’s the most typical workflow (the “inner loop” programming, I think). So as long as it’s relatively quick to link and you just recompile the one object for that test, I think that’s fine. We actually wanted to move to this at some point but found it too difficult with make.

How important are each of these?

  1. Row vector tests

It’s funny, but I actually had it originally the way you want it, then decided dist tests without row vectors was redundant and deleted it. I’ll restore the functionality.

  1. <stan/*>

I keep everything under the <stan/*> directory so that stuff is properly namespaced; it was a thing from Stan Math.

  1. stanc and MPI

This is my mistake, you’re right, stanc doens’t change. Cmdstan does.

  1. Conan

Yes, but I don’t recommend this. As a scripting language, we can use CMake to build arbitrary projects. However, it is not good at handling non-CMake dependencies, so you have to shell out and do everything there. I don’t think Stan developers should be in the business of maintaining their own package manager - let the system distros handle it, or the maintainers of RStan/PyStan who may need their own workarounds anyway for CRAN, PyPi, Conda, etc (e.g., Sundials printf stripping). Building any of the non-header-only Boost libraries in CMake is a hassle; Stan developers would end up
maintaining a fork of https://github.com/Orphis/boost-cmake, for example.

Conan is there as a fallback; I don’t need it at all when building on Arch Linux. I actually had a version of the monorepo where I used CMake to download everything at configure time, and it’s a pain handling build logic not only for yourself, but also your dependencies. It is less work for package maintainers and developers if we outsource this to a package manager; and just one more command for individual users python3 -m venv env && source env/bin/activate && pip3 install conan.

Still, it is possible, let me know and I can reinsert the appropriate logic.

  1. Link times

To compile and run a single test is pretty fast… as long as you are using a linker that supports incremental linking. As of right now, that’s LLD and Visual Studio.

  1. Necessity of using LLD, Ninja, CCache

These are not necessary, of course, but they will make your dev life more pleasant because you spend much less time waiting for things to compile and link. I have no benchmarks, I just tried them and realized they drastically cut down my dev idle time. On MSYS2 and Ubuntu they are availabe in the system package managers, not sure about Homebrew.

PyStan doesn’t use GNU make and will not use CMake. It uses the Python build machinery (distutils). I don’t foresee any problems.

That all sounds pretty reasonable to me. Does anyone else want to take a look at this stage? /cc @syclik @Bob_Carpenter

I think once the tests are all passing we’ll try to get @serban-nicusor to work on setting up a Jenkins job pointing to your fork of the monorepo. Once that’s running all the old tests we can finally force push to stan-dev/stan, run the import scripts, and rename math and cmdstan to something like math-deprecated-use-stan-monorepo or something.