Providing built stanc3 with additional (non-master branch) features

We currently have a separate branch of Cmdstan+Stan+Math that supports some additional custom functions we added to stanc2.

The functions are very specific and it probably makes no sense to try to add them to the Stan language. We would now like to move the whole thing to Stanc3 and provide the users with a similar experience as the current develop cmdstan has. Specifically the “download the stanc3 binary based on the OS” thing.

I guess one way of providing a similar experience without much overhead for us would be to ship the JS stanc3. But that means that the users need to have V8 installed or some other JS engine. Which at the end of the day is probably not so bad.

But if want we to provide the OS specific binaries we will need to build stanc3 for all three OSes.

@seantalts @serban-nicusor do we have any scripts to do that that are available for sharing. Besides that we probably only need the ability to spawn virtual machines for Mac, Linux, Windows with stanc3/ocaml prerequisites. Anything else?

I am guessing with stanc3 making adding stuff much simpler there might be more people that will want something like this. I am happy to document my process and make a wiki if anyone else would be interested in doing something similar from scratch.

On a scale from 1 to Catalina, that is a 2.

3 Likes

Haha, yeah, fair point :) I TA an embedded systems class where there is an added nuance to all this Catalina compiler stuff with JTAG debuggers and all that and the final conclusion was to just install a Windows virtual machine.

And you know things are not great when people rather use Windows for low-level development :)

I think the Catalina problems are behind us now. But it is bizarre that Windows has become the easiest operating system to install Stan on.

2 Likes

I was going to use Conan and bintray to package stanc3 when it was released, since I’m putting Stan math / stan services / cmdstan on there too. It may be an option you’re interested in for hosting binaries.

Another open-source project I’m involved in has had good success using Nix to manage binary packages, along with auto builds via Azure Pipelines. Now that all Github projects come with certain free amounts of CI/CD build time via Github actions, that is also an option. This is something that you could do that would not affect the main Jenkins CI pipeline. (I guess the issue here is, Nix is pretty terrible on Windows.)

1 Like

Is the question “How do you build stanc3 for each OS?” If so it’s all in the Jenkinsfile. You guys are making a custom stanc3?

1 Like

Yeah, that would be the shorter version, haha. Thank you!

The long story is that we have a very specific Stan Math function currently made with precomputed gradients that is only suitable for specific model(s) for temporal progression modelling on data from clinical trials focusing on neurogenerative diseases from https://c-path.org/. The Stan Math function has an OpenCL optimization that makes it ~100 times faster (sampling goes from 3 days to 45mins). But it uses a weird way of caching that we had to resort to (it works but really for only this model). This also makes it really hard to maintain and extend for further work. Hence my wish to move to stanc3.

Currently we make cmdstan executables for that model for Windows and Linux (the end-users are only interested in those for now) and ship that with a R package that is just a cmdstan executable wrapper for this specific use-case (we started working on this before cmdstanr, we would just use that now). I would want to move stanc3 and also provide the option for the users to compile the model themselves.

The R package lives here: https://github.com/bstatcomp/project-cpath but its still a WIP.

2 Likes

Cool. Is this something that could move to using CmdStanR as the backend at some point?

Yes, that is the plan for the next time we will need to extend the package.

1 Like