Packaging python project with cmdstan as dependency

Interfaces
,
1

Hi everyone,

I am currently trying to figure out the dos and don’ts of building and publishing python packages. It’s all new and cryptic to me.

My package uses cmdstanpy and therefore has cmdstan as dependency. I set up my package to install cmdstan on the first run including the toolchain (for windows machines). It’s working, but takes time and cmdstan just eats up so much disk space.

My reference is Facebook’s Prophet and they managed to boil down cmdstan to only some 10 MB. I guess they kept only files for executing but not compiling models, which I could confirm when I stumbled over this post. However, I really don’t understand which files to keep and which are disposable. Is that straightforward to explain?

Also, I do everything on Windows and fear a bit what happens when a Mac or Linux users wants to install my package.

  • is there no way they install the toolchain automatically (like with the compile=true flag for Windows)? I understand Linux should have the toolchain but is this something I can rely on?
  • if I want to pack only the executables, I have to pre-compile my models for all systems and architectures I like to support? And I guess the cmdstan files required for running the models also differ between systems?

Ah, I am so clueless I am not even sure my questions make sense. I hope you know what I like to achieve and would appreciate if you had some pointers for me.

Thanks a lot!

1 Like
2

I was one of the authors of the way Facebook prophet currently distributes their package. I’ve created a cookiecutter template for people who want to do similar things. It can set up the basics of copying what they do for you.

You are correct that the basic idea is to keep only the executable and one library from cmdstan as the bare necessities for running a model. This is then combined with a github action that builds a separate wheel for each of the major platforms, which means the vast majority of end users will not need a toolchain, they will just get what you’ve compiled for them (on GitHub’s servers)

Happy to answer more questions if you give it a try!

3 Likes