Experiment Frameworks for Running/Comparing Lots of models/configs/results

Funny timing, I just gave a talk on Stan + MLFlow at StanConnect Ecology part 1 – we’ve used this extensively over the past few months to great effect. I find it works well once you get things set up, and fits neatly in a Bayesian workflow/MLOps pipeline.

Edit: The compelling use case for me is that a Bayesian workflow involves a lot of experiments. Tracking experiments helps organize your work and more systematically see whether your development effort on a model is resulting in improvements. That said, diligent tracking of experiments is hard when it must be done manually. The value proposition of tools like MLFlow is it automates this tracking, which makes it easier to navigate the Bayesian workflow. As a nice side effect, MLFlow also provides a way to share results and deploy models more easily.

Some very minimal examples with cmdstanr, brms, and lm here, along with slides: GitHub - mbjoseph/mlflow-stan: MLFlow with cmdstanr

6 Likes