Perf-math benchmark

Is it possible to benchmark multiple branches at once?

Do you mean the perf-math repository? That has since been superseeded by the benchmarks in Stan Math: math/ at develop · stan-dev/math · GitHub

This python script generates the Google Benchmarks and generates CSV files and even plots speedups and comparisons.

The perf-math itself didnt handle multiple branches. The workflow is: run for branch 1, switch branch and clean, run for branch 2.

1 Like

oooh, ok that’s great! Would you happen to have some setup steps, let’s say for example GitHub - spinkney/math at faster_inv_phi and/or

Here is a quick guide:

  • Step 1: Setup Google Benchmark
cd lib/benchmark_1.5.1
mkdir build && cd build
cd ../../..
  • Step 2: Run for the base branch
git checkout base_branch
./benchmarks/ "inv_Phi(vector) => vector" --skip_similar_signatures --csv base.csv
  • Step 3: Run for the new branch and plot the speedups
git checkout test_branch
./benchmarks/ "inv_Phi(vector) => vector" --skip_similar_signatures --csv test_branch.csv --plot_reference base.csv --plot window

Let me know if you need any more details. ./benchmarks/ --help also reveals a few more arguments that you can use if you need to test different sizes, or change the number of repetitions.

It works, thanks!

Thanks @rok_cesnovar for your help. I have my 3 benchmark csvs in a folder, what’s the best way to plot them using that script?

Once you already have the CSV files its just plotting.

The benchmark script only supports plotting immediately after a run, but you can easily use the functions inside the script:


Not a a computer right now, but can give an example if needed.

Yea no worries, I ended up just taking that plot function and working with it.

Output at updated inv_Phi by spinkney · Pull Request #2566 · stan-dev/math · GitHub

1 Like

Nice! Very cool.