Perf-math benchmark

Developers
1

Is it possible to benchmark multiple branches at once?

2

Do you mean the perf-math repository? That has since been superseeded by the benchmarks in Stan Math: math/benchmark.py at develop · stan-dev/math · GitHub

This python script generates the Google Benchmarks and generates CSV files and even plots speedups and comparisons.

The perf-math itself didnt handle multiple branches. The workflow is: run for branch 1, switch branch and clean, run for branch 2.

1 Like
3

oooh, ok that’s great! Would you happen to have some setup steps, let’s say for example GitHub - spinkney/math at faster_inv_phi and/or https://github.com/spinkney/math/tree/boost_inv_phi:)?

4

Here is a quick guide:

  • Step 1: Setup Google Benchmark
cd lib/benchmark_1.5.1
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=RELEASE
make
cd ../../..
  • Step 2: Run for the base branch
git checkout base_branch
./benchmarks/benchmark.py "inv_Phi(vector) => vector" --skip_similar_signatures --csv base.csv
  • Step 3: Run for the new branch and plot the speedups
git checkout test_branch
./benchmarks/benchmark.py "inv_Phi(vector) => vector" --skip_similar_signatures --csv test_branch.csv --plot_reference base.csv --plot window

Let me know if you need any more details. ./benchmarks/benchmark.py --help also reveals a few more arguments that you can use if you need to test different sizes, or change the number of repetitions.

5

It works, thanks!

6

Thanks @rok_cesnovar for your help. I have my 3 benchmark csvs in a folder, what’s the best way to plot them using that script?

7

Once you already have the CSV files its just plotting.

The benchmark script only supports plotting immediately after a run, but you can easily use the functions inside the script:

And

Not a a computer right now, but can give an example if needed.

8

Yea no worries, I ended up just taking that plot function and working with it.

Output at updated inv_Phi by spinkney · Pull Request #2566 · stan-dev/math · GitHub

1 Like
9

Nice! Very cool.