Is it possible to benchmark multiple branches at once?
Do you mean the perf-math repository? That has since been superseeded by the benchmarks in Stan Math: math/benchmark.py at develop · stan-dev/math · GitHub
This python script generates the Google Benchmarks and generates CSV files and even plots speedups and comparisons.
The perf-math itself didnt handle multiple branches. The workflow is: run for branch 1, switch branch and clean, run for branch 2.
oooh, ok that’s great! Would you happen to have some setup steps, let’s say for example GitHub - spinkney/math at faster_inv_phi and/or https://github.com/spinkney/math/tree/boost_inv_phi:)?
Here is a quick guide:
- Step 1: Setup Google Benchmark
cd lib/benchmark_1.5.1 mkdir build && cd build cmake .. -DCMAKE_BUILD_TYPE=RELEASE make cd ../../..
- Step 2: Run for the base branch
git checkout base_branch ./benchmarks/benchmark.py "inv_Phi(vector) => vector" --skip_similar_signatures --csv base.csv
- Step 3: Run for the new branch and plot the speedups
git checkout test_branch ./benchmarks/benchmark.py "inv_Phi(vector) => vector" --skip_similar_signatures --csv test_branch.csv --plot_reference base.csv --plot window
Let me know if you need any more details.
./benchmarks/benchmark.py --help also reveals a few more arguments that you can use if you need to test different sizes, or change the number of repetitions.
It works, thanks!
Thanks @rok_cesnovar for your help. I have my 3 benchmark csvs in a folder, what’s the best way to plot them using that script?
Once you already have the CSV files its just plotting.
The benchmark script only supports plotting immediately after a run, but you can easily use the functions inside the script:
Not a a computer right now, but can give an example if needed.
Yea no worries, I ended up just taking that plot function and working with it.
Nice! Very cool.