Provides CMake functionality to register benchmars with google benchmark (similar to ctest and gtest). Provides python scripts to execute and visualize the results from the benchmarks.
- Google Benchmark 1.5.0
- CMake 3.18
- Python 3.8
- Jinja2 python library - used for creating plots
- Google charts forzen version 49 - used for creating plots
This repo provides 3 components:
- CMake
RunTests/benchmark_gather.cmake
used to register benchmarks programmatically via CMake. - Python
RunTests/run_tests.py
used to execute all registed benchmarks. - Python
DrawGraphs/main.py
used to create various plots for the generated benchmarks.
Include RunTests/benchmark_gather.cmake
in the top level CMakeLists.txt
file of the project and call enable_benchmarks
.
To see how to generate Visual Studio custom target to run all tests and other additional options see: enable_benchmarks()
When CMake project is generated benchmark_targets_list.txt
will be created. This file will contain list of files (separated by newline),
where each file in the list contains information needed to execute the benchmarks i.e. number of repetitions, min run time, filter for tests
(most of the parameters correspond to command line arguments for google benchmark). In benchmark_targets_list.txt
there is one file per google benchmark executable.
To register benchmarks use add_benchmark()
. For additional options see: add_benchmark(). When add_benchmark()
is called for the first time, it creates a file which will hold all properties for the given target and registers that file in benchmark_targets_list.txt
. Each next call only adds configurations to the already created config file.
To execute all registered benchmarks run RunTests/run_tests.py
. For information about command line arguments check: run_tests.py
To create plots using the result from RunTests/run_tests.py
run DrawGraphs/main.py
. For information about command line arguments check: here
This must be called from the top level CMakeLists.txt
file. It will set some utility files and variables which are going to be used by other functions. It can take the following parameters:
If defined will create custom target RUN_PERF_TESTS
which will execute all tests in TARGETS_DESCRIPTIONS_LIST_FILE
Path to file where CMake will write paths to all files with benchmark descriptions which will be run. By default it will be in the top level build directory.
Path to python executable which RUN_PERF_TESTS
will use to execute all benchmarks. By default it assumes that it's in the path environment variable and it's value is python
Must be provided in case ADD_TARGET is defined. Path to the python script (run_tests.py) which will read TARGETS_DESCRIPTIONS_LIST_FILE and execute all tests
The format in which the result from RUN_PERF_TESTS
will be generated (json, console). By default is console.
Example use:
enable_benchmarks(
ADD_TARGET
PYTHON_EXECUTOR_PATH ./run_tests.py
FORMAT "json"
)
Add functions from the a given google benchmark executable alongside with options to execute them. Calling add_benchmark() with the same target multiple times is allowed, no checks are made wheter the same benchmark function will be run multiple times. Paramters:
Mandatory. CMake target (result from add_executable) which is going to be executed.
How many times to repeat the benchmark (corresponds to --benchmark_repetitions)
Regex for benchmarks from the target. Only benchmarks matching the regex will be executed. This must follow the google benchmark rules for regular expressions. This is not mandatory, by default all benchmarks will be executed. (corresponds to --benchmark_filter)
Minimum number of seconds we should run benchmark before results are considered significant (corresponds to --benchmark_min_time)
Overwrite the unit of the benchmarks. The executor script will change the unit to the given one. Allowed units are
- s - seconds
- ms - milliseconds
- us - microseconds
- ns - nanoseconds Note: google benchmark does not provide seconds natively.
Example use.
Given the following benchmark.cpp
file
#include <benchmark/benchmark.h>
static void benchmark_heavy(benchmark::State& state) {
for (auto _ : state)
// Some code to be benchmarked
}
// Register the function as a benchmark
BENCHMARK(benchmark_heavy);
// Define another benchmark
static void benchmark(benchmark::State& state) {
for (auto _ : state)
// Some code to be benchmarked
}
BENCHMARK(benchmark);
BENCHMARK_MAIN();
The following CMakeLists.txt
will generate one executable file and will register two benchmarks which will be ran from that executable. The executable will
be ran twice and only the mathcing tests will be executed with the given parametres.
- All benchmarks with suffix
_heavy
will be executed. They will be repeated 20 times (mean, median and standard deviation will be reported), the result will be given in json format andrun_tests.py
will convertreal_time
andcpu_time
to seconds. - All benchmarks which do not end with
_heavy
will be executed. The result will be reported in json format, google benchmark will probe for at least 1 second before deciding that the results are stable
project(benchmark)
add_executable(${PROJECT_NAME} benchmark.cpp)
target_link_libraries(${PROJECT_NAME} CONAN_PKG::benchmark)
add_benchmark(
BINARY_TARGET ${PROJECT_NAME}
REPETITIONS 20
OUTPUT_FORMAT "json"
FILTER "_heavy$"
UNIT "s"
)
add_benchmark(
BINARY_TARGET ${PROJECT_NAME}
OUTPUT_FORMAT "json"
FILTER "-_heavy$"
MIN_TIME 1
)
This is a python a python script whic takes benchmark_targets_list.txt
goes trough all files in it and for each file executes all benchmark benchmarks with the
given descriptions. There are a number of command line arguments which could be given to run_tests.py
in order to override the options with wich the benchmarks
will be executed. The options are as follow:
Path tobenchmark_targets_list.txt
. By default search for benchmark_targets_list.txt
in the folder from which the script is executed.
Path to a file where the results are going to be stored. The results from all executed benchmakrs are stored in one single file. If nothing is passed the result will
be printed to the stdout
.
Format of the output one of json, console
. Console corresponds to google benchmark console output. Default is json
.
Global filter for benchmark names. First local filters (the ones specified by FILTER parameter) are applied and then the global filter is applied to the resulting benchmark names. Only the benchmarks which are matched by both regular expressions will be executed. Important this must follow standard python (UNIX) syntax for regular expressions.
All benchmarks will be runned with this number of repetitions. Overrides REPETITIONS parameter for all benchmarks.
Minimum amout of time, in seconds, which all benchmark will be run. Overrides MIN_TIME parameter for all benchmarks.
Verbosity of the console output. 0 being the most verbose and 5 being the least verbose. Default is 3.
This is a python script which is used to generate plots using the output from run_tests.py, or it can work with the json output generated by google benchmark directly. It has two modes:
- Scan a given folder and parse all
.json
files in the folder. Use the dates in the files to create plots for each benchmark. The x-axis will be the date while the y-axis will be thereal_time
returned by google benchmark. Benchmarks which are in the same fixture will be reported on the same plot. - Take a singe file as input and for each fixture create a bar plot (plotting the
real_time
returned by google benchmark).
First positional argument. This is mandatory. If --type=plot
this must be a path to a folder from which a time plot will be created, if --type=bar
it must be
a path to file (einther produced directly by google benchmark or one produced by run_tests.py)
Second positional argumend. This is mandatory. File where to write the generated plots. The plots will be generated via google charts in html
format.
One of plot, bar
. The type of charts which are going to be generated. Default is plot