cpp-testing/ut-benchmark

Benchmarks - Unit Testing Frameworks

Benchmarks - Unit Testing Frameworks

Framework	Version	Standard	License	Linkage	Test configuration
Boost.Test	1.71.0	C++03	Boost 1.0	single header/library	static library
GoogleTest	1.10.0	C++11	BSD-3	library	static library
Catch	2.10.2	C++11	Boost 1.0	single header	CATCH_CONFIG_FAST_COMPILE
Doctest	2.3.5	C++11	MIT	single header	DOCTEST_CONFIG_SUPER_FAST_ASSERTS
μt	1.1.0	C++20	Boost 1.0	single header/module

Include / 0 tests, 0 asserts, 1 cpp file
Assert / 1 test, 1'000'000 asserts, 1 cpp file
Test / 1'000 tests, 0 asserts, 1 cpp file
Suite / 10'000 tests, 0 asserts, 100 cpp files
Suite+Assert / 10'000 tests, 40'000 asserts, 100 cpp files
Suite+Assert+STL / 10'000 tests, 20'000 asserts, 100 cpp files
Incremental Build - Suite+Assert+STL / 1 cpp file change (1'000 tests, 20'000 asserts, 100 cpp files)
Suite+Assert+STL / 10'000 tests, 20'000 asserts, 100 cpp files (Headers vs Precompiled headers vs C++20 Modules)

---

To run benchmarks

cd benchmarks && ./bench.sh

ut-benchmark's Issues

Difference in measurements between googletest and the rest of these libraries

Catch2 and doctest are "single header" but they are effectively a .cpp file and a .h file clumped together. I see that ut has also introduced the concept of BOOST_UT_IMPLEMENTATION but it currently provides much less functionality compared to Catch2 and doctest. In order to make the measurements completely fair one would need to either include the compilation of the googletest static library itself (which is obviously weird), or make static libraries for the implementations (test runners) of Catch2/doctest (and ut) and link against them (and when measuring time the compilation of the static library should be excluded - only the other .cpp file + linking should be considered). I'd argue that all 4 libraries should go through the process of linking a static library to make things equal.

doctest even provides the 2 parts (.cpp and a .h from which the final header is assembled) as separate files: https://github.com/onqtam/doctest/tree/master/doctest/parts

It would be cool if you could incorporate the use of FAST_CHECK_EQ(a, b) in a separate additional benchmark for doctest instead of the tradicional CHECK(a == b), but this is irrelevant to the current issue.