google / benchmark Goto Github PK
View Code? Open in Web Editor NEWA microbenchmark support library
License: Apache License 2.0
A microbenchmark support library
License: Apache License 2.0
None of the regex checks in cmake pass when clang++ is version > 3.3 and std is set to c++14. This is due to a libc mismatch where ::gets has been removed from the global namespace.
The fix is to define -stdlib=libc++ and make sure that is installed. However, this doesn't link:
[22/22] Linking CXX executable test/re_test
FAILED: : && /usr/bin/clang++-3.5 --std=c++14 --stdlib=libc++ -Wall -Wshadow -Werror -pedantic-errors test/CMakeFiles/re_test.dir/re_test.cc.o -o test/re_test -L/home/dominic/git/benchmark/_build/third_party/src/googletest-build -rdynamic src/libbenchmark_re.a -lgtest -lgtest_main -lpthread -Wl,-rpath,/home/dominic/git/benchmark/build/third_party/src/googletest-build && :
test/CMakeFiles/re_test.dir/re_test.cc.o: In function testing::internal::PrintTo(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_ostream<char, std::__1::char_traits<char> >*)': ../test/re_test.cc:(.text._ZN7testing8internal7PrintToERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEPNS1_13basic_ostreamIcS4_EE[_ZN7testing8internal7PrintToERKNSt3__112basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEPNS1_13basic_ostreamIcS4_EE]+0x19): undefined reference to
testing::internal::PrintStringTo(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_ostream<char, std::__1::char_traits >)'
/home/dominic/git/benchmark/_build/third_party/src/googletest-build/libgtest.a(gtest-all.cc.o): In function testing::internal::BoolFromGTestEnv(char const_, bool)': /home/dominic/git/benchmark/_build/third_party/gtest/src/gtest-all.cc:(.text+0x2a): undefined reference to
std::string::c_str() const'
/home/dominic/git/benchmark/_build/third_party/gtest/src/gtest-all.cc:(.text+0x97): undefined reference tostd::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()' /home/dominic/git/benchmark/_build/third_party/gtest/src/gtest-all.cc:(.text+0xb7): undefined reference to
std::basic_string<char, std::char_traits, std::allocator >::~basic_string()'
/home/dominic/git/benchmark/_build/third_party/src/googletest-build/libgtest.a(gtest-all.cc.o): In functiontesting::internal::StringFromGTestEnv(char const_, char const_)': /home/dominic/git/benchmark/_build/third_party/gtest/src/gtest-all.cc:(.text+0x106): undefined reference to
std::string::c_str() const'
/home/dominic/git/benchmark/_build/third_party/gtest/src/gtest-all.cc:(.text+0x164): undefined reference tostd::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()' /home/dominic/git/benchmark/_build/third_party/gtest/src/gtest-all.cc:(.text+0x180): undefined reference to
std::basic_string<char, std::char_traits, std::allocator >::~basic_string()'
....
drone.io is ok, but travis CI has some rather nice features that make it easier to build a matrix of supported platforms/compilers.
Is it possible to enable appveyor to keep the Windows build from breaking? The appveyor.yml
is ready and waiting.
When doing a fresh cmake
, an error is raised.
[...]
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE
CMake Error at test/CMakeLists.txt:48 (string):
string no output variable specified
-- Configuring incomplete, errors occurred!
See also "/Users/davidcoeurjolly/local/src/benchmark/build/CMakeFiles/CMakeOutput.log".
See also "/Users/davidcoeurjolly/local/src/benchmark/build/CMakeFiles/CMakeError.log".
The output looks like something the following:
$ ./test/benchmark_test
Run on (4 X 3067 MHz CPU s
2015-03-31 17:53:04
_WARNING_ CPU scaling is enabled, the benchmark real time measurements may be noisy and will incure extra overhead.
_WARNING_ Library was built as DEBUG. Timings may be affected.
...
When doing a
BENCHMARK_TEMPLATE(BM_Constructor, std::map< char,int> );
I get macro related issues (clang5):
mytest.cpp:256:51: error:
too many arguments provided to function-like macro invocation
BENCHMARK_TEMPLATE(BM_Constructor, std::map< char,int> );
^
/usr/local/include/benchmark/benchmark.h:528:9: note: macro 'BENCHMARK_TEMPLATE' defined here
#define BENCHMARK_TEMPLATE(n, a) \
^
mytest.cpp:256:1: error: C++
requires a type specifier for all declarations
BENCHMARK_TEMPLATE(BM_Constructor, std::map< char,int> );
^~~~~~~~~~~~~~~~~~
CMake fails if Git is not installed on the host system.
There is no check if Git is present in the system before running Git to find out this project version from tags in Benchmark. See cmake/GetGitVersion.cmake.
Running CMake without Git yields:
CMake Error at cmake/GetGitVersion.cmake:28 (string):
string sub-command STRIP requires two arguments.
Call Stack (most recent call first):
CMakeLists.txt:11 (get_git_version)
CMake Error at cmake/GetGitVersion.cmake:29 (string):
string sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command.
Call Stack (most recent call first):
CMakeLists.txt:11 (get_git_version)
-- git Version:
CMake Error at CMakeLists.txt:14 (string):
string sub-command REGEX, mode MATCH needs at least 5 arguments total to
command.
-- Version:
CMake Error at CMakeLists.txt:19 (string):
string sub-command SUBSTRING requires four arguments.
Im using CMake 2.8.11 (also tested CMake 3.0.2) on Linux and Windows.
I can submit a pull request, but before I want to check with you guys that this is an issue.
Tried to run first example (master @ d750144) and got following result:
terminate called after throwing an instance of 'std::system_error'
what(): Unknown error -1
As I checked any of following lines (even when run alone) cause that problem:
benchmark::Initialize(&argc, (const char **) argv);
benchmark::RunSpecifiedBenchmarks();
Full file for reference:
#include <string>
#include "benchmark/benchmark.h"
static void BM_StringCreation(benchmark::State& state) {
while (state.KeepRunning())
std::string empty_string;
}
// Register the function as a benchmark
BENCHMARK(BM_StringCreation);
// Define another benchmark
static void BM_StringCopy(benchmark::State& state) {
std::string x = "hello";
while (state.KeepRunning())
std::string copy(x);
}
BENCHMARK(BM_StringCopy);
// Augment the main() program to invoke benchmarks if specified
// via the --benchmarks command line flag. E.g.,
// my_unittest --benchmark_filter=all
// my_unittest --benchmark_filter=BM_StringCreation
// my_unittest --benchmark_filter=String
// my_unittest --benchmark_filter='Copy|Creation'
int main(int argc, char** argv) {
benchmark::Initialize(&argc, (const char **) argv);
benchmark::RunSpecifiedBenchmarks();
return 0;
}
I've been on a mission to get https://github.com/facebook/osquery to build on Debian Wheezy. One of the issues I ran into was building benchmark, a dependency for osquery. Turns out that the libstd++ on Debian Wheezy doesn't have is_trivially_destructable
and this situation is quite tricky to work around. I found (like, literally :)) a workaround, which basically selects is_trivially_destructable
or has_trivial_destructor
based on availability.
OSQuery had a separate project to provide dependencies like benchmark and that fix has been applied there. But they've understandably moved to just using vanilla builds of benchmark to prevent unnecessary forking.
The pull request I submitted for OSQuery is here osquery/third-party#7 . Could that be added to benchmark?
When I try to build the library in the debug mode:
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make
It will be built without errors.
But when I try to build the library in the release mode:
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make VERBOSE=1
I will get the error:
[ 50%] Building CXX object externals/benchmark/src/CMakeFiles/benchmark.dir/benchmark.cc.o
cd /home/gluttton/benchmark/build/externals/benchmark/src && /usr/bin/c++ -DARCH_X86 -DHAVE_POSIX_REGEX -DOS_LINUX --std=c++11 -Wall -Wshadow -Werror -pedantic-errors --std=c++11 -Wall -Wshadow -Werror -pedantic-errors_RELEASE -fno-strict-aliasing -I/home/gluttton/benchmark/build/third_party/gtest/include -I/home/gluttton/benchmark/externals/benchmark/include -I/home/gluttton/benchmark/externals/benchmark/src -o CMakeFiles/benchmark.dir/benchmark.cc.o -c /home/gluttton/benchmark/externals/benchmark/src/benchmark.cc
c++: error: unrecognized command line option ‘-pedantic-errors_RELEASE’
This came up as part of the discussion on #29, see #29 (comment).
The title says it all.
the project does not compile on MacOs (maverick 10.9).
(as traced in the cmake output, the compiler is Clang 5.0)
src/benchmark.cc:796:35: error: variable length array of non-POD element type 'std::unique_ptr'
std::unique_ptr runners[b.threads];
$ clang --version
clang version 3.4 (trunk 193323)
Target: x86_64-unknown-linux-gnu
Thread model: posix
$ which clang
/home/tfarina/chromium/src/third_party/llvm-build/Release+Asserts/bin/clang
$ uname -a
Linux i3 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 12.04
according to standard 3.6.1.2, the signature of main
should be int main(int argc, char* argv[])
.
documentation has it right, BENCHMARK_MAIN
is wrong.
this hinders integration with other libraries.
It would be nice if benchmark can be integrated with Google Test. Is it possible? If so, it is not obvious how.
I'm not sure where best to discuss this, so I made a post on the mailing list. See https://groups.google.com/forum/#!topic/benchmark-discuss/3yqEnfqQYKQ
depends on #109
Once we have test fixtures, we can pull the KeepRunning loop into the framework and remove the boilerplate. We'll still pass in the State and have it mutable but it will be unused in many simple cases.
Currently the types used in Benchmark are inconsistent. In particular those used in the documentation vs those used in the actual code. I think we have all agreed on using size_t or int instead of int64_t.
I think we should audit the documentation and the public interface for occurrences of int64_t and remove them.
Most internal usages of int64_t can stay so long as they don't need to interact with the API types for now.
If I trying compile example from README:
gluttton@tiptop:~/Projects/benchmark$ cat main.cpp
#include <benchmark/benchmark.h>
static void BM_StringCreation(benchmark::State& state) {
while (state.KeepRunning())
std::string empty_string;
}
// Register the function as a benchmark
BENCHMARK(BM_StringCreation);
// Define another benchmark
static void BM_StringCopy(benchmark::State& state) {
std::string x = "hello";
while (state.KeepRunning())
std::string copy(x);
}
BENCHMARK(BM_StringCopy);
// Augment the main() program to invoke benchmarks if specified
// via the --benchmarks command line flag. E.g.,
// my_unittest --benchmark_filter=all
// my_unittest --benchmark_filter=BM_StringCreation
// my_unittest --benchmark_filter=String
// my_unittest --benchmark_filter='Copy|Creation'
int main(int argc, char** argv) {
benchmark::Initialize(&argc, argv);
benchmark::RunSpecifiedBenchmarks();
return 0;
}
I will get error:
user@machine:~/Projects/benchmark$ g++ main.cpp -o main -Llib -lbenchmark -lpthread -std=c++11 -Iinclude
main.cpp: In function ‘int main(int, char**)’:
main.cpp:26:40: error: invalid conversion from ‘char**’ to ‘const char**’ [-fpermissive]
benchmark::Initialize(&argc, argv);
^
In file included from main.cpp:1:0:
include/benchmark/benchmark.h:151:6: error: initializing argument 2 of ‘void benchmark::Initialize(int*, const char**)’ [-fpermissive]
void Initialize(int* argc, const char** argv);
^
System information:
user@machine:~/Projects$ uname -a
Linux tiptop 3.11.0-17-generic #30-Ubuntu SMP Fri Jan 17 22:26:34 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
user@machine:~/Projects$ gcc --version
gcc (Ubuntu/Linaro 4.8.1-10ubuntu9) 4.8.1
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Adding const
:
int main(int argc, const char** argv) {
can fix it, but I'm not sure that it's correct way.
There should be a way to invalidate a benchmark run. For example if I do a check after the benchmark loop and find that the results are incorrect then I don't want to to be mislead.
I can fake this by doing state.SetLabel("INVALID") but something like state.SetInvalid() would be preferable.
The project uses BENCHMARK_ENABLE_SHARED
to create a shared library rather than a static library. Usually the BUILD_SHARED_LIBS
global variable is used for that.
I'm not saying anything is right or wrong, just a suggestions that:
BENCHMARK_ENABLE_SHARED
then should there be a BENCHMARK_ENABLE_STATIC
(default ON
) so you can build both libraries in one build?BENCHMARK_ENABLE_SHARED
respects BUILD_SHARED_LIBS
and is set to YES
if BUILD_SHARED_LIBS=YES
Happy to put up a PR for either if you think they are decent suggestions.
I have the Project with structure like this:
. ├── CMakeLists.txt ├── externals │ ├── benchmark │ └── gmock ├── include ├── src └── test
My top level CMakeLists.txt
has following content:
cmake_minimum_required (VERSION 2.8) project (MasterProject) add_subdirectory (src) add_subdirectory (externals/benchmark) add_subdirectory (externals/gmock)
My workflow looks like and it works good:
mkdir build cd build cmake .. make
But when I add warning options in the top level CMakeLists.txt
:
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wzero-as-null-pointer-constant")
I will get error on configuration stage:
$ cmake .. -- The C compiler identification is GNU 4.8.2 -- The CXX compiler identification is GNU 4.8.2 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Looking for include file pthread.h -- Looking for include file pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - not found -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE -- Performing Test HAVE_FLAG_CXX_14 -- Performing Test HAVE_FLAG_CXX_14 - Failed -- Performing Test HAVE_FLAG_CXX_11 -- Performing Test HAVE_FLAG_CXX_11 - Success -- Performing Test HAVE_FLAG_CXX_0X -- Performing Test HAVE_FLAG_CXX_0X - Success -- Performing Test HAVE_WALL -- Performing Test HAVE_WALL - Success -- Performing Test HAVE_WSHADOW -- Performing Test HAVE_WSHADOW - Success -- Performing Test HAVE_WERROR -- Performing Test HAVE_WERROR - Success -- Performing Test HAVE_PEDANTIC_ERRORS -- Performing Test HAVE_PEDANTIC_ERRORS - Success -- Performing Test HAVE_FNO_STRICT_ALIASING -- Performing Test HAVE_FNO_STRICT_ALIASING - Success -- git Version: v0.0.0 -- Version: 0.0.0 -- Performing Test HAVE_STD_REGEX -- Performing Test HAVE_STD_REGEX -- Failed -- Performing Test HAVE_GNU_POSIX_REGEX -- Performing Test HAVE_GNU_POSIX_REGEX -- Failed -- Performing Test HAVE_POSIX_REGEX -- Performing Test HAVE_POSIX_REGEX -- Failed CMake Error at externals/benchmark/src/CMakeLists.txt:12 (message): Failed to determine the source files for the regular expression backend -- Configuring incomplete, errors occurred! See also "/home/gluttton/benchmark/build/CMakeFiles/CMakeOutput.log". See also "/home/gluttton/benchmark/build/CMakeFiles/CMakeError.log".
I get the following crash:
third_party/benchmark/src/benchmark.cc:1130: bool benchmark::State::FinishInterval(): Assertion `false' failed.
int64 GetCurrentTimeMicros() {
timespec ts;
clock_gettime(CLOCK_REALTIME, &ts);
return ts.tv_sec * 1000000 + ts.tv_nsec / 1000;
}
static void BM_GetCurrentTimeMicros(benchmark::State& state) {
while (state.KeepRunning()) {
int64 x = GetCurrentTimeMicros();
}
}
BENCHMARK(BM_GetCurrentTimeMicros);
there's some confusion between size_t and int64_t in the codebase. These should be cleaned up.
iteration counts should be size_t (unsigned and large).
Just wanted to try first example with head version of master branch (d750144):
error: cannot initialize a parameter of type 'const char **' with an lvalue of type 'char **'
benchmark::Initialize(&argc, argv);
Of course it can be easily fixed manually, but it's always better if example compiles :)
For reference I just copied example (and added proper include):
#include <string>
#include "benchmark/benchmark.h"
static void BM_StringCreation(benchmark::State& state) {
while (state.KeepRunning())
std::string empty_string;
}
// Register the function as a benchmark
BENCHMARK(BM_StringCreation);
// Define another benchmark
static void BM_StringCopy(benchmark::State& state) {
std::string x = "hello";
while (state.KeepRunning())
std::string copy(x);
}
BENCHMARK(BM_StringCopy);
// Augment the main() program to invoke benchmarks if specified
// via the --benchmarks command line flag. E.g.,
// my_unittest --benchmark_filter=all
// my_unittest --benchmark_filter=BM_StringCreation
// my_unittest --benchmark_filter=String
// my_unittest --benchmark_filter='Copy|Creation'
int main(int argc, char** argv) {
benchmark::Initialize(&argc, argv);
benchmark::RunSpecifiedBenchmarks();
return 0;
}
Hello!
On my Win7 x64
+ Qt 5.4.1(with MinGW 4.9.1)
, I found that localtime_r
and gmtime_r
are implemented as macros:
/* Recursive API emulation. */
#undef localtime_r
#define localtime_r(_Time, _Tm) ({ struct tm *___tmp_tm; \
pthread_testcancel(); \
___tmp_tm = localtime((_Time));\
if (___tmp_tm) { \
*(_Tm) = *___tmp_tm; \
___tmp_tm = (_Tm); \
} \
___tmp_tm; })
#undef gmtime_r
#define gmtime_r(_Time,_Tm) ({ struct tm *___tmp_tm; \
pthread_testcancel(); \
___tmp_tm = gmtime((_Time)); \
if (___tmp_tm) { \
*(_Tm) = *___tmp_tm; \
___tmp_tm = (_Tm); \
} \
___tmp_tm; })
so this code snippet in walltime.cc causes an error:
if (local) {
::localtime_r(&now, &timeinfo);
} else {
::gmtime_r(&now, &timeinfo);
}
Can we remove the scope operator here? Just like this(It works for me):
if (local) {
localtime_r(&now, &timeinfo);
} else {
gmtime_r(&now, &timeinfo);
}
Thanks!
The compilation with g++-5 on OS X does not success with emitting the following message
because the usage of 0 as null pointer is in OS X specific part of sysinfo.cc
,
which is illegal when -Wzero-as-null-pointer-constant
is given to the compiler.
$ cd /path/to/google/benchmark
$ mkdir build
$ cd build
$ CC=gcc-5 CXX=g++-5 cmake ..
$ make
--- snip ---
/path/to/google/benchmark/src/sysinfo.cc: In function 'void benchmark::{anonymous}::InitializeSystemInfo()':
/path/to/google/benchmark/src/sysinfo.cc:270:77: error: zero as null pointer constant [-Werror=zero-as-null-pointer-constant]
if (::sysctl(numcpus_name, arraysize(numcpus_name), &num_cpus, &size, 0, 0) ==
$ g++-5 --version
g++-5 (Homebrew gcc 5.1.0) 5.1.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
The possible fix is use nullptr
instead of 0
in the argument of sysctl
, as shown in umireon@72ce39d .
This patch works on my Yosemite environment.
The add_cxx_compiler_flag(-Wshorten-64-to-32)
modification done to solve the issue #72 gives some compilation errors while compiling with mingw64 on Windows:
g++.exe: error: unrecognized command line option '-Wshorten-64-to-32'
Details: gcc version 4.8.2 (x86_64-win32-seh, built by MinGW-W64 project)
[15:20:53| euskadi31 :~/Projects/Github/benchmark (master) e5a4319 > make
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/euskadi31/Projects/Github/benchmark
[ 5%] Performing download step (download, verify and extract) for 'googletest'
-- downloading...
src='http://googletest.googlecode.com/files/gtest-1.7.0.zip'
dst='/Users/euskadi31/Projects/Github/benchmark/third_party/src/gtest-1.7.0.zip'
timeout='none'
-- [download 0% complete]
-- [download 6% complete]
-- [download 76% complete]
-- [download 100% complete]
-- downloading... done
-- verifying file...
file='/Users/euskadi31/Projects/Github/benchmark/third_party/src/gtest-1.7.0.zip'
-- verifying file... warning: did not verify file - no URL_HASH specified?
-- extracting...
src='/Users/euskadi31/Projects/Github/benchmark/third_party/src/gtest-1.7.0.zip'
dst='/Users/euskadi31/Projects/Github/benchmark/third_party/gtest'
-- extracting... [tar xfz]
-- extracting... [analysis]
-- extracting... [rename]
-- extracting... [clean up]
-- extracting... done
[ 11%] No patch step for 'googletest'
[ 16%] No update step for 'googletest'
[ 22%] Performing configure step for 'googletest'
-- The CXX compiler identification is GNU 4.8.2
-- The C compiler identification is GNU 4.8.2
-- Checking whether CXX compiler has -isysroot
-- Checking whether CXX compiler has -isysroot - yes
-- Checking whether CXX compiler supports OSX deployment target flag
-- Checking whether CXX compiler supports OSX deployment target flag - yes
-- Check for working CXX compiler: /opt/local/bin/c++
-- Check for working CXX compiler: /opt/local/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Checking whether C compiler has -isysroot
-- Checking whether C compiler has -isysroot - yes
-- Checking whether C compiler supports OSX deployment target flag
-- Checking whether C compiler supports OSX deployment target flag - yes
-- Check for working C compiler: /opt/local/bin/gcc
-- Check for working C compiler: /opt/local/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Found PythonInterp: /opt/local/bin/python (found version "2.4.6")
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/euskadi31/Projects/Github/benchmark/third_party/src/googletest-build
[ 27%] Performing build step for 'googletest'
Scanning dependencies of target gtest
[ 50%] Building CXX object CMakeFiles/gtest.dir/src/gtest-all.cc.o
Linking CXX static library libgtest.a
[ 50%] Built target gtest
Scanning dependencies of target gtest_main
[100%] Building CXX object CMakeFiles/gtest_main.dir/src/gtest_main.cc.o
Linking CXX static library libgtest_main.a
[100%] Built target gtest_main
[ 33%] No install step for 'googletest'
[ 38%] Completed 'googletest'
[ 44%] Built target googletest
Scanning dependencies of target benchmark
[ 50%] Building CXX object src/CMakeFiles/benchmark.dir/benchmark.cc.o
/Users/euskadi31/Projects/Github/benchmark/src/benchmark.cc:372:19: error: no matching member
function for call to 'insert'
benchmarks->insert(benchmarks->end(), instances.begin(), instances.end());
~~~~~~~~~~~~^~~~~~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:718:14: note:
candidate function not viable: no known conversion from 'iterator'
(aka '__wrap_iter<pointer>') to 'size_type' (aka 'unsigned long') for 2nd argument
iterator insert(const_iterator __position, size_type __n, const_reference __x);
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:722:14: note:
candidate template ignored: disabled by 'enable_if' [with _InputIterator =
std::__1::__wrap_iter<benchmark::internal::Benchmark::Instance *>]
__is_input_iterator <_InputIterator>::value &&
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:733:13: note:
candidate template ignored: disabled by 'enable_if' [with _ForwardIterator =
std::__1::__wrap_iter<benchmark::internal::Benchmark::Instance *>]
__is_forward_iterator<_ForwardIterator>::value &&
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:710:14: note:
candidate function not viable: requires 2 arguments, but 3 were provided
iterator insert(const_iterator __position, const_reference __x);
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:712:14: note:
candidate function not viable: requires 2 arguments, but 3 were provided
iterator insert(const_iterator __position, value_type&& __x);
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:742:14: note:
candidate function not viable: requires 2 arguments, but 3 were provided
iterator insert(const_iterator __position, initializer_list<value_type> __il)
^
/Users/euskadi31/Projects/Github/benchmark/src/benchmark.cc:376:21: error: no matching member
function for call to 'insert'
benchmarks->insert(benchmarks->end(), instances.begin(),
~~~~~~~~~~~~^~~~~~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:718:14: note:
candidate function not viable: no known conversion from 'iterator'
(aka '__wrap_iter<pointer>') to 'size_type' (aka 'unsigned long') for 2nd argument
iterator insert(const_iterator __position, size_type __n, const_reference __x);
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:722:14: note:
candidate template ignored: disabled by 'enable_if' [with _InputIterator =
std::__1::__wrap_iter<benchmark::internal::Benchmark::Instance *>]
__is_input_iterator <_InputIterator>::value &&
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:733:13: note:
candidate template ignored: disabled by 'enable_if' [with _ForwardIterator =
std::__1::__wrap_iter<benchmark::internal::Benchmark::Instance *>]
__is_forward_iterator<_ForwardIterator>::value &&
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:710:14: note:
candidate function not viable: requires 2 arguments, but 3 were provided
iterator insert(const_iterator __position, const_reference __x);
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:712:14: note:
candidate function not viable: requires 2 arguments, but 3 were provided
iterator insert(const_iterator __position, value_type&& __x);
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:742:14: note:
candidate function not viable: requires 2 arguments, but 3 were provided
iterator insert(const_iterator __position, initializer_list<value_type> __il)
^
/Users/euskadi31/Projects/Github/benchmark/src/benchmark.cc:383:23: error: no matching member
function for call to 'insert'
benchmarks->insert(benchmarks->end(), instances.begin(),
~~~~~~~~~~~~^~~~~~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:718:14: note:
candidate function not viable: no known conversion from 'iterator'
(aka '__wrap_iter<pointer>') to 'size_type' (aka 'unsigned long') for 2nd argument
iterator insert(const_iterator __position, size_type __n, const_reference __x);
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:722:14: note:
candidate template ignored: disabled by 'enable_if' [with _InputIterator =
std::__1::__wrap_iter<benchmark::internal::Benchmark::Instance *>]
__is_input_iterator <_InputIterator>::value &&
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:733:13: note:
candidate template ignored: disabled by 'enable_if' [with _ForwardIterator =
std::__1::__wrap_iter<benchmark::internal::Benchmark::Instance *>]
__is_forward_iterator<_ForwardIterator>::value &&
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:710:14: note:
candidate function not viable: requires 2 arguments, but 3 were provided
iterator insert(const_iterator __position, const_reference __x);
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:712:14: note:
candidate function not viable: requires 2 arguments, but 3 were provided
iterator insert(const_iterator __position, value_type&& __x);
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1/vector:742:14: note:
candidate function not viable: requires 2 arguments, but 3 were provided
iterator insert(const_iterator __position, initializer_list<value_type> __il)
^
3 errors generated.
make[2]: *** [src/CMakeFiles/benchmark.dir/benchmark.cc.o] Error 1
make[1]: *** [src/CMakeFiles/benchmark.dir/all] Error 2
make: *** [all] Error 2
Hi, thanks for this awesome library! Unfortunatelly, I ran into a small issue with it. I tried to build it on freebsd 10.0 RELEASE and build failed. Compiler reported that header <sys/sysctl.h> has undefiened type u_int. The problem is that <sys/types.h> header should be included before it, cause this type is defined there. Can you please fix it, if you support freebsd?
I encountered a problem when using this library in linux with gcc 4.9.1. If pthread is not linked against then a test crashes:
terminate called after throwing an instance of 'std::system_error'
what(): Unknown error -1
Program received signal SIGABRT, Aborted.
0x00007ffff7241967 in raise () from /usr/lib/libc.so.6
(gdb) where
#0 0x00007ffff7241967 in raise () from /usr/lib/libc.so.6
#1 0x00007ffff7242d3a in abort () from /usr/lib/libc.so.6
#2 0x00007ffff7b2d085 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6
#3 0x00007ffff7b2af06 in __cxxabiv1::__terminate(void ()()) () from /usr/lib/libstdc++.so.6
#4 0x00007ffff7b2af51 in std::terminate() () from /usr/lib/libstdc++.so.6
#5 0x00007ffff7b2b168 in __cxa_throw () from /usr/lib/libstdc++.so.6
#6 0x00007ffff7b852ff in std::throw_system_error(int) () from /usr/lib/libstdc++.so.6
#7 0x000000000047dc88 in void std::call_once<void (&)()>(std::once_flag&, void (&)()) ()
#8 0x000000000047db20 in benchmark::CyclesPerSecond() ()
#9 0x000000000047dfce in benchmark::walltime::Initialize() ()
#10 0x0000000000472888 in benchmark::Initialize(int, char const) ()
This seems to be gcc/std library problem but it is probably a good idea to mention this in readme.
Hi Dominic,
I am hoping to put a copy of "benchmark" within the LLVM repository soon. It would be nice if Benchmark reached a state that we would be happy to call a
What do you think "Benchmark" needs to do before we could release version 1.0?
From benchmark_api.h
:
// For more complex patterns of inputs, passing a custom function
// to Apply allows programmatic specification of an
// arbitrary set of arguments to run the microbenchmark on.
// The following example enumerates a dense range on
// one parameter, and a sparse range on the second.
static benchmark::internal::Benchmark* CustomArguments(
benchmark::internal::Benchmark* b) {
for (int i = 0; i <= 10; ++i)
for (int j = 32; j <= 1024*1024; j *= 8)
b = b->ArgPair(i, j);
return b;
}
BENCHMARK(BM_SetInsert)->Apply(CustomArguments);
Benchmark::Apply() takes a function that returns void, so this sample code doesn't compile.
I have created a simple test:
#include <benchmark/benchmark.h>
static void BM_Test (benchmark::State & state)
{
std::string x (1024, 'x');
while (state.KeepRunning () ) {
std::string copy (x);
}
}
BENCHMARK (BM_Test);
int main (int argc, const char ** argv) {
benchmark::Initialize (& argc, argv);
benchmark::RunSpecifiedBenchmarks ();
return 0;
}
And now I try to launch it with specified number of iterations and repetitions:
$ ./test --benchmark_iterations=10000
Reading /proc/self/cputime_ns failed. Using getrusage().
Benchmarking on 8 X 2401 MHz CPUs
2014/10/04-11:12:01
CPU scaling is enabled: Benchmark timings may be noisy.
DEBUG: Benchmark Time(ns) CPU(ns) Iterations
--------------------------------------------------
DEBUG: BM_Test 15 0 10000
$ ./test --benchmark_repetitions=5
Reading /proc/self/cputime_ns failed. Using getrusage().
Benchmarking on 8 X 2401 MHz CPUs
2014/10/04-11:12:48
CPU scaling is enabled: Benchmark timings may be noisy.
DEBUG: Benchmark Time(ns) CPU(ns) Iterations
------------------------------------------------------
DEBUG: BM_Test 7 27 3742286
DEBUG: BM_Test 8 27 3735508
DEBUG: BM_Test 6 26 3843937
DEBUG: BM_Test 6 26 3875998
DEBUG: BM_Test 6 26 3853809
DEBUG: BM_Test_mean 7 26 3811229
DEBUG: BM_Test_stddev 1 0 59109
$ ./test --benchmark_repetitions=5 --benchmark_iterations=10000
or
$ ./test --benchmark_iterations=10000 --benchmark_repetitions=5
only number of iterations can be specified:
Reading /proc/self/cputime_ns failed. Using getrusage().
Benchmarking on 8 X 2401 MHz CPUs
2014/10/04-11:27:29
CPU scaling is enabled: Benchmark timings may be noisy.
DEBUG: Benchmark Time(ns) CPU(ns) Iterations
------------------------------------------------------
DEBUG: BM_Test 64 80 10000
When looking into #17, Valgrind turned up a race between thread 0 destroying test_vector
in BM_SetupTeardown
and the other threads finishing the while (state.KeepRunning()) { ... }
loop.
This race is also found after compiling with Clang 3.3 with -fsanitize=thread
:
==================
WARNING: ThreadSanitizer: data race (pid=8083)
Read of size 8 at 0x7d0800002fe8 by thread T35:
#0 ~vector /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/bits/stl_vector.h:415 (exe+0x00000006cfa0)
#1 ~vector /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/bits/stl_vector.h:415 (exe+0x00000006cf30)
#2 BM_SetupTeardown(benchmark::State&) /home/ckennelly/projects/benchmark/test/benchmark_test.cc:131 (exe+0x00000006b331)
#3 std::_Function_handler<void (benchmark::State&), void (*)(benchmark::State&)>::_M_invoke(std::_Any_data const&, benchmark::State&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2072 (exe+0x00000006f770)
#4 std::function<void (benchmark::State&)>::operator()(benchmark::State&) const /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2464 (exe+0x000000081de8)
#5 benchmark::State::Run() /home/ckennelly/projects/benchmark/src/benchmark.cc:1142 (exe+0x00000007a832)
#6 benchmark::State::RunWrapper(void*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1159 (exe+0x00000007c916)
Previous write of size 8 at 0x7d0800002fe8 by thread T37 (mutexes: write M104):
#0 std::vector<int, std::allocator<int> >::push_back(int const&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/bits/stl_vector.h:907 (exe+0x00000006ceaf)
#1 BM_SetupTeardown(benchmark::State&) /home/ckennelly/projects/benchmark/test/benchmark_test.cc:124 (exe+0x00000006b287)
#2 std::_Function_handler<void (benchmark::State&), void (*)(benchmark::State&)>::_M_invoke(std::_Any_data const&, benchmark::State&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2072 (exe+0x00000006f770)
#3 std::function<void (benchmark::State&)>::operator()(benchmark::State&) const /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2464 (exe+0x000000081de8)
#4 benchmark::State::Run() /home/ckennelly/projects/benchmark/src/benchmark.cc:1142 (exe+0x00000007a832)
#5 benchmark::State::RunWrapper(void*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1159 (exe+0x00000007c916)
Location is heap block of size 24 at 0x7d0800002fe0 allocated by thread T35:
#0 operator new(unsigned long) ??:0 (exe+0x000000038ab6)
#1 BM_SetupTeardown(benchmark::State&) /home/ckennelly/projects/benchmark/test/benchmark_test.cc:118 (exe+0x00000006b185)
#2 std::_Function_handler<void (benchmark::State&), void (*)(benchmark::State&)>::_M_invoke(std::_Any_data const&, benchmark::State&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2072 (exe+0x00000006f770)
#3 std::function<void (benchmark::State&)>::operator()(benchmark::State&) const /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2464 (exe+0x000000081de8)
#4 benchmark::State::Run() /home/ckennelly/projects/benchmark/src/benchmark.cc:1142 (exe+0x00000007a832)
#5 benchmark::State::RunWrapper(void*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1159 (exe+0x00000007c916)
Mutex M104 created at:
#0 pthread_mutex_init ??:0 (exe+0x00000003bdc5)
#1 BM_SetupTeardown(benchmark::State&) /home/ckennelly/projects/benchmark/test/benchmark_test.cc:116 (exe+0x00000006b178)
#2 std::_Function_handler<void (benchmark::State&), void (*)(benchmark::State&)>::_M_invoke(std::_Any_data const&, benchmark::State&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2072 (exe+0x00000006f770)
#3 std::function<void (benchmark::State&)>::operator()(benchmark::State&) const /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2464 (exe+0x000000081de8)
#4 benchmark::State::Run() /home/ckennelly/projects/benchmark/src/benchmark.cc:1142 (exe+0x00000007a832)
#5 benchmark::State::RunWrapper(void*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1159 (exe+0x00000007c916)
Thread T35 (tid=8207, running) created by main thread at:
#0 pthread_create ??:0 (exe+0x00000003b812)
#1 benchmark::State::RunAsThread() /home/ckennelly/projects/benchmark/src/benchmark.cc:1150 (exe+0x00000007a736)
#2 benchmark::internal::Benchmark::RunInstance(benchmark::internal::Benchmark::Instance const&, benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:855 (exe+0x00000007a06d)
#3 benchmark::internal::RunMatchingBenchmarks(std::string const&, benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1201 (exe+0x00000007d065)
#4 benchmark::RunSpecifiedBenchmarks(benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1221 (exe+0x00000007da9a)
#5 main /home/ckennelly/projects/benchmark/test/benchmark_test.cc:152 (exe+0x00000006b5dc)
Thread T37 (tid=8209, running) created by main thread at:
#0 pthread_create ??:0 (exe+0x00000003b812)
#1 benchmark::State::RunAsThread() /home/ckennelly/projects/benchmark/src/benchmark.cc:1150 (exe+0x00000007a736)
#2 benchmark::internal::Benchmark::RunInstance(benchmark::internal::Benchmark::Instance const&, benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:855 (exe+0x00000007a06d)
#3 benchmark::internal::RunMatchingBenchmarks(std::string const&, benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1201 (exe+0x00000007d065)
#4 benchmark::RunSpecifiedBenchmarks(benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1221 (exe+0x00000007da9a)
#5 main /home/ckennelly/projects/benchmark/test/benchmark_test.cc:152 (exe+0x00000006b5dc)
SUMMARY: ThreadSanitizer: data race /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/bits/stl_vector.h:415 ~vector
==================
==================
WARNING: ThreadSanitizer: data race (pid=8083)
Write of size 8 at 0x7d040000dff0 by thread T35:
#0 operator delete(void*) ??:0 (exe+0x000000038f4c)
#1 __gnu_cxx::new_allocator<int>::deallocate(int*, unsigned long) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/ext/new_allocator.h:110 (exe+0x00000006d49f)
#2 std::_Vector_base<int, std::allocator<int> >::_M_deallocate(int*, unsigned long) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/bits/stl_vector.h:174 (exe+0x00000006d2d2)
#3 ~_Vector_base /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/bits/stl_vector.h:160 (exe+0x00000006d1a9)
#4 ~vector /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/bits/stl_vector.h:416 (exe+0x00000006cfd4)
#5 ~vector /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/bits/stl_vector.h:415 (exe+0x00000006cf30)
#6 BM_SetupTeardown(benchmark::State&) /home/ckennelly/projects/benchmark/test/benchmark_test.cc:131 (exe+0x00000006b331)
#7 std::_Function_handler<void (benchmark::State&), void (*)(benchmark::State&)>::_M_invoke(std::_Any_data const&, benchmark::State&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2072 (exe+0x00000006f770)
#8 std::function<void (benchmark::State&)>::operator()(benchmark::State&) const /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2464 (exe+0x000000081de8)
#9 benchmark::State::Run() /home/ckennelly/projects/benchmark/src/benchmark.cc:1142 (exe+0x00000007a832)
#10 benchmark::State::RunWrapper(void*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1159 (exe+0x00000007c916)
Previous write of size 4 at 0x7d040000dff4 by thread T37 (mutexes: write M104):
#0 void __gnu_cxx::new_allocator<int>::construct<int, int const&>(int*, int const&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/ext/new_allocator.h:120 (exe+0x00000006f179)
#1 _ZNSt16allocator_traitsISaIiEE12_S_constructIiJRKiEEENSt9enable_ifIXsr18__construct_helperIT_DpT0_EE5valueEvE4typeERS0_PS6_DpOS7_ /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/bits/alloc_traits.h:254 (exe+0x00000006f098)
#2 _ZNSt16allocator_traitsISaIiEE9constructIiJRKiEEEDTcl12_S_constructfp_fp0_spclsr3stdE7forwardIT0_Efp1_EEERS0_PT_DpOS5_ /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/bits/alloc_traits.h:393 (exe+0x00000006d628)
#3 std::vector<int, std::allocator<int> >::push_back(int const&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/bits/stl_vector.h:905 (exe+0x00000006ce89)
#4 BM_SetupTeardown(benchmark::State&) /home/ckennelly/projects/benchmark/test/benchmark_test.cc:124 (exe+0x00000006b287)
#5 std::_Function_handler<void (benchmark::State&), void (*)(benchmark::State&)>::_M_invoke(std::_Any_data const&, benchmark::State&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2072 (exe+0x00000006f770)
#6 std::function<void (benchmark::State&)>::operator()(benchmark::State&) const /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2464 (exe+0x000000081de8)
#7 benchmark::State::Run() /home/ckennelly/projects/benchmark/src/benchmark.cc:1142 (exe+0x00000007a832)
#8 benchmark::State::RunWrapper(void*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1159 (exe+0x00000007c916)
Location is heap block of size 16 at 0x7d040000dff0 allocated by thread T37:
#0 operator new(unsigned long) ??:0 (exe+0x000000038ab6)
#1 __gnu_cxx::new_allocator<int>::allocate(unsigned long, void const*) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/ext/new_allocator.h:104 (exe+0x00000006ecea)
#2 std::_Vector_base<int, std::allocator<int> >::_M_allocate(unsigned long) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/bits/stl_vector.h:168 (exe+0x00000006de4c)
#3 void std::vector<int, std::allocator<int> >::_M_emplace_back_aux<int const&>(int const&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/bits/vector.tcc:404 (exe+0x00000006d6e5)
#4 std::vector<int, std::allocator<int> >::push_back(int const&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/bits/stl_vector.h:911 (exe+0x00000006ced8)
#5 BM_SetupTeardown(benchmark::State&) /home/ckennelly/projects/benchmark/test/benchmark_test.cc:124 (exe+0x00000006b287)
#6 std::_Function_handler<void (benchmark::State&), void (*)(benchmark::State&)>::_M_invoke(std::_Any_data const&, benchmark::State&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2072 (exe+0x00000006f770)
#7 std::function<void (benchmark::State&)>::operator()(benchmark::State&) const /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2464 (exe+0x000000081de8)
#8 benchmark::State::Run() /home/ckennelly/projects/benchmark/src/benchmark.cc:1142 (exe+0x00000007a832)
#9 benchmark::State::RunWrapper(void*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1159 (exe+0x00000007c916)
Mutex M104 created at:
#0 pthread_mutex_init ??:0 (exe+0x00000003bdc5)
#1 BM_SetupTeardown(benchmark::State&) /home/ckennelly/projects/benchmark/test/benchmark_test.cc:116 (exe+0x00000006b178)
#2 std::_Function_handler<void (benchmark::State&), void (*)(benchmark::State&)>::_M_invoke(std::_Any_data const&, benchmark::State&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2072 (exe+0x00000006f770)
#3 std::function<void (benchmark::State&)>::operator()(benchmark::State&) const /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2464 (exe+0x000000081de8)
#4 benchmark::State::Run() /home/ckennelly/projects/benchmark/src/benchmark.cc:1142 (exe+0x00000007a832)
#5 benchmark::State::RunWrapper(void*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1159 (exe+0x00000007c916)
Thread T35 (tid=8207, running) created by main thread at:
#0 pthread_create ??:0 (exe+0x00000003b812)
#1 benchmark::State::RunAsThread() /home/ckennelly/projects/benchmark/src/benchmark.cc:1150 (exe+0x00000007a736)
#2 benchmark::internal::Benchmark::RunInstance(benchmark::internal::Benchmark::Instance const&, benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:855 (exe+0x00000007a06d)
#3 benchmark::internal::RunMatchingBenchmarks(std::string const&, benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1201 (exe+0x00000007d065)
#4 benchmark::RunSpecifiedBenchmarks(benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1221 (exe+0x00000007da9a)
#5 main /home/ckennelly/projects/benchmark/test/benchmark_test.cc:152 (exe+0x00000006b5dc)
Thread T37 (tid=8209, finished) created by main thread at:
#0 pthread_create ??:0 (exe+0x00000003b812)
#1 benchmark::State::RunAsThread() /home/ckennelly/projects/benchmark/src/benchmark.cc:1150 (exe+0x00000007a736)
#2 benchmark::internal::Benchmark::RunInstance(benchmark::internal::Benchmark::Instance const&, benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:855 (exe+0x00000007a06d)
#3 benchmark::internal::RunMatchingBenchmarks(std::string const&, benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1201 (exe+0x00000007d065)
#4 benchmark::RunSpecifiedBenchmarks(benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1221 (exe+0x00000007da9a)
#5 main /home/ckennelly/projects/benchmark/test/benchmark_test.cc:152 (exe+0x00000006b5dc)
SUMMARY: ThreadSanitizer: data race ??:0 operator delete(void*)
==================
==================
WARNING: ThreadSanitizer: data race (pid=8083)
Write of size 8 at 0x7d0800002ff0 by thread T35:
#0 operator delete(void*) ??:0 (exe+0x000000038f4c)
#1 BM_SetupTeardown(benchmark::State&) /home/ckennelly/projects/benchmark/test/benchmark_test.cc:131 (exe+0x00000006b33a)
#2 std::_Function_handler<void (benchmark::State&), void (*)(benchmark::State&)>::_M_invoke(std::_Any_data const&, benchmark::State&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2072 (exe+0x00000006f770)
#3 std::function<void (benchmark::State&)>::operator()(benchmark::State&) const /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2464 (exe+0x000000081de8)
#4 benchmark::State::Run() /home/ckennelly/projects/benchmark/src/benchmark.cc:1142 (exe+0x00000007a832)
#5 benchmark::State::RunWrapper(void*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1159 (exe+0x00000007c916)
Previous read of size 8 at 0x7d0800002ff0 by thread T37 (mutexes: write M104):
#0 std::vector<int, std::allocator<int> >::push_back(int const&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/bits/stl_vector.h:903 (exe+0x00000006ce34)
#1 BM_SetupTeardown(benchmark::State&) /home/ckennelly/projects/benchmark/test/benchmark_test.cc:124 (exe+0x00000006b287)
#2 std::_Function_handler<void (benchmark::State&), void (*)(benchmark::State&)>::_M_invoke(std::_Any_data const&, benchmark::State&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2072 (exe+0x00000006f770)
#3 std::function<void (benchmark::State&)>::operator()(benchmark::State&) const /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2464 (exe+0x000000081de8)
#4 benchmark::State::Run() /home/ckennelly/projects/benchmark/src/benchmark.cc:1142 (exe+0x00000007a832)
#5 benchmark::State::RunWrapper(void*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1159 (exe+0x00000007c916)
Location is heap block of size 24 at 0x7d0800002fe0 allocated by thread T35:
#0 operator new(unsigned long) ??:0 (exe+0x000000038ab6)
#1 BM_SetupTeardown(benchmark::State&) /home/ckennelly/projects/benchmark/test/benchmark_test.cc:118 (exe+0x00000006b185)
#2 std::_Function_handler<void (benchmark::State&), void (*)(benchmark::State&)>::_M_invoke(std::_Any_data const&, benchmark::State&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2072 (exe+0x00000006f770)
#3 std::function<void (benchmark::State&)>::operator()(benchmark::State&) const /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2464 (exe+0x000000081de8)
#4 benchmark::State::Run() /home/ckennelly/projects/benchmark/src/benchmark.cc:1142 (exe+0x00000007a832)
#5 benchmark::State::RunWrapper(void*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1159 (exe+0x00000007c916)
Mutex M104 created at:
#0 pthread_mutex_init ??:0 (exe+0x00000003bdc5)
#1 BM_SetupTeardown(benchmark::State&) /home/ckennelly/projects/benchmark/test/benchmark_test.cc:116 (exe+0x00000006b178)
#2 std::_Function_handler<void (benchmark::State&), void (*)(benchmark::State&)>::_M_invoke(std::_Any_data const&, benchmark::State&) /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2072 (exe+0x00000006f770)
#3 std::function<void (benchmark::State&)>::operator()(benchmark::State&) const /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.2/include/g++-v4/functional:2464 (exe+0x000000081de8)
#4 benchmark::State::Run() /home/ckennelly/projects/benchmark/src/benchmark.cc:1142 (exe+0x00000007a832)
#5 benchmark::State::RunWrapper(void*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1159 (exe+0x00000007c916)
Thread T35 (tid=8207, running) created by main thread at:
#0 pthread_create ??:0 (exe+0x00000003b812)
#1 benchmark::State::RunAsThread() /home/ckennelly/projects/benchmark/src/benchmark.cc:1150 (exe+0x00000007a736)
#2 benchmark::internal::Benchmark::RunInstance(benchmark::internal::Benchmark::Instance const&, benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:855 (exe+0x00000007a06d)
#3 benchmark::internal::RunMatchingBenchmarks(std::string const&, benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1201 (exe+0x00000007d065)
#4 benchmark::RunSpecifiedBenchmarks(benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1221 (exe+0x00000007da9a)
#5 main /home/ckennelly/projects/benchmark/test/benchmark_test.cc:152 (exe+0x00000006b5dc)
Thread T37 (tid=8209, finished) created by main thread at:
#0 pthread_create ??:0 (exe+0x00000003b812)
#1 benchmark::State::RunAsThread() /home/ckennelly/projects/benchmark/src/benchmark.cc:1150 (exe+0x00000007a736)
#2 benchmark::internal::Benchmark::RunInstance(benchmark::internal::Benchmark::Instance const&, benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:855 (exe+0x00000007a06d)
#3 benchmark::internal::RunMatchingBenchmarks(std::string const&, benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1201 (exe+0x00000007d065)
#4 benchmark::RunSpecifiedBenchmarks(benchmark::BenchmarkReporter const*) /home/ckennelly/projects/benchmark/src/benchmark.cc:1221 (exe+0x00000007da9a)
#5 main /home/ckennelly/projects/benchmark/test/benchmark_test.cc:152 (exe+0x00000006b5dc)
SUMMARY: ThreadSanitizer: data race ??:0 operator delete(void*)
==================
The current design has an implicit setup/teardown in each benchmark function with the assumption that microbenchmarks shouldn't care about complicated setup. However, there are some cases where test fixtures would help coverage. Specifically benchmarking containers with non-integer value types and a variety of key values.
Given a simple foo_test.cc file that uses both Google Test and google/benchmark, there should be a way to run only the benchmarks without also running the unit tests. When using both googletest and google/benchmarks, a typical main() function might look like the following:
int main(int argc, char* argv[]) {
testing::InitGoogleTest(&argc, argv);
benchmark::Initialize(&argc, argv);
benchmark::RunSpecifiedBenchmarks();
return RUN_ALL_TESTS();
}
Which will initialize googletest, then the benchmark library, then it will need to run both of them. However, the result of the above code is that the benchmarks will be run (if specified via --benchmark_filter), and then the unit tests will still be run.
More desirable behavior would be enable the ability to run only the benchmarks without also running the unit tests. Actually, the existing documentation for benchmark::RunSpecifiedBenchmarks()
indicates that it should exit after running the benchmarks, but this does not happen. That documentation does not match the implementation.
...
// Otherwise, run all benchmarks specified by the --benchmark_filter flag,
// and exit after running the benchmarks.
void RunSpecifiedBenchmarks();
void RunSpecifiedBenchmarks(BenchmarkReporter* reporter);
...
RunSpecifiedBenchmarks()
to exit IFF it ran any benchmarks. This would then match the existing documentation and it would make the example main()
function I included up top work.RunSpecifiedBenchmarks()
to return some indicator as to whether or not it ran some benchmarks. This way, the code in main could then conditionally call Google Test's RUN_ALL_TESTS() function. The return indicator could be bool (true=benchmarks were run), or an int indicating how many benchmarks were run (maybe 0).I had encountered a reproducible crash issue.
My testing environment is mac osx 10.9.5 with xcode 6 and clang-600.0.5.
Here is the test code (copied from README):
#include "benchmark/benchmark.h"
static void BM_StringCreation(benchmark::State& state) {
while (state.KeepRunning())
std::string empty_string;
}
// Register the function as a benchmark
BENCHMARK(BM_StringCreation);
// Define another benchmark
static void BM_StringCopy(benchmark::State& state) {
std::string x = "hello";
while (state.KeepRunning())
std::string copy(x);
}
BENCHMARK(BM_StringCopy);
int main(int argc, const char** argv) {
// BENCHMARK(BM_StringCreation);
// BENCHMARK(BM_StringCopy);
benchmark::Initialize(&argc, argv);
benchmark::RunSpecifiedBenchmarks();
return 0;
}
At the end of the execution for everytime, a crash message shows up:
Reading /proc/self/cputime_ns failed. Using getrusage().
Benchmarking on 4 X 1000 MHz CPUs
2014/10/05-11:41:53
DEBUG: Benchmark Time(ns) CPU(ns) Iterations
---------------------------------------------------------
DEBUG: BM_StringCreation 14 28 18115649
DEBUG: BM_StringCopy 9 23 22298993
libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument
Abort trap: 6
When I move BENCHMARK()
lines into main()
, the issue disappeared.
I debugged this issue for a few hours, and then I realized that the problem is: the static std::mutex benchmark_mutex
has been destroyed before static BenchmarkFamilies instance
. The destructor of Benchmark
calls BenchmarkFamilies::RemoveBenchmark()
, but the mutex already invalid, and then it just crashes.
Some part of code is Windows-aware(ifdef'ed with OS_WINDOWS), but still incomplete(e.g. pthread.h in benchmark.h)
Hello,
I tried to use the function to use realtime instead of CPU time:
// If a particular benchmark is I/O bound, or if for some reason CPU
// timings are not representative, call this method from within the
// benchmark routine. If called, the elapsed time will be used to
// control how many iterations are run, and in the printing of
// items/second or MB/seconds values. If not called, the cpu time
// used by the benchmark will be used.
void UseRealTime();
like this:
static void BM_foo(benchmark::State& state) {
benchmark::UseRealTime();
while (state.KeepRunning()) {
// Stuff
}
}
But this won't link.
The function is actually defined in the benchmark::internal
namespace. Simply moving it to the benchmark
namespace seems to work (at least to compile).
Is this function supposed to be used like this ?
Also the function void SetLabel(const std::string& label);
(the free function, not the State
member) is not defined at all.
Cheers,
I've been trying to compile benchmark on Android, however there's some sort of issue with cycleclock.h.
"In file included from jni/../src/sysinfo.cc:33:0:
jni/../src/cycleclock.h:127:2: error: #error You need to define CycleTimer for your OS and CPU"
If I basically copy the instruction >= ARM 6 I get an illegal instruction exception while running on the emulator... Could you provide some advice?
valgrind reports some memory leaks inside of google benchmark. This should probably be fixed.
It looks like walltime can be simplified/replaced and made portable by using std::chrono::steady_clock
It would be useful to have a CSV BenchmarkReporter. An option such as --reporter=csv makes sense to me.
I'm pretty sure this is new: negative times for 'real' time.
Note, the below is release build.
$ test/basic_test
Run on (4 X 775 MHz CPUs)
2015/03/12-21:22:09
***WARNING*** CPU scaling is enabled, the benchmark timings may be noisy
Benchmark Time(ns) CPU(ns) Iterations
---------------------------------------------------------------------------------------
BM_empty 1 2 360765235
BM_empty/threads:4 3 3 221043324
BM_spin_empty/8 -4 4 194933402
BM_spin_empty/512 552 172 4084371
BM_spin_empty/8k -7250 2660 265357
BM_spin_empty/8/threads:4 7 9 79947008
BM_spin_empty/512/threads:4 326 396 1797916
BM_spin_empty/8k/threads:4 5087 6155 113608
BM_spin_pause_before/8 -10 4 194293327
BM_spin_pause_before/512 554 172 4085134
BM_spin_pause_before/8k -7281 2655 265210
BM_spin_pause_before/8/threads:4 7 9 80895864
BM_spin_pause_before/512/threads:4 334 388 1841124
it seems to be related to ranges and i see it in a range of different benchmarks.
from @mattyclarkson: there is a guide to pull the version number from the git
tags. Then you can just tag the repo at v0.0.1
and CMake would sort itself out.
Since benchmark/macros.h
defines CHECK_XXX
, it prevents to use both glog/logging.h
and benchmark/benchmark.h
in the same file.
We currently print 'DEBUG' at the start of every line of output when built in debug mode. This is noisy and the same warning can be captured with a header stanza instead.
There are a number of warnings that get treated as errors if compiling with -Werror if the platform being compiled on is not 64-bit (at least, I assume this is the source of the warnings; they appear on iOS, but not any desktop platform I've used so far) — for instance, iterations_ is an int64_t but iterations() returns an int.
Acceptable solutions would be to either change the internal counter to "int" or change the API to "int64_t" in all cases.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.