Use jmh for benchmarking

[vlsi]

Use jmh for benchmarking

[vladmihalcea]

Thanks for the tip. System.currentTimeMillis() is indeed bad, and nanoTime() is not monotonical and it has a varying precision between 10-1000 nanoseconds. For low latency benchmarks, nanoTime() is not very helpful, especially if the error keeps on accumulating (like measuring inside a loop).

I've been also using Codahale/Dropwizard metrics and they have two types of Clocks:

• the default one is based on System.nanoTime()
• the second one uses the CPU-time ManagementFactory.getThreadMXBean().getCurrentThreadCpuTime().

But when it comes to a RDBMS, we are often talking about millisecond precision. Most test results are in milliseconds and even if we assume that nanoTime has a one microsecond precision, it shouldn't affect the test results as I need to round the results anyway. In one test, I think I showed the results rounded to 10 microseconds though.

[vlsi]

1. The thing is JMH forces you into well-structured, understandable benchmarks.
   That is a big plus since that enables review of the benchmarks by experienced engineers.

2. Even in JDBC world there are nano-second scale measurements.
   For instance, .prepareStatement(String) & close() is in that range.

3. "average" is not sufficient. You need to show standard deviation, test under multiple threads, etc, etc.

[vladmihalcea]

The JMH uses GPLv2. Aren't the stubs it generates also licensed as GPLv2? If that's the case, it won't be compatible with Apache 2.0 license of this particular repository.

[vladmihalcea]

The idea is good but the real scope of these benchmarks is to prove we have an improvement, not to provide absolute or relative gains which will still be environment depending.