bheisler/cargo-criterion

Cargo extension for running Criterion.rs benchmarks

cargo-criterion's Issues

Returns wrong exit code on panic

I'm really liking criterion so far ❤️ , I did encounter an issue where the wrong exit code is returned.

To reproduce:

• start from the criterion-rs readme example
• add panic!("!!") here:

fn criterion_benchmark(c: &mut Criterion) {
    panic!("!!");
    c.bench_function("fib 20", |b| b.iter(|| fibonacci(black_box(20))));
}

• run cargo criterion
• check the exit status of the previous command with echo $?
• this will return 0 but should be 101 because of the panic

What's strange is that the correct exit code is returned when the panic is moved to here:

fn fibonacci(n: u64) -> u64 {
    panic!("PANIC");
    match n {
        ...

The correct exit code is returned when using cargo bench.
Returning the wrong exit code is problematic in a CI context.

Clean up duplicated code between gnuplot & plotters backends

There's a lot of duplicated code between the two backends - stuff like doing KDE sweeps on the data, interpolating the height values of things, etc. There have been some bugs which were fixed in one and not the other. Fixing everything twice is a real drag, and now that all of that code is also duplicated in cargo-criterion (for the time being) I have to fix everything four times.

Add CDF plot to benchmark reports

CDF plots are less commonly understood, but once one learns to read them they're actually much better at conveying information than the PDF plots cargo-criterion generates. We should generate a CDF plot as well as the PDF plot for the more advanced users.

Could maybe generate a normal-quantile plot for the really advanced users as well?

Hangs when using a log scale

It appears that cargo-criterion does not like it when I change the axis scaling of a group to be logarithmic. For example, the code below will run through the tests, and at the end the cargo-criterion seems to hang indefinitely when the plot_config line is uncommented:

pub fn insert_empty(c: &mut Criterion) {
    let mut group = c.benchmark_group("BTreeMap Insert Empty");
    // group.plot_config(PlotConfiguration::default().summary_scale(AxisScale::Logarithmic));

    for i in [1, 10, 100, 1000, 10_000, 100_000] {
        group.bench_function(BenchmarkId::from_parameter(i), |b| {
            let mut sl: BTreeMap<usize, usize> = BTreeMap::new();
            let mut rng = StdRng::seed_from_u64(0x1234abcd);

            b.iter(|| {
                for _ in 0..i {
                    sl.insert(rng.gen(), rng.gen());
                }
            })
        });
    }
}

I haven't checked yet whether this is specific to the logarithmic scaling or other plot configuration options.

Broken Pipe / Resource temporarily unavailable

Hello,

When running cargo criterion it compiles fine but immediately gives me this error:

Error: Unable to open connection to bench target bron_kerbosch

Caused by:
    Resource temporarily unavailable (os error 35)
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: IoError(Os { code: 32, kind: BrokenPipe, message: "Broken pipe" })', /Users/heychsea/.cargo/registry/src/github.com-1ecc6299db9ec823/criterion-0.3.3/src/lib.rs:1437:13
stack backtrace:
   0:        0x10b013b0f - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h8ad47541fb823094
   1:        0x10b03c98e - core::fmt::write::he23e9bd6497f06c3
   2:        0x10b00e6c7 - std::io::Write::write_fmt::h91528f1131e6004e
   3:        0x10b0159ba - std::panicking::default_hook::{{closure}}::h1983f014a45a477e
   4:        0x10b0156fc - std::panicking::default_hook::hb0d9b93b5be8b120
   5:        0x10b015f88 - std::panicking::rust_panic_with_hook::h9f3faafe0983de5c
   6:        0x10b015b52 - rust_begin_unwind
   7:        0x10b05dc1f - core::panicking::panic_fmt::h0a2d1b3ca0b14316
   8:        0x10b05db25 - core::result::unwrap_failed::h71fe5d08237d0e52
   9:        0x10ae40576 - criterion::Criterion<M>::benchmark_group::ha490315f5602edbe
  10:        0x10ae4c62d - bron_kerbosch::seq::h54a8f4033907fcc9
  11:        0x10ae4ca90 - bron_kerbosch::main::hae67f97122857ce5
  12:        0x10ae48566 - std::rt::lang_start::{{closure}}::ha1ddb8b9e0daa0b0
  13:        0x10b0163b9 - std::rt::lang_start_internal::h795af50241e12334
  14:        0x10ae4cbe9 - main

On the other side, running cargo bench works perfectly.

I'm using cargo-criterion v1.0.0-alpha3 and criterion-rs v0.3.3 and I'm working on macOS 15.5.5.

Implement machine-readable output

There have been occasional requests for machine-readable output from Criterion.rs over the years. The existing design (raw CSV file with no analysis) is clearly inadequate. I like the design of Cargo's --message-format option and will probably use something similar for cargo-criterion.

Missing commit and tag for 1.0.1

Hi,

I just noticed that there's no matching tag for the 1.0.1 release, and that at least one commit (where you bumped the version number and updated the changelog) is missing.

---

cargo-criterion-1.0.1/.cargo_vcs_info.json:

{
  "git": {
    "sha1": "c18ad626d6f9e33028f7e3e32e10658cd3ab7d30"
  }
}

Passing `--measurement-time` works but returns error 101

When I use --measurement-time criterion it works as expected, but has an exit status: 101 and echo $? returns 1. But without itecho $? returns 0. I've used time so with you can see that "measurement-time" increased. See below for a little more info.

$ time cargo criterion -- --measurement-time=10
    Finished bench [optimized] target(s) in 0.03s
sm_enum_msgs/sm_enum_msgs                                                                             
                        time:   [48.865 ns 48.942 ns 49.027 ns]
                        change: [-4.7610% -4.3953% -4.0512%] (p = 0.00 < 0.05)
                        Performance has improved.

sm_enum_msgs_any/sm_enum_msgs_any                                                                             
                        time:   [62.483 ns 62.589 ns 62.693 ns]
                        change: [-5.8203% -5.0130% -4.3315%] (p = 0.00 < 0.05)
                        Performance has improved.

sm_individual_msgs_any/sm_individual_msgs_any                                                                             
                        time:   [45.092 ns 45.393 ns 45.704 ns]
                        change: [-3.6137% -2.9714% -2.2768%] (p = 0.00 < 0.05)
                        Performance has improved.

error: Unrecognized option: 'measurement-time'
Error: Non-Criterion.rs benchmark target exper_message_trait exited with exit status: 101

real    0m53.730s
user    1m14.721s
sys     0m0.818s

$ echo $?
1

$ time cargo criterion
    Finished bench [optimized] target(s) in 0.03s
sm_enum_msgs/sm_enum_msgs                                                                             
                        time:   [50.109 ns 50.196 ns 50.283 ns]
                        change: [+2.1998% +2.4297% +2.6668%] (p = 0.00 < 0.05)
                        Performance has regressed.

sm_enum_msgs_any/sm_enum_msgs_any                                                                            
                        time:   [63.337 ns 63.442 ns 63.566 ns]
                        change: [+1.6191% +2.0968% +2.5335%] (p = 0.00 < 0.05)
                        Performance has regressed.

sm_individual_msgs_any/sm_individual_msgs_any                                                                             
                        time:   [48.363 ns 48.574 ns 48.759 ns]
                        change: [+4.3939% +5.0563% +5.7090%] (p = 0.00 < 0.05)
                        Performance has regressed.


running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s


real    0m39.131s
user    1m1.939s
sys     0m0.796s

$ echo $?
0

If you interrupt the "interactive output" you can see "estimated 10.000 s (195M iterations)":

$ time cargo criterion -- --measurement-time=10
    Finished bench [optimized] target(s) in 0.03s
Benchmarking sm_enum_msgs/sm_enum_msgs: Collecting 100 samples in estimated 10.000 s (195M iterations)^C

real    0m5.631s
user    0m0.177s
sys     0m0.051s

Where as without --measurement-time we see "estimated 5.0001 s (97M iterations)":

$ time cargo criterion
    Finished bench [optimized] target(s) in 0.02s
Benchmarking sm_enum_msgs/sm_enum_msgs: Collecting 100 samples in estimated 5.0001 s (97M iterations)^C

real    0m4.583s
user    0m0.152s
sys     0m0.058s

More robust outlier detection?

I read this a while ago: https://aakinshin.net/posts/harrell-davis-double-mad-outlier-detector/

It describes a way to detect outliers which is designed for benchmarks and generates less noise on real benchmark results. I've asked a friend of mine who knows more about statistics than I do and he said it made sense to him as well.

Fails to compile due to crossbeam-epoch error

When I tried to do a standard install using cargo install cargo-criterion I get the following error:

error[E0658]: trait bounds other than `Sized` on const fn parameters are unstable
   --> C:\Users\Administrator\.cargo\registry\src\github.com-1ecc6299db9ec823\crossbeam-epoch-0.9.8\src\atomic.rs:314:6
    |
314 | impl<T: ?Sized + Pointable> Atomic<T> {
    |      ^
...
346 |     pub const fn null() -> Atomic<T> {
    |     -------------------------------- function declared as const here
    |
    = note: see issue #93706 <https://github.com/rust-lang/rust/issues/93706> for more information
    = help: add `#![feature(const_fn_trait_bound)]` to the crate attributes to enable

For more information about this error, try `rustc --explain E0658`.
error: could not compile `crossbeam-epoch` due to previous error
warning: build failed, waiting for other jobs to finish...
error: failed to compile `cargo-criterion v1.1.0`

Generate Markdown summary report for GitHub Actions CI

I was taking a stab tonight at adding benchmarking runs to my CI (I know the data quality would be lousy due to the oversubscribed & virtualized environment, but it's still useful enough for my use cases).

Unfortunately, the https://github.com/actions-rs/cargo step doesn't support capturing console output, so I can't just grab stdout from that and shove it into a comment. (And even if I could, that'd have a lot of compilation, etc. noise.)

As an alternative, it'd be great if cargo criterion produced a single, summary Markdown report file in target/criterion/reports. I could just grab that file's contents, shove it into a GitHub comment, and bingo: a great benchmarking UX as part of CI for my projects.

Status of project?

Hello, what is the status of this project, As I understood at one time the idea was to move logic from criterion here, but the last commit seem to have been roughly 3 years ago. Should this project be deemed abandoned? @lemmih what is your thought on this as you seem to be the most active contributor? do you have push access to this repo?

Add breadcrumb links to reports

It's sort of difficult to browse around through the reports because most of the links only go one way. It would be helpful to add breadcrumb links to the top of each of the reports so that the user can click back to the index easily.

Historical plots on group reports

I can definitely see a need for historical plots showing all members of a benchmark group. This is not currently supported, but would be nice to have.

There is one unresolved design question though: since the user can run arbitrary subsets of their benchmarks, there may be gaps in the history of some benchmarks. How should this be handled? For that matter, it's not clear at the moment how to match entries in the history between different benchmarks at all.

Integration tests fail with `attempt to multiply with overflow` on rust 1.52.1

Running with

cd integration_tests
cargo test

causes:

Click for full error

Compiling integration_tests v0.1.0 (/home/anton/gitrepos/cargo-criterion/integration_tests) Finished test [unoptimized + debuginfo] target(s) in 0.67s Running unittests (target/debug/deps/integration_tests-6295f0772152a35b)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

 Running tests/integration_test.rs (target/debug/deps/integration_test-aa3b4ce9c54b5adc)

running 3 tests
test test_cargo_criterion_gnuplot ... ignored
{"reason":"benchmark-complete","id":"norm","report_directory":"/tmp/.tmpCJecdo/reports/norm","iteration_count":[57,114,171,228,285,342,399,456,513,570,627,684,741,798,855,912,969,1026,1083,1140,1197,1254,1311,1368,1425,1482,1539,1596,1653,1710,1767,1824,1881,1938,1995,2052,2109,2166,2223,2280,2337,2394,2451,2508,2565,2622,2679,2736,2793,2850,2907,2964,3021,3078,3135,3192,3249,3306,3363,3420,3477,3534,3591,3648,3705,3762,3819,3876,3933,3990,4047,4104,4161,4218,4275,4332,4389,4446,4503,4560,4617,4674,4731,4788,4845,4902,4959,5016,5073,5130,5187,5244,5301,5358,5415,5472,5529,5586,5643,5700],"measured_values":[104590.0,196240.0,296570.0,397450.0,490520.0,590620.0,692499.0,789310.0,886919.0,986400.0,1079389.0,1179220.0,1279159.0,1381020.0,1477029.0,1569979.0,1676369.0,1764929.0,1902729.0,1970770.0,2072729.0,2151349.0,2252008.0,2353279.0,2456759.0,2570469.0,2686609.0,2771668.0,2870060.0,2965728.0,3073488.0,3162109.0,3261248.0,3358338.0,3453810.0,3560108.0,3660537.0,3748309.0,3886598.0,3968319.0,4051058.0,4155497.0,4247939.0,4350427.0,4454529.0,4544837.0,4654628.0,4747088.0,4889158.0,4918798.0,5079917.0,5205978.0,5324388.0,5434687.0,5536257.0,5697117.0,5729278.0,5834597.0,5928708.0,5948567.0,6158457.0,6728567.0,6322957.0,6335527.0,6402397.0,6513906.0,6594378.0,6696247.0,6843286.0,6886227.0,7005258.0,7085265.0,7176107.0,7256087.0,7431566.0,7473197.0,7577377.0,7657116.0,7770837.0,7894835.0,7952597.0,8047377.0,8191986.0,8278396.0,8362525.0,8446627.0,8559706.0,8668266.0,8708086.0,8888326.0,8937896.0,9053475.0,9144577.0,9310344.0,9372786.0,9471816.0,9507886.0,9624865.0,9773416.0,9804945.0],"unit":"ns","throughput":[],"typical":{"estimate":1734.0875055675247,"lower_bound":1730.1024732323847,"upper_bound":1739.7038966958373,"unit":"ns"},"mean":{"estimate":1736.718499176603,"lower_bound":1732.6683570862585,"upper_bound":1741.6644392954806,"unit":"ns"},"median":{"estimate":1731.2782738095239,"lower_bound":1728.8162768031189,"upper_bound":1733.3474446987034,"unit":"ns"},"median_abs_dev":{"estimate":8.036445465877234,"lower_bound":6.842009988926375,"upper_bound":10.89675236537435,"unit":"ns"},"slope":{"estimate":1734.0875055675247,"lower_bound":1730.1024732323847,"upper_bound":1739.7038966958373,"unit":"ns"},"change":null}
{"reason":"group-complete","group_name":"norm","benchmarks":["norm"],"report_directory":"/tmp/.tmpCJecdo/reports/norm"}
{"reason":"benchmark-complete","id":""*group/"/"*benchmark/" '","report_directory":"/tmp/.tmpCJecdo/reports/group/benchmark '","iteration_count":[5138,10276,15414,20552,25690,30828,35966,41104,46242,51380,56518,61656,66794,71932,77070,82208,87346,92484,97622,102760,107898,113036,118174,123312,128450,133588,138726,143864,149002,154140,159278,164416,169554,174692,179830,184968,190106,195244,200382,205520,210658,215796,220934,226072,231210,236348,241486,246624,251762,256900,262038,267176,272314,277452,282590,287728,292866,298004,303142,308280,313418,318556,323694,328832,333970,339108,344246,349384,354522,359660,364798,369936,375074,380212,385350,390488,395626,400764,405902,411040,416178,421316,426454,431592,436730,441868,447006,452144,457282,462420,467558,472696,477834,482972,488110,493248,498386,503524,508662,513800],"measured_values":[98160.0,195760.0,293640.0,391490.0,489370.0,589350.0,685100.0,783499.0,917310.0,983929.0,1081049.0,1175810.0,1279080.0,1374140.0,1480999.0,1569269.0,1665128.0,1767079.0,1868649.0,1958550.0,2062509.0,2164549.0,2262379.0,2353290.0,2455708.0,2554249.0,2647478.0,2754289.0,2841089.0,2963798.0,3046730.0,3140758.0,3247878.0,3334728.0,3441859.0,3544998.0,3633129.0,3729418.0,3822348.0,3937288.0,4020988.0,4123069.0,4223318.0,4329167.0,4420179.0,4508737.0,4618619.0,4718507.0,4812608.0,4907527.0,5008677.0,5098889.0,5213348.0,5323227.0,5403537.0,5523188.0,5670868.0,5773466.0,5873078.0,5968717.0,6056017.0,6154987.0,6270727.0,6368148.0,6467026.0,6573707.0,6658798.0,6758826.0,6874327.0,6964767.0,7065197.0,7118536.0,7203017.0,7275277.0,7369337.0,7460985.0,7576507.0,7669797.0,7759876.0,7869757.0,7955585.0,8053807.0,8144627.0,8268215.0,8361447.0,9085305.0,8560136.0,8649976.0,8772106.0,8834867.0,8964915.0,9037467.0,9126394.0,9268486.0,9384275.0,9554336.0,9661756.0,9780925.0,9834236.0,9973405.0],"unit":"ns","throughput":[],"typical":{"estimate":19.247054600546708,"lower_bound":19.19395421527887,"upper_bound":19.328581839400233,"unit":"ns"},"mean":{"estimate":19.19382629069943,"lower_bound":19.161240326089256,"upper_bound":19.234006852505,"unit":"ns"},"median":{"estimate":19.136364813476533,"lower_bound":19.119665542075506,"upper_bound":19.149412236265096,"unit":"ns"},"median_abs_dev":{"estimate":0.05590961631305071,"lower_bound":0.03771709537543891,"upper_bound":0.08572723881869536,"unit":"ns"},"slope":{"estimate":19.247054600546708,"lower_bound":19.19395421527887,"upper_bound":19.328581839400233,"unit":"ns"},"change":null}
{"reason":"group-complete","group_name":""*group/"","benchmarks":[""*group/"/"*benchmark/" '"],"report_directory":"/tmp/.tmpCJecdo/reports/group"}
{"reason":"benchmark-complete","id":"sampling_mode/Auto (short)","report_directory":"/tmp/.tmpCJecdo/reports/sampling_mode/Auto (short)","iteration_count":[4617,9234,13851,18468,23085,27702,32319,36936,41553,46170,50787,55404,60021,64638,69255,73872,78489,83106,87723,92340,96957,101574,106191,110808,115425,120042,124659,129276,133893,138510,143127,147744,152361,156978,161595,166212,170829,175446,180063,184680,189297,193914,198531,203148,207765,212382,216999,221616,226233,230850,235467,240084,244701,249318,253935,258552,263169,267786,272403,277020,281637,286254,290871,295488,300105,304722,309339,313956,318573,323190,327807,332424,337041,341658,346275,350892,355509,360126,364743,369360,373977,378594,383211,387828,392445,397062,401679,406296,410913,415530,420147,424764,429381,433998,438615,443232,447849,452466,457083,461700],"measured_values":[97340.0,193490.0,317950.0,393660.0,483720.0,585430.0,677199.0,773940.0,872599.0,972550.0,1084179.0,1164079.0,1257619.0,1356690.0,1458809.0,1556070.0,1656779.0,1750569.0,1977389.0,1945170.0,2039119.0,2128249.0,2236758.0,2352099.0,2435849.0,2521609.0,2625839.0,2716399.0,2819979.0,2914548.0,3010558.0,3325769.0,3211439.0,3303798.0,3412787.0,3508590.0,3619748.0,3689309.0,3778978.0,3891307.0,3982169.0,4074528.0,4169348.0,4276618.0,4367447.0,4463749.0,4592018.0,4658988.0,4762767.0,4893748.0,4950647.0,5053509.0,5146367.0,5260898.0,5412647.0,5491318.0,5616356.0,5628769.0,5769087.0,5827697.0,5970347.0,6039817.0,6117678.0,6225356.0,6306558.0,6408107.0,6514146.0,6688088.0,6704207.0,6798856.0,6896707.0,7156027.0,7099907.0,7214726.0,7285887.0,7404297.0,7496717.0,7568955.0,7683457.0,7786397.0,7872166.0,7957696.0,8087736.0,8179557.0,8295936.0,8450476.0,8537537.0,8626034.0,8770697.0,8876995.0,8947686.0,9034456.0,9106926.0,9172916.0,9249266.0,9368466.0,9433595.0,9533576.0,9629355.0,9718776.0],"unit":"ns","throughput":[],"typical":{"estimate":21.13472126561526,"lower_bound":21.10567325686512,"upper_bound":21.167039187037854,"unit":"ns"},"mean":{"estimate":21.145141945876116,"lower_bound":21.09442823021782,"upper_bound":21.208269559921067,"unit":"ns"},"median":{"estimate":21.06492311024475,"lower_bound":21.052485711633583,"upper_bound":21.085170030857164,"unit":"ns"},"median_abs_dev":{"estimate":0.06526001689064082,"lower_bound":0.050597343851349105,"upper_bound":0.09013168116381864,"unit":"ns"},"slope":{"estimate":21.13472126561526,"lower_bound":21.10567325686512,"upper_bound":21.167039187037854,"unit":"ns"},"change":null}
{"reason":"benchmark-complete","id":"sampling_mode/Auto (long)","report_directory":"/tmp/.tmpCJecdo/reports/sampling_mode/Auto (long)","iteration_count":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1],"measured_values":[10057535.0,10057245.0,10057416.0,10056135.0,10057255.0,10055876.0,10057545.0,10056215.0,10055546.0,10055065.0,10056865.0,10058606.0,10059525.0,10055335.0,10055696.0,10055855.0,10056115.0,10056116.0,10057655.0,10190685.0,10056006.0,10054875.0,10054865.0,10054766.0,10055515.0,10055905.0,11740765.0,10055985.0,10056205.0,10057406.0,10055035.0,10054565.0,10055666.0,10054335.0,10058845.0,10057906.0,10057055.0,10056136.0,10056305.0,10056335.0,10056425.0,10055766.0,10055635.0,10055916.0,10054465.0,10055565.0,10058485.0,10057766.0,10058395.0,10055386.0,10056765.0,10055666.0,10056484.0,10056006.0,10055655.0,10054556.0,10056445.0,10056586.0,10057895.0,10059416.0,10058534.0,10063756.0,10065545.0,10056716.0,10056155.0,10056416.0,10008524.0,10054976.0,10053825.0,10057346.0,10055855.0,10055436.0,10056074.0,10055817.0,10058384.0,10056496.0,10056465.0,10054806.0,10055855.0,10057976.0,10053684.0,10056237.0,10055424.0,10055626.0,10057865.0,10059376.0,10056064.0,10056357.0,10055004.0,10060386.0,10055965.0,10057896.0,10058495.0,10058096.0,10058594.0,10057767.0,10056414.0,10058686.0,10057815.0,10057606.0],"unit":"ns","throughput":[],"typical":{"estimate":10074403.54,"lower_bound":10055635.99,"upper_bound":10109829.4355,"unit":"ns"},"mean":{"estimate":10074403.54,"lower_bound":10055635.99,"upper_bound":10109829.4355,"unit":"ns"},"median":{"estimate":10056271.0,"lower_bound":10056040.0,"upper_bound":10056586.0,"unit":"ns"},"median_abs_dev":{"estimate":1415.8829748630524,"lower_bound":867.1727246046069,"upper_bound":1904.3996661901474,"unit":"ns"},"slope":null,"change":null}
{"reason":"benchmark-complete","id":"sampling_mode/Linear","report_directory":"/tmp/.tmpCJecdo/reports/sampling_mode/Linear","iteration_count":[4220,8440,12660,16880,21100,25320,29540,33760,37980,42200,46420,50640,54860,59080,63300,67520,71740,75960,80180,84400,88620,92840,97060,101280,105500,109720,113940,118160,122380,126600,130820,135040,139260,143480,147700,151920,156140,160360,164580,168800,173020,177240,181460,185680,189900,194120,198340,202560,206780,211000,215220,219440,223660,227880,232100,236320,240540,244760,248980,253200,257420,261640,265860,270080,274300,278520,282740,286960,291180,295400,299620,303840,308060,312280,316500,320720,324940,329160,333380,337600,341820,346040,350260,354480,358700,362920,367140,371360,375580,379800,384020,388240,392460,396680,400900,405120,409340,413560,417780,422000],"measured_values":[88740.0,176830.0,265240.0,353630.0,442029.0,530430.0,618830.0,715379.0,795630.0,884010.0,972410.0,1068950.0,1149209.0,1722789.0,2058979.0,2345250.0,2458068.0,2542889.0,1697958.0,2745219.0,3153089.0,3300958.0,3479389.0,3671409.0,3800017.0,3959669.0,4197048.0,2925678.0,4369139.0,4197777.0,2836209.0,2883679.0,3074098.0,3302769.0,3138649.0,3218748.0,3301948.0,3403869.0,3485668.0,3574979.0,3703978.0,3745398.0,3845848.0,3921739.0,4013838.0,4112548.0,4192427.0,4304149.0,4373948.0,4455608.0,4535688.0,4625427.0,4723569.0,5160427.0,4874698.0,4977827.0,5068588.0,5155258.0,5233357.0,5347897.0,5407988.0,5517028.0,5624527.0,5710458.0,5776297.0,5859737.0,5956207.0,6400687.0,6145358.0,6257216.0,6326118.0,6425727.0,6539196.0,6600638.0,6675487.0,6778896.0,6865347.0,6957838.0,7024105.0,7359577.0,7303607.0,7307226.0,7386237.0,7476027.0,7564357.0,7659196.0,7716496.0,7847456.0,7890647.0,7998167.0,8400896.0,8185956.0,8261475.0,8369977.0,8472256.0,8540176.0,8641907.0,8703145.0,8794597.0,9173264.0],"unit":"ns","throughput":[],"typical":{"estimate":21.514037339696337,"lower_bound":21.335271914092147,"upper_bound":21.74311732746399,"unit":"ns"},"mean":{"estimate":23.233500708452116,"lower_bound":22.279706780292788,"upper_bound":24.234025928446673,"unit":"ns"},"median":{"estimate":21.13243897658839,"lower_bound":21.110307038973307,"upper_bound":21.15266466776284,"unit":"ns"},"median_abs_dev":{"estimate":0.12127936812712264,"lower_bound":0.08636496915708732,"upper_bound":0.18171665842273743,"unit":"ns"},"slope":{"estimate":21.514037339696337,"lower_bound":21.335271914092147,"upper_bound":21.74311732746399,"unit":"ns"},"change":null}
{"reason":"benchmark-complete","id":"sampling_mode/Flat","report_directory":"/tmp/.tmpCJecdo/reports/sampling_mode/Flat","iteration_count":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1],"measured_values":[10058416.0,10058066.0,10058145.0,10060815.0,10058285.0,10058116.0,10057825.0,10055906.0,10056215.0,10055826.0,10056765.0,10057885.0,10055435.0,10055626.0,10058975.0,10057226.0,10056945.0,10058805.0,10060406.0,10055625.0,10054815.0,10056126.0,10057235.0,10057075.0,10054256.0,10056855.0,10057105.0,10059856.0,10062995.0,10064155.0,10058956.0,10059075.0,10058565.0,10058566.0,10057735.0,10056635.0,10058346.0,10060645.0,10059505.0,10062716.0,10061875.0,10064245.0,10057736.0,10060305.0,10064245.0,10060396.0,10056705.0,10061725.0,10058316.0,10063905.0,10057065.0,10058586.0,10063905.0,10058965.0,10059976.0,10057795.0,10059315.0,10059766.0,10058925.0,10058686.0,10058705.0,10059115.0,10059457.0,10058164.0,10059426.0,10058285.0,10058856.0,10057984.0,10058577.0,10058584.0,10057726.0,10061615.0,10059286.0,10058285.0,10061066.0,10063884.0,10127967.0,10059154.0,10059136.0,10057735.0,10058266.0,10059214.0,10059257.0,10059604.0,10027906.0,10061415.0,10059906.0,10058345.0,10058866.0,10062914.0,10057716.0,10058565.0,10057156.0,10058096.0,10057015.0,10061955.0,10062786.0,10061615.0,10056335.0,10063576.0],"unit":"ns","throughput":[],"typical":{"estimate":10059384.77,"lower_bound":10058146.83,"upper_bound":10061188.15925,"unit":"ns"},"mean":{"estimate":10059384.77,"lower_bound":10058146.83,"upper_bound":10061188.15925,"unit":"ns"},"median":{"estimate":10058585.0,"lower_bound":10058285.0,"upper_bound":10058975.0,"unit":"ns"},"median_abs_dev":{"estimate":1817.66756772995,"lower_bound":1165.3050468116999,"upper_bound":2534.5046550035477,"unit":"ns"},"slope":null,"change":null}
{"reason":"group-complete","group_name":"sampling_mode","benchmarks":["sampling_mode/Auto (short)","sampling_mode/Auto (long)","sampling_mode/Linear","sampling_mode/Flat"],"report_directory":"/tmp/.tmpCJecdo/reports/sampling_mode"}
{"reason":"benchmark-complete","id":"throughput/Bytes","report_directory":"/tmp/.tmpCJecdo/reports/throughput/Bytes","iteration_count":[5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195,200,205,210,215,220,225,230,235,240,245,250,255,260,265,270,275,280,285,290,295,300,305,310,315,320,325,330,335,340,345,350,355,360,365,370,375,380,385,390,395,400,405,410,415,420,425,430,435,440,445,450,455,460,465,470,475,480,485,490,495,500],"measured_values":[111150.0,221030.0,331409.0,430690.0,544220.0,641900.0,748880.0,856069.0,962870.0,1076920.0,1208679.0,1318389.0,1391840.0,1489989.0,1596330.0,1704419.0,1818449.0,1930649.0,2034708.0,2143139.0,2236210.0,2345049.0,2450968.0,2554020.0,2661909.0,2774878.0,2878828.0,2981469.0,3087519.0,3195738.0,3334009.0,3406978.0,3512828.0,3621829.0,3725548.0,3832229.0,3938238.0,4102918.0,4178578.0,4622077.0,7057408.0,7296257.0,7508635.0,4777949.0,4880448.0,4925037.0,5031457.0,5170608.0,5238368.0,5347647.0,5637158.0,9957226.0,10460704.0,9719687.0,5976336.0,6017668.0,6130757.0,6235777.0,6377987.0,6439947.0,6504237.0,6626406.0,6739197.0,6841607.0,6945517.0,7059357.0,7166197.0,7268296.0,7361647.0,7487277.0,7576676.0,7706617.0,7802416.0,7905376.0,8011757.0,8123225.0,8236197.0,8335106.0,8454046.0,8541647.0,8669025.0,8766926.0,8865296.0,8971696.0,9074316.0,9201806.0,9276756.0,9389255.0,9481366.0,9606976.0,9699575.0,9805116.0,9927795.0,10021665.0,10133096.0,10227965.0,10335305.0,10447725.0,10556905.0,10673015.0],"unit":"ns","throughput":[{"per_iteration":1024,"unit":"bytes"}],"typical":{"estimate":22006.762317127235,"lower_bound":21567.60816806777,"upper_bound":22584.99714767802,"unit":"ns"},"mean":{"estimate":22344.335776907872,"lower_bound":21715.162660224963,"upper_bound":23075.11634305545,"unit":"ns"},"median":{"estimate":21381.92010950722,"lower_bound":21364.685333333335,"upper_bound":21396.619047619046,"unit":"ns"},"median_abs_dev":{"estimate":91.42096260248718,"lower_bound":61.978455125645915,"upper_bound":126.44208751539911,"unit":"ns"},"slope":{"estimate":22006.762317127235,"lower_bound":21567.60816806777,"upper_bound":22584.99714767802,"unit":"ns"},"change":null}
{"reason":"benchmark-complete","id":"throughput/Elem","report_directory":"/tmp/.tmpCJecdo/reports/throughput/Elem","iteration_count":[4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,100,104,108,112,116,120,124,128,132,136,140,144,148,152,156,160,164,168,172,176,180,184,188,192,196,200,204,208,212,216,220,224,228,232,236,240,244,248,252,256,260,264,268,272,276,280,284,288,292,296,300,304,308,312,316,320,324,328,332,336,340,344,348,352,356,360,364,368,372,376,380,384,388,392,396,400],"measured_values":[96640.0,181300.0,278170.0,362280.0,452680.0,553930.0,629679.0,716860.0,810319.0,900780.0,992629.0,1081340.0,1171390.0,1261049.0,1352830.0,1441299.0,1529519.0,1621689.0,1717809.0,1803589.0,1891850.0,1985749.0,2071658.0,2159920.0,3126269.0,3527448.0,4078397.0,3844739.0,4539718.0,4706458.0,4876727.0,5035498.0,5224187.0,4101799.0,4181418.0,5052648.0,3326828.0,3372918.0,3452329.0,3533018.0,3621649.0,3710808.0,3805038.0,3888928.0,3974119.0,4064348.0,4150528.0,4256477.0,4331169.0,4392058.0,5030458.0,4615167.0,4655798.0,4732718.0,4843938.0,4923068.0,5007057.0,5099748.0,5181348.0,5278267.0,5347147.0,5436998.0,5506928.0,5592076.0,7316308.0,8763866.0,8532896.0,6452597.0,9321546.0,6176037.0,6244867.0,6314577.0,6491367.0,6505727.0,6612287.0,6662277.0,6751137.0,6835407.0,6915626.0,6991377.0,7084617.0,7181656.0,7268117.0,7341897.0,7425046.0,7516207.0,7603616.0,7699087.0,7829556.0,7865867.0,7953566.0,8040856.0,8126437.0,8226846.0,8311687.0,8393535.0,8477886.0,8565606.0,8651576.0,8744367.0],"unit":"ns","throughput":[{"per_iteration":1024,"unit":"elements"}],"typical":{"estimate":22956.656585636174,"lower_bound":22436.930208152335,"upper_bound":23551.09468440884,"unit":"ns"},"mean":{"estimate":24223.119471187307,"lower_bound":23368.78021542976,"upper_bound":25123.166510192925,"unit":"ns"},"median":{"estimate":22096.99141929499,"lower_bound":22005.517415730337,"upper_bound":22488.535714285714,"unit":"ns"},"median_abs_dev":{"estimate":366.8552926245737,"lower_bound":229.25960399162793,"upper_bound":785.9451954118167,"unit":"ns"},"slope":{"estimate":22956.656585636174,"lower_bound":22436.930208152335,"upper_bound":23551.09468440884,"unit":"ns"},"change":null}
{"reason":"group-complete","group_name":"throughput","benchmarks":["throughput/Bytes","throughput/Elem"],"report_directory":"/tmp/.tmpCJecdo/reports/throughput"}
{"reason":"benchmark-complete","id":"log_scale/sleep (micros)/1","report_directory":"/tmp/.tmpCJecdo/reports/log_scale/sleep (micros)/1","iteration_count":[2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102,104,106,108,110,112,114,116,118,120,122,124,126,128,130,132,134,136,138,140,142,144,146,148,150,152,154,156,158,160,162,164,166,168,170,172,174,176,178,180,182,184,186,188,190,192,194,196,198,200],"measured_values":[110510.0,220590.0,313920.0,437860.0,548030.0,658179.0,767260.0,875059.0,988690.0,1090579.0,1204099.0,1314339.0,1432320.0,1524620.0,1592759.0,1754279.0,1863979.0,1971559.0,2020210.0,2186659.0,2250438.0,2544170.0,2498239.0,2631378.0,2735168.0,2791569.0,2977909.0,3110068.0,3211880.0,3273728.0,3392878.0,3503088.0,3677619.0,3767338.0,3834419.0,5216937.0,4580228.0,4155438.0,4208158.0,4342758.0,4500507.0,4593829.0,4717347.0,4815609.0,4933186.0,5045759.0,5102967.0,5159778.0,5272537.0,5385118.0,5472758.0,7385176.0,5736938.0,5829167.0,5960367.0,6042918.0,6191876.0,6409138.0,6419147.0,6635706.0,6545448.0,6587757.0,6787586.0,6904007.0,7082158.0,7240525.0,7381177.0,7485387.0,7615836.0,7783047.0,7743656.0,7904646.0,8075407.0,8211176.0,8389906.0,9924896.0,8278005.0,8440167.0,8667756.0,8751756.0,8879186.0,15662733.0,8846745.0,8958556.0,10615126.0,9387775.0,9516516.0,9502005.0,9675506.0,9752805.0,9814046.0,9953615.0,10046056.0,10187365.0,10252685.0,10373747.0,10478934.0,10590326.0,10738234.0,10837115.0],"unit":"ns","throughput":[],"typical":{"estimate":55778.4001980198,"lower_bound":54513.06665644231,"upper_bound":57835.83846156276,"unit":"ns"},"mean":{"estimate":55527.83994282419,"lower_bound":54691.35116544173,"upper_bound":56846.37609628225,"unit":"ns"},"median":{"estimate":54691.90409482759,"lower_bound":54425.132566585955,"upper_bound":54771.61961309524,"unit":"ns"},"median_abs_dev":{"estimate":748.8030454102377,"lower_bound":529.9193783922312,"upper_bound":906.4056481937964,"unit":"ns"},"slope":{"estimate":55778.4001980198,"lower_bound":54513.06665644231,"upper_bound":57835.83846156276,"unit":"ns"},"change":null}
{"reason":"benchmark-complete","id":"log_scale/sleep (micros)/100","report_directory":"/tmp/.tmpCJecdo/reports/log_scale/sleep (micros)/100","iteration_count":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100],"measured_values":[154280.0,308350.0,467390.0,615780.0,770469.0,922080.0,1075740.0,1229310.0,1383019.0,1537129.0,1690699.0,1845260.0,2009959.0,2153729.0,2310088.0,2467669.0,2628449.0,2764889.0,2918698.0,3076149.0,3213269.0,3378368.0,3537769.0,3688138.0,3846168.0,3998169.0,4159827.0,4303569.0,4471757.0,4603288.0,4760658.0,4920408.0,5071717.0,5225828.0,5327437.0,5537699.0,5699976.0,5852218.0,5997157.0,24600319.0,6302357.0,6451106.0,6627728.0,6774937.0,6921806.0,7082557.0,7229867.0,7392806.0,7537207.0,7685336.0,7854147.0,7993097.0,8146545.0,8311337.0,8518106.0,8683445.0,8773827.0,8924226.0,9080046.0,9244035.0,9373696.0,10872515.0,9747256.0,9844845.0,10002135.0,10145456.0,10242315.0,10415855.0,10572025.0,10770555.0,10927595.0,11080036.0,11266344.0,11581535.0,11533145.0,11701444.0,11865334.0,12008566.0,12918173.0,12219704.0,12421425.0,12610994.0,22912519.0,12867655.0,13067934.0,13240384.0,13371884.0,13546534.0,13677853.0,13832514.0,14009093.0,14143514.0,14298713.0,14415013.0,14597664.0,14750433.0,14899493.0,15062944.0,15215962.0,15379913.0],"unit":"ns","throughput":[],"typical":{"estimate":158894.0320437417,"lower_bound":153953.75428601197,"upper_bound":166788.0917634852,"unit":"ns"},"mean":{"estimate":159997.91670160487,"lower_bound":153953.3272688686,"upper_bound":170528.17986642872,"unit":"ns"},"median":{"estimate":153811.74673913044,"lower_bound":153745.0010752688,"upper_bound":153872.4827586207,"unit":"ns"},"median_abs_dev":{"estimate":214.10731637537555,"lower_bound":160.68737643869426,"upper_bound":279.8384583512592,"unit":"ns"},"slope":{"estimate":158894.0320437417,"lower_bound":153953.75428601197,"upper_bound":166788.0917634852,"unit":"ns"},"change":null}
{"reason":"benchmark-complete","id":"log_scale/sleep (micros)/10000","report_directory":"/tmp/.tmpCJecdo/reports/log_scale/sleep (micros)/10000","iteration_count":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1],"measured_values":[10057005.0,21513100.0,10055766.0,10055805.0,10054566.0,10056455.0,10055936.0,10055606.0,10056655.0,10060966.0,10056504.0,10056636.0,10057475.0,10057836.0,10061915.0,10054666.0,10055155.0,10056185.0,10055806.0,10056505.0,10056975.0,10059016.0,10060695.0,10056055.0,10056386.0,10057625.0,10059635.0,10061856.0,10060165.0,10057195.0,10059346.0,10057005.0,10055475.0,10054276.0,10057165.0,10056666.0,10059855.0,10061896.0,13252344.0,10056255.0,10056866.0,10058805.0,10061195.0,10057896.0,10057495.0,10058006.0,10058195.0,10056525.0,10056126.0,10059475.0,10056435.0,10057956.0,10061635.0,10058385.0,10057836.0,10056875.0,10057965.0,10060586.0,10057205.0,10058316.0,10057305.0,10058976.0,10056935.0,10058126.0,10059334.0,10055867.0,10065274.0,10060446.0,10058246.0,10056965.0,10056306.0,10061805.0,10059135.0,10058645.0,10062666.0,10058805.0,10061036.0,10057675.0,10056235.0,10061396.0,10056545.0,10058295.0,10057016.0,10016165.0,10057935.0,10059766.0,10057815.0,10058166.0,10058625.0,10057715.0,10057795.0,10057946.0,10059966.0,10079735.0,10061096.0,10058934.0,10060797.0,10057544.0,10058466.0,10058215.0],"unit":"ns","throughput":[],"typical":{"estimate":10204399.22,"lower_bound":10057439.21825,"upper_bound":10467325.421499997,"unit":"ns"},"mean":{"estimate":10204399.22,"lower_bound":10057439.21825,"upper_bound":10467325.421499997,"unit":"ns"},"median":{"estimate":10057866.0,"lower_bound":10057400.0,"upper_bound":10058215.0,"unit":"ns"},"median_abs_dev":{"estimate":2002.9925644397736,"lower_bound":1457.3216441273692,"upper_bound":2499.0519831329575,"unit":"ns"},"slope":null,"change":null}
{"reason":"group-complete","group_name":"log_scale","benchmarks":["log_scale/sleep (micros)/1","log_scale/sleep (micros)/100","log_scale/sleep (micros)/10000"],"report_directory":"/tmp/.tmpCJecdo/reports/log_scale"}
{"reason":"benchmark-complete","id":"linear_scale/sleep (millis)/1","report_directory":"/tmp/.tmpCJecdo/reports/linear_scale/sleep (millis)/1","iteration_count":[5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5],"measured_values":[5280217.0,5268888.0,5275357.0,5275848.0,5258397.0,5275588.0,5273298.0,5271087.0,5234539.0,5272336.0,5278008.0,5269428.0,5224338.0,5267786.0,5271989.0,5270427.0,5246038.0,5269117.0,5271548.0,5272518.0,5243447.0,5281158.0,10502965.0,5275357.0,5274938.0,5269847.0,5282288.0,5275628.0,5270827.0,5273317.0,5273068.0,5278989.0,5272726.0,5277458.0,5275687.0,5271549.0,5278087.0,5279188.0,5277637.0,5280698.0,5276088.0,5241987.0,5276067.0,5276978.0,5280208.0,5276347.0,5278658.0,5276137.0,5276728.0,5281518.0,5290257.0,5276147.0,5276938.0,5279209.0,5275666.0,5248218.0,5278977.0,5248939.0,5293706.0,5238098.0,5272368.0,5277978.0,5275676.0,5275639.0,5249967.0,5286058.0,5279148.0,5275027.0,5283648.0,5280557.0,5287328.0,5279367.0,5287538.0,5258277.0,5279758.0,5281017.0,5278178.0,5277818.0,5277987.0,5280619.0,5279886.0,5279009.0,5281437.0,5281888.0,5277716.0,5275789.0,5279057.0,5249088.0,5270677.0,5290188.0,5276778.0,5278407.0,5280958.0,5278817.0,5290598.0,5284217.0,5279408.0,5281267.0,5280258.0,5275308.0],"unit":"ns","throughput":[],"typical":{"estimate":1065206.3719999997,"lower_bound":1054372.8328500001,"upper_bound":1086429.2095999992,"unit":"ns"},"mean":{"estimate":1065206.3719999997,"lower_bound":1054372.8328500001,"upper_bound":1086429.2095999992,"unit":"ns"},"median":{"estimate":1055350.6,"lower_bound":1055135.2999999998,"upper_bound":1055601.6,"unit":"ns"},"median_abs_dev":{"estimate":1065.9893810749054,"lower_bound":701.0548105536581,"upper_bound":1349.3253555446872,"unit":"ns"},"slope":null,"change":null}
{"reason":"benchmark-complete","id":"linear_scale/sleep (millis)/2","report_directory":"/tmp/.tmpCJecdo/reports/linear_scale/sleep (millis)/2","iteration_count":[3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3],"measured_values":[6165657.0,6169427.0,6169488.0,6169066.0,6172658.0,6167178.0,6166837.0,6174216.0,6170268.0,6168257.0,6170477.0,6169487.0,6167817.0,6165188.0,6165126.0,6170648.0,6164998.0,6169137.0,6168606.0,22928819.0,6165248.0,6166458.0,6172746.0,6170708.0,6163597.0,6172397.0,6169277.0,6165567.0,6163777.0,6166398.0,6168346.0,6167508.0,6164758.0,6165866.0,6164558.0,6168367.0,6165927.0,6166797.0,6171977.0,6177947.0,6169168.0,6163576.0,6166378.0,6118708.0,6165246.0,6163418.0,6167497.0,6164017.0,6164407.0,6163557.0,6165387.0,6169728.0,6164156.0,6172978.0,6168698.0,6165986.0,6163428.0,6166137.0,6167257.0,6168927.0,6163377.0,6164827.0,6164278.0,6164826.0,6166418.0,6168868.0,6173886.0,6156278.0,6168717.0,6168477.0,6172947.0,6170637.0,6170177.0,6166958.0,6175606.0,6169418.0,6166248.0,6180096.0,6156958.0,6168007.0,6163407.0,6165677.0,6164757.0,6165917.0,6163318.0,6165076.0,6165378.0,6173328.0,6164476.0,9820906.0,6164297.0,6167456.0,6166538.0,6124018.0,6170357.0,6163657.0,6166147.0,6171527.0,6169388.0,6167536.0],"unit":"ns","throughput":[],"typical":{"estimate":2123567.9933333327,"lower_bound":2055388.4932500003,"upper_bound":2247576.9606666663,"unit":"ns"},"mean":{"estimate":2123567.9933333327,"lower_bound":2055388.4932500003,"upper_bound":2247576.9606666663,"unit":"ns"},"median":{"estimate":2055632.5,"lower_bound":2055355.4583333333,"upper_bound":2056085.6666666667,"unit":"ns"},"median_abs_dev":{"estimate":1139.1309797762674,"lower_bound":914.2699837685782,"upper_bound":1383.5128754378516,"unit":"ns"},"slope":null,"change":null}
{"reason":"benchmark-complete","id":"linear_scale/sleep (millis)/3","report_directory":"/tmp/.tmpCJecdo/reports/linear_scale/sleep (millis)/3","iteration_count":[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2],"measured_values":[6110296.0,6119108.0,6112928.0,6111516.0,6109938.0,6112897.0,6112437.0,6117718.0,6066756.0,6112048.0,6110958.0,6109676.0,6110278.0,6113467.0,6113327.0,6110197.0,6114367.0,6113017.0,6110608.0,6110606.0,6112628.0,6113508.0,6112316.0,6111068.0,6112057.0,6114007.0,6110507.0,6113337.0,6110338.0,6112478.0,6112606.0,6109998.0,6109747.0,6113747.0,6117007.0,6112947.0,6110797.0,6110248.0,6110356.0,6115298.0,6116928.0,6111786.0,6112578.0,6111877.0,6109427.0,6110437.0,6110237.0,6112588.0,6099548.0,6119396.0,6115808.0,6110457.0,6117137.0,6081517.0,6114597.0,6116347.0,6120367.0,6111107.0,6113338.0,6112528.0,6113446.0,6111568.0,6114507.0,6113547.0,6114957.0,6112437.0,6111898.0,6114948.0,6120016.0,6111378.0,6116717.0,6111377.0,6077847.0,6113067.0,6112807.0,6114977.0,6110717.0,6112138.0,6111658.0,6114116.0,6112438.0,6116247.0,6113197.0,6112537.0,6110667.0,6124528.0,8360666.0,6115667.0,6110987.0,6116628.0,6089276.0,6114428.0,6112608.0,6119366.0,6110148.0,6115437.0,6119917.0,6112068.0,6116186.0,6117088.0],"unit":"ns","throughput":[],"typical":{"estimate":3067146.75,"lower_bound":3055243.694,"upper_bound":3090108.975875,"unit":"ns"},"mean":{"estimate":3067146.75,"lower_bound":3055243.694,"upper_bound":3090108.975875,"unit":"ns"},"median":{"estimate":3056291.5,"lower_bound":3056034.0,"upper_bound":3056566.0,"unit":"ns"},"median_abs_dev":{"estimate":1423.2959747314453,"lower_bound":1022.252681851387,"upper_bound":1735.3832691907883,"unit":"ns"},"slope":null,"change":null}
{"reason":"group-complete","group_name":"linear_scale","benchmarks":["linear_scale/sleep (millis)/1","linear_scale/sleep (millis)/2","linear_scale/sleep (millis)/3"],"report_directory":"/tmp/.tmpCJecdo/reports/linear_scale"}
Blocking waiting for file lock on package cache
Blocking waiting for file lock on package cache
Blocking waiting for file lock on package cache
Compiling integration_tests v0.1.0 (/home/anton/gitrepos/cargo-criterion/integration_tests)
Finished test [unoptimized + debuginfo] target(s) in 0.94s

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Benchmarking norm
Benchmarking norm: Warming up for 250.00 ms
Benchmarking norm: Collecting 100 samples in estimated 503.80 ms (288k iterations)
Benchmarking norm: Analyzing
norm time: [1.7301 us 1.7341 us 1.7397 us]

running 1 test
test bencher_test ... bench: 14,685 ns/iter (+/- 523)

test result: ok. 0 passed; 0 failed; 0 ignored; 1 measured

Benchmarking "*group/"/"*benchmark/" '
Benchmarking "*group/"/"*benchmark/" ': Warming up for 250.00 ms
Benchmarking "*group/"/"*benchmark/" ': Collecting 100 samples in estimated 500.09 ms (26M iterations)
Benchmarking "*group/"/"*benchmark/" ': Analyzing
"*group/"/"*benchmark/" '
time: [19.194 ns 19.247 ns 19.329 ns]

Benchmarking sampling_mode/Auto (short)
Benchmarking sampling_mode/Auto (short): Warming up for 250.00 ms
Benchmarking sampling_mode/Auto (short): Collecting 100 samples in estimated 500.09 ms (23M iterations)
Benchmarking sampling_mode/Auto (short): Analyzing
sampling_mode/Auto (short)
time: [21.106 ns 21.135 ns 21.167 ns]
Benchmarking sampling_mode/Auto (long)
Benchmarking sampling_mode/Auto (long): Warming up for 250.00 ms

Warning: Unable to complete 100 samples in 500.0ms. You may wish to increase target time to 1.0s, or reduce sample count to 40.
Benchmarking sampling_mode/Auto (long): Collecting 100 samples in estimated 1.0056 s (100 iterations)
Benchmarking sampling_mode/Auto (long): Analyzing
sampling_mode/Auto (long)
time: [10.056 ms 10.074 ms 10.110 ms]
Benchmarking sampling_mode/Linear
Benchmarking sampling_mode/Linear: Warming up for 250.00 ms
Benchmarking sampling_mode/Linear: Collecting 100 samples in estimated 500.09 ms (21M iterations)
Benchmarking sampling_mode/Linear: Analyzing
sampling_mode/Linear time: [21.335 ns 21.514 ns 21.743 ns]
Benchmarking sampling_mode/Flat
Benchmarking sampling_mode/Flat: Warming up for 250.00 ms

Warning: Unable to complete 100 samples in 500.0ms. You may wish to increase target time to 1.0s, or reduce sample count to 40.
Benchmarking sampling_mode/Flat: Collecting 100 samples in estimated 1.0057 s (100 iterations)
Benchmarking sampling_mode/Flat: Analyzing
sampling_mode/Flat time: [10.058 ms 10.059 ms 10.061 ms]

Benchmarking throughput/Bytes
Benchmarking throughput/Bytes: Warming up for 250.00 ms
Benchmarking throughput/Bytes: Collecting 100 samples in estimated 593.50 ms (25k iterations)
Benchmarking throughput/Bytes: Analyzing
throughput/Bytes time: [21.568 us 22.007 us 22.585 us]
thrpt: [43.239 MiB/s 44.376 MiB/s 45.279 MiB/s]
Benchmarking throughput/Elem
Benchmarking throughput/Elem: Warming up for 250.00 ms
Benchmarking throughput/Elem: Collecting 100 samples in estimated 503.05 ms (20k iterations)
Benchmarking throughput/Elem: Analyzing
throughput/Elem time: [22.437 us 22.957 us 23.551 us]
thrpt: [43.480 Melem/s 44.606 Melem/s 45.639 Melem/s]

Benchmarking log_scale/sleep (micros)/1
Benchmarking log_scale/sleep (micros)/1: Warming up for 250.00 ms
Benchmarking log_scale/sleep (micros)/1: Collecting 100 samples in estimated 583.26 ms (10k iterations)
Benchmarking log_scale/sleep (micros)/1: Analyzing
log_scale/sleep (micros)/1
time: [54.513 us 55.778 us 57.836 us]
Benchmarking log_scale/sleep (micros)/100
Benchmarking log_scale/sleep (micros)/100: Warming up for 250.00 ms

Warning: Unable to complete 100 samples in 500.0ms. You may wish to increase target time to 776.3ms, enable flat sampling, or reduce sample count to 50.
Benchmarking log_scale/sleep (micros)/100: Collecting 100 samples in estimated 776.29 ms (5050 iterations)
Benchmarking log_scale/sleep (micros)/100: Analyzing
log_scale/sleep (micros)/100
time: [153.95 us 158.89 us 166.79 us]
Benchmarking log_scale/sleep (micros)/10000
Benchmarking log_scale/sleep (micros)/10000: Warming up for 250.00 ms

Warning: Unable to complete 100 samples in 500.0ms. You may wish to increase target time to 1.0s, or reduce sample count to 40.
Benchmarking log_scale/sleep (micros)/10000: Collecting 100 samples in estimated 1.0057 s (100 iterations)
Benchmarking log_scale/sleep (micros)/10000: Analyzing
log_scale/sleep (micros)/10000
time: [10.057 ms 10.204 ms 10.467 ms]

Benchmarking linear_scale/sleep (millis)/1
Benchmarking linear_scale/sleep (millis)/1: Warming up for 250.00 ms
Benchmarking linear_scale/sleep (millis)/1: Collecting 100 samples in estimated 536.54 ms (500 iterations)
Benchmarking linear_scale/sleep (millis)/1: Analyzing
linear_scale/sleep (millis)/1
time: [1.0544 ms 1.0652 ms 1.0864 ms]
Benchmarking linear_scale/sleep (millis)/2
Benchmarking linear_scale/sleep (millis)/2: Warming up for 250.00 ms
Benchmarking linear_scale/sleep (millis)/2: Collecting 100 samples in estimated 616.70 ms (300 iterations)
Benchmarking linear_scale/sleep (millis)/2: Analyzing
linear_scale/sleep (millis)/2
time: [2.0554 ms 2.1236 ms 2.2476 ms]
Benchmarking linear_scale/sleep (millis)/3
Benchmarking linear_scale/sleep (millis)/3: Warming up for 250.00 ms
Benchmarking linear_scale/sleep (millis)/3: Collecting 100 samples in estimated 611.16 ms (200 iterations)
Benchmarking linear_scale/sleep (millis)/3: Analyzing
linear_scale/sleep (millis)/3
time: [3.0552 ms 3.0671 ms 3.0901 ms]

thread 'main' panicked at 'attempt to multiply with overflow', /home/anton/.cargo/registry/src/github.com-1ecc6299db9ec823/criterion-0.3.3/src/routine.rs:237:13
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
Benchmarking forced_zero_time_test
Benchmarking forced_zero_time_test: Warming up for 250.00 ms
thread 'main' panicked at 'Unexpected message FinishedBenchmarkGroup { group: "forced_zero_time_test" }', src/bench_target.rs:302:26
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
thread 'main' panicked at 'called Result::unwrap() on an Err value: IoError(Custom { kind: UnexpectedEof, error: "failed to fill whole buffer" })', /home/anton/.cargo/registry/src/github.com-1ecc6299db9ec823/criterion-0.3.3/src/benchmark_group.rs:378:18
stack backtrace:
0: 0x559bd66bd350 - std::backtrace_rs::backtrace::libunwind::trace::h63b7a90188ab5fb3
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
1: 0x559bd66bd350 - std::backtrace_rs::backtrace::trace_unsynchronized::h80aefbf9b851eca7
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x559bd66bd350 - std::sys_common::backtrace::_print_fmt::hbef05ae4237a4d72
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/sys_common/backtrace.rs:67:5
3: 0x559bd66bd350 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h28abce2fdb9884c2
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/sys_common/backtrace.rs:46:22
4: 0x559bd66e08df - core::fmt::write::h3b84512577ca38a8
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/core/src/fmt/mod.rs:1092:17
5: 0x559bd66b80c2 - std::io::Write::write_fmt::h465f8feea02e2aa1
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/io/mod.rs:1572:15
6: 0x559bd66bf485 - std::sys_common::backtrace::_print::h525280ee0d29bdde
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/sys_common/backtrace.rs:49:5
7: 0x559bd66bf485 - std::sys_common::backtrace::print::h1f0f5b9f3ef8fb78
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/sys_common/backtrace.rs:36:9
8: 0x559bd66bf485 - std::panicking::default_hook::{{closure}}::ha5838f6faa4a5a8f
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panicking.rs:208:50
9: 0x559bd66bef33 - std::panicking::default_hook::hfb9fe98acb0dcb3b
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panicking.rs:225:9
10: 0x559bd66bfbfd - std::panicking::rust_panic_with_hook::hb89f5f19036e6af8
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panicking.rs:591:17
11: 0x559bd66bf767 - std::panicking::begin_panic_handler::{{closure}}::h119e7951427f41da
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panicking.rs:497:13
12: 0x559bd66bd80c - std::sys_common::backtrace::__rust_end_short_backtrace::hce386c44bf47a128
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/sys_common/backtrace.rs:141:18
13: 0x559bd66bf6c9 - rust_begin_unwind
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panicking.rs:493:5
14: 0x559bd66df0b1 - core::panicking::panic_fmt::h2242888e8769cd33
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/core/src/panicking.rs:92:14
15: 0x559bd66deed3 - core::option::expect_none_failed::hb1edf11f73e63728
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/core/src/option.rs:1329:5
16: 0x559bd60c09c1 - core::result::Result<T,E>::unwrap::hdbbcc96a64755104
at /home/anton/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:1037:23
17: 0x559bd60d74ef - <criterion::benchmark_group::BenchmarkGroup as core::ops::drop::Drop>::drop::ha9afe23adb2092ab
at /home/anton/.cargo/registry/src/github.com-1ecc6299db9ec823/criterion-0.3.3/src/benchmark_group.rs:377:13
18: 0x559bd60e62f3 - core::ptr::drop_in_place<criterion::benchmark_group::BenchmarkGroupcriterion::measurement::WallTime>::h24d74e3e03da61c5
at /home/anton/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:187:1
19: 0x559bd60bc509 - criterion::Criterion::bench_function::h4f790dcca79c6a49
at /home/anton/.cargo/registry/src/github.com-1ecc6299db9ec823/criterion-0.3.3/src/lib.rs:1474:59
20: 0x559bd60ea396 - test_benchmarks::zero_test::hcdcd0ffda4ea7690
at /home/anton/gitrepos/cargo-criterion/integration_tests/benches/test_benchmarks.rs:93:5
21: 0x559bd60ea53c - test_benchmarks::benches::h6161147846fe1bbd
at /home/anton/.cargo/registry/src/github.com-1ecc6299db9ec823/criterion-0.3.3/src/macros.rs:71:17
22: 0x559bd60ea5fc - test_benchmarks::main::h94deb0b0f36d0f27
at /home/anton/.cargo/registry/src/github.com-1ecc6299db9ec823/criterion-0.3.3/src/macros.rs:124:17
23: 0x559bd60e61bb - core::ops::function::FnOnce::call_once::h1741d060fd330d97
at /home/anton/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
24: 0x559bd60e596e - std::sys_common::backtrace::__rust_begin_short_backtrace::hd8912e8360ba1189
at /home/anton/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125:18
25: 0x559bd60c2d41 - std::rt::lang_start::{{closure}}::h39b017130ca5f5ac
at /home/anton/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:66:18
26: 0x559bd66c00fa - core::ops::function::impls::<impl core::ops::function::FnOnce for &F>::call_once::h44574effd2120c86
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/core/src/ops/function.rs:259:13
27: 0x559bd66c00fa - std::panicking::try::do_call::h10b0bd4879c8dfb0
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panicking.rs:379:40
28: 0x559bd66c00fa - std::panicking::try::h60c6780d33419e92
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panicking.rs:343:19
29: 0x559bd66c00fa - std::panic::catch_unwind::h111f33e08c52e2ce
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panic.rs:431:14
30: 0x559bd66c00fa - std::rt::lang_start_internal::h126f2e09345dbfda
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/rt.rs:51:25
31: 0x559bd60c2d20 - std::rt::lang_start::h7581d7c526af79fb
at /home/anton/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:65:5
32: 0x559bd60ea68c - main
33: 0x7f84e0da70b3 - __libc_start_main
34: 0x559bd60a380e - _start
35: 0x0 -
thread panicked while panicking. aborting.
test test_json_message_format ... FAILED

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

running 1 test
test bencher_test ... bench: 14,567 ns/iter (+/- 5,231)

test result: ok. 0 passed; 0 failed; 0 ignored; 1 measured

Warning: Unable to complete 100 samples in 500.0ms. You may wish to increase target time to 1.0s, or reduce sample count to 40.

Warning: Unable to complete 100 samples in 500.0ms. You may wish to increase target time to 1.0s, or reduce sample count to 40.

Warning: Unable to complete 100 samples in 500.0ms. You may wish to increase target time to 780.0ms, enable flat sampling, or reduce sample count to 50.

Warning: Unable to complete 100 samples in 500.0ms. You may wish to increase target time to 1.0s, or reduce sample count to 40.
Blocking waiting for file lock on package cache
Blocking waiting for file lock on package cache
Blocking waiting for file lock on package cache
Blocking waiting for file lock on build directory
Finished test [unoptimized + debuginfo] target(s) in 0.97s
Benchmarking norm
Benchmarking norm: Warming up for 250.00 ms
Benchmarking norm: Collecting 100 samples in estimated 506.50 ms (288k iterations)
Benchmarking norm: Analyzing
norm time: [1.7485 us 1.7689 us 1.8061 us]

Benchmarking "*group/"/"*benchmark/" '
Benchmarking "*group/"/"*benchmark/" ': Warming up for 250.00 ms
Benchmarking "*group/"/"*benchmark/" ': Collecting 100 samples in estimated 500.09 ms (26M iterations)
Benchmarking "*group/"/"*benchmark/" ': Analyzing
"*group/"/"*benchmark/" '
time: [20.303 ns 21.215 ns 22.171 ns]

Benchmarking sampling_mode/Auto (short)
Benchmarking sampling_mode/Auto (short): Warming up for 250.00 ms
Benchmarking sampling_mode/Auto (short): Collecting 100 samples in estimated 500.03 ms (24M iterations)
Benchmarking sampling_mode/Auto (short): Analyzing
sampling_mode/Auto (short)
time: [22.399 ns 23.482 ns 24.617 ns]
Benchmarking sampling_mode/Auto (long)
Benchmarking sampling_mode/Auto (long): Warming up for 250.00 ms
Benchmarking sampling_mode/Auto (long): Collecting 100 samples in estimated 1.0059 s (100 iterations)
Benchmarking sampling_mode/Auto (long): Analyzing
sampling_mode/Auto (long)
time: [10.058 ms 10.179 ms 10.330 ms]
Benchmarking sampling_mode/Linear
Benchmarking sampling_mode/Linear: Warming up for 250.00 ms
Benchmarking sampling_mode/Linear: Collecting 100 samples in estimated 500.02 ms (24M iterations)
Benchmarking sampling_mode/Linear: Analyzing
sampling_mode/Linear time: [22.459 ns 23.576 ns 24.778 ns]
Benchmarking sampling_mode/Flat
Benchmarking sampling_mode/Flat: Warming up for 250.00 ms
Benchmarking sampling_mode/Flat: Collecting 100 samples in estimated 1.0060 s (100 iterations)
Benchmarking sampling_mode/Flat: Analyzing
sampling_mode/Flat time: [10.057 ms 10.058 ms 10.059 ms]

Benchmarking throughput/Bytes
Benchmarking throughput/Bytes: Warming up for 250.00 ms
Benchmarking throughput/Bytes: Collecting 100 samples in estimated 623.95 ms (25k iterations)
Benchmarking throughput/Bytes: Analyzing
throughput/Bytes time: [22.016 us 22.405 us 22.950 us]
thrpt: [42.551 MiB/s 43.586 MiB/s 44.357 MiB/s]
Benchmarking throughput/Elem
Benchmarking throughput/Elem: Warming up for 250.00 ms
Benchmarking throughput/Elem: Collecting 100 samples in estimated 623.25 ms (25k iterations)
Benchmarking throughput/Elem: Analyzing
throughput/Elem time: [24.016 us 24.634 us 25.428 us]
thrpt: [40.271 Melem/s 41.569 Melem/s 42.638 Melem/s]

Benchmarking log_scale/sleep (micros)/1
Benchmarking log_scale/sleep (micros)/1: Warming up for 250.00 ms
Benchmarking log_scale/sleep (micros)/1: Collecting 100 samples in estimated 555.72 ms (10k iterations)
Benchmarking log_scale/sleep (micros)/1: Analyzing
log_scale/sleep (micros)/1
time: [54.571 us 57.193 us 61.113 us]
Benchmarking log_scale/sleep (micros)/100
Benchmarking log_scale/sleep (micros)/100: Warming up for 250.00 ms
Benchmarking log_scale/sleep (micros)/100: Collecting 100 samples in estimated 779.99 ms (5050 iterations)
Benchmarking log_scale/sleep (micros)/100: Analyzing
log_scale/sleep (micros)/100
time: [155.02 us 158.24 us 162.85 us]
Benchmarking log_scale/sleep (micros)/10000
Benchmarking log_scale/sleep (micros)/10000: Warming up for 250.00 ms
Benchmarking log_scale/sleep (micros)/10000: Collecting 100 samples in estimated 1.0139 s (100 iterations)
Benchmarking log_scale/sleep (micros)/10000: Analyzing
log_scale/sleep (micros)/10000
time: [10.060 ms 10.061 ms 10.063 ms]

Benchmarking linear_scale/sleep (millis)/1
Benchmarking linear_scale/sleep (millis)/1: Warming up for 250.00 ms
Benchmarking linear_scale/sleep (millis)/1: Collecting 100 samples in estimated 527.41 ms (500 iterations)
Benchmarking linear_scale/sleep (millis)/1: Analyzing
linear_scale/sleep (millis)/1
time: [1.0557 ms 1.0567 ms 1.0582 ms]
Benchmarking linear_scale/sleep (millis)/2
Benchmarking linear_scale/sleep (millis)/2: Warming up for 250.00 ms
Benchmarking linear_scale/sleep (millis)/2: Collecting 100 samples in estimated 616.68 ms (300 iterations)
Benchmarking linear_scale/sleep (millis)/2: Analyzing
linear_scale/sleep (millis)/2
time: [2.0559 ms 2.0563 ms 2.0568 ms]
Benchmarking linear_scale/sleep (millis)/3
Benchmarking linear_scale/sleep (millis)/3: Warming up for 250.00 ms
Benchmarking linear_scale/sleep (millis)/3: Collecting 100 samples in estimated 611.16 ms (200 iterations)
Benchmarking linear_scale/sleep (millis)/3: Analyzing
linear_scale/sleep (millis)/3
time: [3.0566 ms 3.0573 ms 3.0577 ms]

thread 'main' panicked at 'attempt to multiply with overflow', /home/anton/.cargo/registry/src/github.com-1ecc6299db9ec823/criterion-0.3.3/src/routine.rs:237:13
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
Benchmarking forced_zero_time_test
Benchmarking forced_zero_time_test: Warming up for 250.00 ms
thread 'main' panicked at 'Unexpected message FinishedBenchmarkGroup { group: "forced_zero_time_test" }', src/bench_target.rs:302:26
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
thread 'main' panicked at 'called Result::unwrap() on an Err value: IoError(Custom { kind: UnexpectedEof, error: "failed to fill whole buffer" })', /home/anton/.cargo/registry/src/github.com-1ecc6299db9ec823/criterion-0.3.3/src/benchmark_group.rs:378:18
stack backtrace:
0: 0x5633d04aa350 - std::backtrace_rs::backtrace::libunwind::trace::h63b7a90188ab5fb3
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
1: 0x5633d04aa350 - std::backtrace_rs::backtrace::trace_unsynchronized::h80aefbf9b851eca7
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x5633d04aa350 - std::sys_common::backtrace::_print_fmt::hbef05ae4237a4d72
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/sys_common/backtrace.rs:67:5
3: 0x5633d04aa350 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h28abce2fdb9884c2
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/sys_common/backtrace.rs:46:22
4: 0x5633d04cd8df - core::fmt::write::h3b84512577ca38a8
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/core/src/fmt/mod.rs:1092:17
5: 0x5633d04a50c2 - std::io::Write::write_fmt::h465f8feea02e2aa1
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/io/mod.rs:1572:15
6: 0x5633d04ac485 - std::sys_common::backtrace::_print::h525280ee0d29bdde
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/sys_common/backtrace.rs:49:5
7: 0x5633d04ac485 - std::sys_common::backtrace::print::h1f0f5b9f3ef8fb78
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/sys_common/backtrace.rs:36:9
8: 0x5633d04ac485 - std::panicking::default_hook::{{closure}}::ha5838f6faa4a5a8f
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panicking.rs:208:50
9: 0x5633d04abf33 - std::panicking::default_hook::hfb9fe98acb0dcb3b
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panicking.rs:225:9
10: 0x5633d04acbfd - std::panicking::rust_panic_with_hook::hb89f5f19036e6af8
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panicking.rs:591:17
11: 0x5633d04ac767 - std::panicking::begin_panic_handler::{{closure}}::h119e7951427f41da
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panicking.rs:497:13
12: 0x5633d04aa80c - std::sys_common::backtrace::__rust_end_short_backtrace::hce386c44bf47a128
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/sys_common/backtrace.rs:141:18
13: 0x5633d04ac6c9 - rust_begin_unwind
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panicking.rs:493:5
14: 0x5633d04cc0b1 - core::panicking::panic_fmt::h2242888e8769cd33
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/core/src/panicking.rs:92:14
15: 0x5633d04cbed3 - core::option::expect_none_failed::hb1edf11f73e63728
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/core/src/option.rs:1329:5
16: 0x5633cfead9c1 - core::result::Result<T,E>::unwrap::hdbbcc96a64755104
at /home/anton/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:1037:23
17: 0x5633cfec44ef - <criterion::benchmark_group::BenchmarkGroup as core::ops::drop::Drop>::drop::ha9afe23adb2092ab
at /home/anton/.cargo/registry/src/github.com-1ecc6299db9ec823/criterion-0.3.3/src/benchmark_group.rs:377:13
18: 0x5633cfed32f3 - core::ptr::drop_in_place<criterion::benchmark_group::BenchmarkGroupcriterion::measurement::WallTime>::h24d74e3e03da61c5
at /home/anton/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:187:1
19: 0x5633cfea9509 - criterion::Criterion::bench_function::h4f790dcca79c6a49
at /home/anton/.cargo/registry/src/github.com-1ecc6299db9ec823/criterion-0.3.3/src/lib.rs:1474:59
20: 0x5633cfed7396 - test_benchmarks::zero_test::hcdcd0ffda4ea7690
at /home/anton/gitrepos/cargo-criterion/integration_tests/benches/test_benchmarks.rs:93:5
21: 0x5633cfed753c - test_benchmarks::benches::h6161147846fe1bbd
at /home/anton/.cargo/registry/src/github.com-1ecc6299db9ec823/criterion-0.3.3/src/macros.rs:71:17
22: 0x5633cfed75fc - test_benchmarks::main::h94deb0b0f36d0f27
at /home/anton/.cargo/registry/src/github.com-1ecc6299db9ec823/criterion-0.3.3/src/macros.rs:124:17
23: 0x5633cfed31bb - core::ops::function::FnOnce::call_once::h1741d060fd330d97
at /home/anton/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
24: 0x5633cfed296e - std::sys_common::backtrace::__rust_begin_short_backtrace::hd8912e8360ba1189
at /home/anton/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125:18
25: 0x5633cfeafd41 - std::rt::lang_start::{{closure}}::h39b017130ca5f5ac
at /home/anton/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:66:18
26: 0x5633d04ad0fa - core::ops::function::impls::<impl core::ops::function::FnOnce for &F>::call_once::h44574effd2120c86
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/core/src/ops/function.rs:259:13
27: 0x5633d04ad0fa - std::panicking::try::do_call::h10b0bd4879c8dfb0
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panicking.rs:379:40
28: 0x5633d04ad0fa - std::panicking::try::h60c6780d33419e92
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panicking.rs:343:19
29: 0x5633d04ad0fa - std::panic::catch_unwind::h111f33e08c52e2ce
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/panic.rs:431:14
30: 0x5633d04ad0fa - std::rt::lang_start_internal::h126f2e09345dbfda
at /rustc/9bc8c42bb2f19e745a63f3445f1ac248fb015e53/library/std/src/rt.rs:51:25
31: 0x5633cfeafd20 - std::rt::lang_start::h7581d7c526af79fb
at /home/anton/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:65:5
32: 0x5633cfed768c - main
33: 0x7f6b2b6e80b3 - __libc_start_main
34: 0x5633cfe9080e - _start
35: 0x0 -
thread panicked while panicking. aborting.
test test_cargo_criterion_plotters ... FAILED

failures:

---- test_json_message_format stdout ----
Running cargo-criterion...
Failed to run cargo-cargo criterion.
stdout:
stderr:
thread 'test_json_message_format' panicked at 'Failed to run cargo-criterion.', tests/integration_test.rs:33:9
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

---- test_cargo_criterion_plotters stdout ----
Running cargo-criterion...
Failed to run cargo-cargo criterion.
stdout:
stderr:
thread 'test_cargo_criterion_plotters' panicked at 'Failed to run cargo-criterion.', tests/integration_test.rs:33:9

failures:
test_cargo_criterion_plotters
test_json_message_format

test result: FAILED. 0 passed; 2 failed; 1 ignored; 0 measured; 0 filtered out; finished in 25.67s

error: test failed, to rerun pass '--test integration_test'

I fixed this in my fork, I also updated and forked criterion, updated the plotters dependency because of this criterion issue and fixed clippy errors. I didn't set up a PR because CONTRIBUTING.md mentions you like to update dependencies yourself.

Building benchmarks with external libraries

For example, Cargo.toml:

[package]
name = "tchbench"
version = "0.1.0"
authors = ["Andre Bogus <[email protected]>"]
edition = "2018"

[dependencies]
tch = "0.1.7"
criterion = "0.3.2"

[[bench]]
name = "bench"
path = "benches/bench.rs"
harness = false

And benches/bench.rs:

use criterion::{black_box, criterion_group, criterion_main, Criterion};
use tch::Tensor;

fn bench_norm(c: &mut Criterion) {
    let t = &Tensor::of_slice(&[0_f32, 0.1f32, 0.5f32, 0.9f32]);
    c.bench_function("norm", |b| b.iter(|| black_box(t).norm()));
}

criterion_group! {
    name = benches;
    config = Criterion::default().sample_size(10);
    targets = bench_norm
}

criterion_main!(benches);

Running with cargo bench works, whereas running with cargo criterion fails with

/path/to/bench/target/release/deps/bench-6d8fcb9c538c7b9c: error while loading shared libraries: libtorch_cpu.so: cannot open shared object file: No such file or directory
Error: Non-Criterion.rs benchmark target bench exited with error code Some(127)

[Feature] Add condensed plot comparing performance of different functions to summary report

Feature Enhancement

I was wondering if you would be interested in giving results with geometric and charting it similiar to how benchmarksgame does their programming language benchmarks?

Allow selection of plots / suppression of iteration times

I've been using a large sample size (100k to get a time distribution with randomised parameters), and the generated plots get large (plots including all iteration times for one benchmark are around 10 MiB, the sum of outputs for one benchmark group is 1.4 GiB). I don't need all of the plots, they take significant time to generate, and attempting to load index.html for a group and scroll down the page crashes Firefox. But I do want some of the plots (at least the main violin.svg comparing the PDF of each method).

Iteration times aren't really useful with this many samples and could be disabled automatically. This, and everything including them (pdf.svg), account for almost all the output size, so simply having an on/off switch for iteration times should be enough.

I guess the correct place for this configuration would be cargo-criterion command-line parameters.

Support open metrics format as output

I'm using GitLab and perform benchmarks on multiple rust tools. GitLab support the openmetrics format so it would be useful if it would be directly supported as a cargo-criterion output format.

https://openmetrics.io
https://docs.gitlab.com/ee/ci/testing/metrics_reports.html

Add tag for new release

The last tag is about a year old, could you please add another tag?

Error: unable to open connection to bench target

D:\repos\cad97\playground> cargo criterion      
    Finished bench [optimized] target(s) in 0.19s
Error: Unable to open connection to bench target main

Caused by:
Gnuplot not found, using plotters backend
    A non-blocking socket operation could not be completed immediately. (os error 10035)thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: IoError(Os { code: 10054, kind: ConnectionReset, message: "An existing connection was forcibly closed by the remote host." })

', Stack backtrace:
D:\.rust\cargo\registry\src\github.com-1ecc6299db9ec823\criterion-0.3.4\src\lib.rs   0: std::backtrace_rs::backtrace::dbghelp::trace
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\..\..\backtrace\src\backtrace\dbghelp.rs:98
   1: std::backtrace_rs::backtrace::trace_unsynchronized
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\..\..\backtrace\src\backtrace\mod.rs:66
   2: std::backtrace::Backtrace::create
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\backtrace.rs:327
   3: std::backtrace::Backtrace::capture
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\backtrace.rs:295
   4: cargo_criterion::connection::Connection::new
   5: cargo_criterion::bench_target::BenchTarget::execute
   6: cargo_criterion::stats::bivariate::Data<X,Y>::bootstrap
   7: std::sys_common::backtrace::__rust_begin_short_backtrace
   8: std::rt::lang_start::{{closure}}
   9: core::ops::function::impls::{{impl}}::call_once
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\library\core\src\ops\function.rs:280
  10: std::panicking::try::do_call
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\panicking.rs:379
  11: std::panicking::try
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\panicking.rs:343
  12: std::panic::catch_unwind
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\panic.rs:431
  13: std::rt::lang_start_internal
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\rt.rs:51
  14: main
  15: invoke_main
             at d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
  16: __scrt_common_main_seh
             at d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
  17: BaseThreadInitThunk
  18: RtlUserThreadStart:
1105:18
stack backtrace:
   0: std::panicking::begin_panic_handler
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\std\src\panicking.rs:493
   1: core::panicking::panic_fmt
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\core\src\panicking.rs:92
   2: core::option::expect_none_failed
             at /rustc/e37a13cc3594004663738bd18d8100e6db9666cf\/library\core\src\option.rs:1300
   3: criterion::Criterion<M>::benchmark_group
   4: criterion::report::_::<impl serde::ser::Serialize for criterion::report::BenchmarkId>::serialize
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

I'm not sure what happened here, I just installed cargo-criterion and did cargo criterion.

OS: Windows
Toolchain: nightly-x86_64-pc-windows-msvc (directory override for '\?\D:\repos\cad97\playground')
rustc 1.52.0-nightly (e37a13cc3 2021-02-28)
Version: cargo-criterion 1.0.1
Criterion: 0.3.4

significance level can only be shown in 2-digits

have set group.sample_size(512).significance_level(0.001);, result like this:

    Finished bench [optimized] target(s) in 5.23s

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Gnuplot not found, using plotters backend
example/A                  time:   [200.41 us 200.89 us 201.38 us]                    
                        change: [+1.9058% +2.1252% +2.3820%] (p = 0.00 < 0.00)
                        Performance has regressed.
example/B                   time:   [209.62 us 209.76 us 209.92 us]                      
                        change: [+0.5592% +0.6910% +0.8199%] (p = 0.00 < 0.00)
                        Change within noise threshold.
example/C                   time:   [210.75 us 211.23 us 211.77 us]                        
                        change: [+4.1680% +4.4006% +4.6917%] (p = 0.00 < 0.00)
                        Performance has regressed.
example/D                   time:   [1.5939 ms 1.6003 ms 1.6079 ms]                      
                        change: [+0.2976% +0.7865% +1.2899%] (p = 0.00 < 0.00)
                        Change within noise threshold.

all (p = 0.00 < 0.00)

cargo-criterion panics

I ran benchmarks using cargo-criterion but encountered the following error message. The panic happens randomly inside the same Docker environment.

Killed
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Io(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })', /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/criterion-0.5.1/src/analysis/mod.rs:107:18
stack backtrace:
   0:     0x55d288dea69a - <unknown>
   1:     0x55d288e11a9f - <unknown>
   2:     0x55d288de6ac5 - <unknown>
   3:     0x55d288dea465 - <unknown>
   4:     0x55d288debb0e - <unknown>
   5:     0x55d288deb8b5 - <unknown>
   6:     0x55d288dec06e - <unknown>
   7:     0x55d288debf69 - <unknown>
   8:     0x55d288deab06 - <unknown>
   9:     0x55d288debcc2 - <unknown>
  10:     0x55d2889e1643 - <unknown>
  11:     0x55d2889e1af3 - <unknown>
  12:     0x55d288a6d83f - <unknown>
  13:     0x55d288a2b2ac - <unknown>
  14:     0x55d288a4746d - <unknown>
  15:     0x55d288a70e2f - <unknown>
  16:     0x55d288a3b883 - <unknown>
  17:     0x55d288a398a9 - <unknown>
  18:     0x55d288de230e - <unknown>
  19:     0x55d288a70ec5 - <unknown>
  20:     0x7fca4a198d90 - <unknown>
  21:     0x7fca4a198e40 - __libc_start_main
  22:     0x55d2889e1c85 - <unknown>
  23:                0x0 - <unknown>
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Io(Os { code: 32, kind: BrokenPipe, message: "Broken pipe" })', /github/home/.cargo/registry/src/index.crates.io-6f17d22bba1[500](https://github.com/fun-poker-game/poker-rs/actions/runs/5448302538/jobs/9917901310?pr=1677#step:4:501)1f/criterion-0.5.1/src/benchmark_group.rs:377:14
stack backtrace:
   0:     0x55d288dea69a - <unknown>
   1:     0x55d288e11a9f - <unknown>
   2:     0x55d288de6ac5 - <unknown>
   3:     0x55d288dea465 - <unknown>
   4:     0x55d288debb0e - <unknown>
   5:     0x55d288deb8b5 - <unknown>
   6:     0x55d288dec06e - <unknown>
   7:     0x55d288debf69 - <unknown>
   8:     0x55d288deab06 - <unknown>
   9:     0x55d288debcc2 - <unknown>
  10:     0x55d2889e1643 - <unknown>
  11:     0x55d2889e1af3 - <unknown>
  12:     0x55d288a17d39 - <unknown>
  13:     0x55d288a55071 - <unknown>
  14:     0x55d288a52e46 - <unknown>
  15:     0x55d288a70e2f - <unknown>
  16:     0x55d288a3b883 - <unknown>
  17:     0x55d288a398a9 - <unknown>
  18:     0x55d288de230e - <unknown>
  19:     0x55d288a70ec5 - <unknown>
  20:     0x7fca4a198d90 - <unknown>
  21:     0x7fca4a198e40 - __libc_start_main
  22:     0x55d2889e1c85 - <unknown>
  23:                0x0 - <unknown>
thread panicked while panicking. aborting.

Better default colors for plots

I accidentally opened this issue bheisler/criterion.rs#547 on criterion.rs instead of here. #11 is also another similar issue.

The problem seems to be that https://github.com/bheisler/cargo-criterion/blob/main/src/config.rs#L25-L34 defines the colors as

static COMPARISON_COLORS: [Color; NUM_COLORS] = [
    Color { r: 8, g: 34, b: 34 },
    Color { r: 6, g: 139, b: 87 },
    Color { r: 0, g: 139, b: 139 },
    Color { r: 5, g: 215, b: 0 },
    Color { r: 0, g: 0, b: 139 },
    Color { r: 0, g: 20, b: 60 },
    Color { r: 9, g: 0, b: 139 },
    Color { r: 0, g: 255, b: 127 },
];

instead of https://github.com/bheisler/criterion.rs/blob/master/src/plot/plotters_backend/summary.rs#L12-L21

static COMPARISON_COLORS: [RGBColor; NUM_COLORS] = [
    RGBColor(178, 34, 34),
    RGBColor(46, 139, 87),
    RGBColor(0, 139, 139),
    RGBColor(255, 215, 0),
    RGBColor(0, 0, 139),
    RGBColor(220, 20, 60),
    RGBColor(139, 0, 139),
    RGBColor(0, 255, 127),
];

Probably in the copy-paste the red channel got somehow reduced modulo 10.

consider drop more outlier?

Hi, would you please consider remove those obvious disturbed outlier sample from the result of final calculation?

assertion failed: slice.len() > 1 && slice.iter().all(|x| !x.is_nan())

when trying to use cargo criterion (on a clean repo and after cargo bench) I am getting:

thread 'main' panicked at 'assertion failed: slice.len() > 1 && slice.iter().all(|x| !x.is_nan())', /home/mozilla/.cargo/registry/src/github.com-1ecc6299db9ec823/cargo-criterion-1.0.0/src/stats/univariate/sample.rs:29:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: IoError(Os { code: 32, kind: BrokenPipe, message: "Broken pipe" })', /home/mozilla/.cargo/registry/src/github.com-1ecc6299db9ec823/criterion-0.3.3/src/benchmark.rs:573:26
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

This tool doesn't work on M1 (aarch64 architecture) Macs.

Hi! A non-blocker, but I've run into a snag where I have some Criterion tests, and the cargo criterion command fails on my M1 Mac but not on an Ubuntu VM.

Rust's support for M1 macs is laid out in rust-lang/rust#73908

Thanks for building criterion! It's an invaluable tool.

cargo-criterion does not output ndJSON

On our project, htsget-rs, we generate single benchmark outputs that formatted fine as JSON

On the other hand, multiple (group) outputs seems to be blindly concatenated, generating a broken, invalid JSON as a result:

Is this by design for some reason or simply an oversight that I could fix and pullrequest, @bheisler?

Graph with the whole previous history

Hey!

Just a thought that arose in my mind: as criterion does quite a few pretty graphs, and already does comparison between the last and current run… do you think it'd make sense for criterion to be able to keep the whole history of benchmarks and graph it all together?

I'm thinking of a use case that'd be having a graph of the performance of the system over all the CI builds, which would I think be buildable with this :)

If you think it's not reasonable to put that in criterion, it'd totally make sense! It's probably possible to use criterion's CSV output to generate it :)

Anyway, thank you for this great piece of software!

Configuration Method for COMPARISON_COLORS

Maybe this is my problem, but I hardly distinguish the first 2 colors in the COMPARISON_COLORS array (https://github.com/bheisler/criterion.rs/blob/ce1738068c9409ae5940d5a7453d8ac015afa490/src/plot/gnuplot_backend/summary.rs#L15)
I would like to be able to manually set the colors for the plotter

[Suggestion] Stop sorting group results by BenchmarkId string representation

I currently use Criterion to compare how a generic function performs for types of different sizes. Kind of like this:

let mut group = c.benchmark_group("create_channel");
group.bench_function(stringify!(()), |b| b.iter(channel::<()>);
group.bench_function(stringify!(u8), |b| b.iter(channel::<u8>);
group.bench_function(stringify!(usize), |b| b.iter(channel::<usize>);
group.bench_function(stringify!([u8; 64]), |b| b.iter(channel::<[u8; 64]>);
group.bench_function(stringify!([u8; 4096]), |b| b.iter(channel::<[u8; 4096]>);
group.finish();

//... from my library:
fn channel<T>() -> (Sender<T>, Receiver<T>)

However. In the written report the tests are sorted like the following. Both in the html and in the group's violin plot:

• ()
• [u8; 4096]
• [u8; 64]
• u8
• usize

I assume because that's how they sort alphabetically. I had my tests in an order for a reason: The size of the type is increasing. The only way around this I can think of currently is to rename my BenchmarkIds to something like [u8; 0000], [u8; 0001], [u8; 0064], [u8;4096] to get what I want. But this does not look nearly as nice IMO. And clippy will complain that I have zero_prefixed_literals.

Is it not expected that users will write their benchmarks in the order they want? What's the use case where it would be hard to write the code in the same order as you would want the results?

Maybe it would be nice with a flag to set on the Criterion or BenchmarkGroup instances to toggle this behavior?

Unable to open socket to connect to Criterion.rs

I get the following error when running cargo criterion:

Error: Unable to open socket to connect to Criterion.rs

Caused by:
    Cannot assign requested address (os error 99)

cargo bench works as expected. I'm on cargo-criterion 1.1.0, rustc 1.56, criterion 0.3.5, running on arch linux.

[Feature request] Tagging and comparing benchmarks

I find myself trying to analyze the exact changes that improve the code change. It's very troublesome to make a change, see it doesn't improve speeds or worsens them, then have to revert to HEAD to start a benchmark again (otherwise, any next change will be compared to the immediately previous one, which might have made it worse, which means reverting would have FIXED the issue but returned it to the HEAD state, leading me to falsely believe I improved speed).

It might be helpful to have a way to tag a benchmark like so: cargo criterion --name head

Then, each subsequent change can be compared to it with: cargo criterion --compare-to head (rather than the immediately previous test)

FR: Support constant/user-supplied data

Thanks for your work on this tool!

I'd like to be able to draw an extra line on the plots generated by cargo criterion. A couple use cases I can imagine:

• I know some theoretical maximum performance that I'm trying to get close to (why I bring it up). For example, my function processes something at X bytes/s, and my CPU is capable of Y bytes/s. I'd like to get a sense in the combined plots of how close I am.

  For this, I'd just like to provide a single throughput rate and have a straight line drawn on a (hypothetical) throughput plot.

• I have some objective I'm trying to hit (timeout, etc.).

  This would be a horizontal line on a benchmark; again, I'd like to just provide a number.

Vaguely related to #9, #10.

Potential workarounds:

• Add a "comparison function" sleeps for the appropriate time. I tried this and it's really noisy (not shocking).
• Hand-generate the data and sticking it in the appropriate location in target/criterion. I'm thinking about doing this.

Chart throughput on parameterized benchmark page by default?

I am using Criterion to benchmark a function that operates on a vector of elements, using something like:

const ALL: &'static [Width; 4] = &[Width::U8, Width::U16, Width::U32, Width::U64];
const ELTS: usize = 20;

fn encode_multiple_benchmark(c: &mut Criterion) {
    let id = "encode_multiple";
    let bm = ParameterizedBenchmark::new(
        id,
        |b, ref n| {
            b.iter(|| {
                let v = vec![n.sample() as u64; ELTS];
                v.fib_encode().expect("should be encodable")
            })
        },
        ALL,
    ).throughput(|_s| Throughput::Elements(ELTS as u32));
    c.bench(id, bm);
}

The benchmark overview page shows the duration that each iteration took, but that number is a bit useless on its own. The thing I'm really interested in over time is the throughput of that function, which is only given under "Additional Statistics" on the details page.

It would be really nice if the benchmark was configurable to show that throughput on the parameterized benchmark's overview page by default.

Benchmarking on remote maschine / Report generation off-line

Thanks a lot for your great benchmarking library!

Now that analyzing and plotting of results has been moved to cargo-criterion, I wonder if there is already some easy way to do benchmarks on a remote machine. Currently, I'm cross-compiling my criterion benchmarks for a Raspberry Pi Zero via cargo bench --target arm-unknown-linux-gnueabihf which successfully generates binaries that I can run on the Pi with --bench. Due to the very limited resources of this device I don't want to install rust and cargo on the Pi itself and thus cannot use cargo-criterion directly. Also report generation (especially plotting) is very slow on the Pi, so I would like to shift that part of the task to my work machine.

I already know that I can disable plot generation via --noplot. Is there some way to generate the reports afterwards on another computer using the raw data from the Pi? Or even better, does cargo-criterion support benchmarking on a remote machine?

Colorblindness Accessibility

Several users have commented that they find it difficult to tell apart the different colors used by the plots, especially in the line chart. I've already made the colors configurable, but that alone might not be enough.

Also, there are only about ~8 colors by default in cargo-criterion's set, so if there are more than 8 functions in a group it would be impossible for even people who aren't colorblind to tell them apart.

I'm not sure what to do to improve on this, but it is a known problem.

Add option to wrap benchmark target command

Criterion.rs benchmarks support a --profile-time option for doing profiling. However, cargo bench offers no way to run the benchmarks in a profiler, so users have to do that manually.

cargo-criterion could allow the user to provide a command template. For example:
cargo criterion --run-with 'perf record -o %TARGET%-perf.data -g %ARGS%' -- --profile-time 120 my_benchmark_name

This would replace %TARGET% with the benchmark target name and %ARGS% with the benchmark target executable and its arguments, then invoke the resulting command.

Stuck when running non-criterion benchmarks.

I didn't test #34 well enough and now cargo-criterion blocks indefinitely when running non-criterion benchmarks. Previously, it would busy-wait (ie. use 100% CPU) while waiting for non-criterion benchmarks to finish.

Would like to use tokio here but maybe we can temporarily fix it with a mutex.

Difference between 'cargo bench' and 'cargo criterion'

I'd like to use cargo-criterion instead of cargo bench, but it seems that running cargo criterion does not add in the report a comparison with the previous run in case of regression (and maybe improvement?)

Is there something extra that I missed that needs to be done to get this comparison in the report? Or is it not supported?

Thanks!

how do results got such unstable?

For same code untouched, it went with very different.

• swapped order of testing functions after first running.
• this is the same server with no other job running at the same time.

user@thisisthehost0 ~/proj (main)# cargo criterion   
   Compiling proj v0.2.1 (/home/user/proj)                                                   
    Finished bench [optimized] target(s) in 8.66s                                           
                                                                                            
running 0 tests                                                                             
                                                                                            
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Gnuplot not found, using plotters backend     
func/AC                 time:   [201.00 us 201.91 us 202.87 us]                             
                        change: [-1.1117% -0.6052% -0.0672%] (p = 0.02 > 0.00)              
                        No change in performance detected.                                  
func/AC-B               time:   [203.46 us 204.87 us 206.34 us]                             
                        change: [+5.0010% +5.7843% +6.4462%] (p = 0.00 < 0.00)                                                                                                          
                        Performance has regressed.                                          
                                              
user@thisisthehost0 ~/proj (main)# cargo criterion                                                 
   Compiling proj v0.2.1 (/home/user/proj)                                                   
    Finished bench [optimized] target(s) in 6.29s                                           
                                                                                            
running 0 tests                                                                             
                                                                                            
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Gnuplot not found, using plotters backend
func/AC-B               time:   [197.71 us 198.60 us 199.54 us]                       
                        change: [-3.8500% -3.0601% -2.2261%] (p = 0.00 < 0.00)
                        Performance has improved.
func/AC                 time:   [333.45 us 334.59 us 335.66 us]                     
                        change: [+64.734% +65.709% +66.591%] (p = 0.00 < 0.00)
                        Performance has regressed.

user@thisisthehost0 ~/proj (main)# cargo criterion
    Finished bench [optimized] target(s) in 0.04s

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Gnuplot not found, using plotters backend
func/AC-B               time:   [189.62 us 189.70 us 189.79 us]                       
                        change: [-4.9340% -4.4826% -4.0451%] (p = 0.00 < 0.00)
                        Performance has improved.
func/AC                 time:   [338.77 us 339.15 us 339.50 us]                     
                        change: [+1.0216% +1.3634% +1.7241%] (p = 0.00 < 0.00)
                        Performance has regressed.

user@thisisthehost0 ~/proj (main)# cargo criterion
    Finished bench [optimized] target(s) in 0.06s

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Gnuplot not found, using plotters backend
func/AC-B               time:   [342.98 us 343.28 us 343.54 us]                       
                        change: [+80.794% +80.955% +81.111%] (p = 0.00 < 0.00)
                        Performance has regressed.
func/AC                 time:   [343.10 us 343.38 us 343.64 us]                     
                        change: [+1.1138% +1.2482% +1.3831%] (p = 0.00 < 0.00)
                        Performance has regressed.

user@thisisthehost0 ~/proj (main)# 

Should cargo-criterion support baselines?

I had this idea kicking around that baselines could be replaced with alternate "timelines" (since cargo-criterion will hopefully soon support a full-history chart).

I was never very satisfied with the workflow of Criterion.rs' baselines (and others seem to agree on that - eg. critcmp largely exists to make up for deficiencies in the workflow supported by baselines). Thing is, I have no idea what sort of workflow would work better.

This will require some design work. Probably won't be available in 1.0.0.

Criterion.toml vs criterion.toml

Hi, the user's guide states the configuration file is named criterion.toml but cargo-criterion actually looks for Criterion.toml.

I think using the lowercase filename makes the most sense. Would you accept a PR addressing this?