Request: CSV output

[martinus]

Request: CSV output

[martinus]

Thanks for the issue, I've been thinking about this too. I'd like to add some very generic solution, if possible. I'm currently experimenting with mustache templates. Stay tuned!

[martinus]

In the latest version, you can now easily create CSV, JSON, or a nice HTML boxplot. It's now easy to create a custom output format too. See the file example_random_number_generators.cpp where I do all that

[e4lam]

I tried using this but I'm having some problems with the output of it. Previously, I was running cfg.title() for each benchmark instance, given it different problem sizes each time. This produced nice output with new row headings each time. However, when I use the csv template, it only outputs the first header row, instead of once per call to title(). It looks like some weirdly outright wrong titles for the results. Using today's a1d1298

[martinus]

Can you post a code sample? I'm not sure what's wrong. Maybe it's because the Config object now caches all the results.

[e4lam]

Sorry for taking so long but here's some test code. Change the "OUTPUT_CSV" define in the code to see the different between .csv and .md outputs.

#define ANKERL_NANOBENCH_IMPLEMENT
#include "nanobench.h"

#include <cstring>
#include <iostream>
#include <list>
#include <vector>

#define OUTPUT_CSV 0

template <typename ContainerT>
static void
testBenchSet(const char* label, int n, ankerl::nanobench::Config& cfg)
{
    ContainerT items;

    cfg.run(label,
            [&]() {
                for (int i = 0; i < n; ++i)
                    items.push_back(i);
            }).doNotOptimizeAway(items);
}

int
main()
{
    ankerl::nanobench::Config cfg;
#if OUTPUT_CSV
    cfg.output(nullptr);
#endif

    for (int n = 100; n < 10000; n *= 10)
    {
        char buf[1024];
        std::sprintf(buf, "Size %d", n);
        cfg.title(buf);

        testBenchSet<std::vector<int>>("std::vector<int>", n, cfg);
        testBenchSet<std::list<int>>("std::list<int>", n, cfg);
    }

#if OUTPUT_CSV
    cfg.render(ankerl::nanobench::templates::csv(), std::cout);
#endif
    return 0;
}

[martinus]

Thanks for the example! I've done some change now to the CSV writer. I've integrated your example (with a few modifications) here: https://github.com/martinus/nanobench/blob/master/src/test/example_csv.cpp

Note that I've now put the cfg.render(...) line into the for loop.

When I run ./nb -tc=example_csv_csv I get this output:

"Size 100"; "relative %"; "s/op"; "min/op"; "max/op"; "error %"; "measurements"; "instructions/op"; "branch/op"; "branch misses/op"
"std::vector<int>"; 1; 1.95857142857143e-07; 1.86746268656716e-07; 4.79853333333333e-07; 0.0414177567205686; 11; 2435.17910447761; 570; 0.0175438596491228
"std::deque<int>"; 0.961852144172586; 2.03625e-07; 1.95307692307692e-07; 6.66212121212121e-07; 0.0425856636471051; 11; 1531.37142857143; 272; 1
"std::list<int>"; 0.152804480481485; 1.28175e-06; 1.257e-06; 2.0085e-05; 0.0171273641052772; 11; 20806.75; 4999; 3.33333333333333

"Size 1000"; "relative %"; "s/op"; "min/op"; "max/op"; "error %"; "measurements"; "instructions/op"; "branch/op"; "branch misses/op"
"std::vector<int>"; 1; 1.3795e-06; 1.36833333333333e-06; 4.8435e-06; 0.00682931915337303; 11; 11888.4; 2623; 5.2
"std::deque<int>"; 0.734688443103142; 1.87766666666667e-06; 1.86866666666667e-06; 4.191e-06; 0.00348653738484237; 11; 13931.1666666667; 2413; 11.2
"std::list<int>"; 0.108391608391608; 1.2727e-05; 1.2616e-05; 2.6271e-05; 0.00346921075455333; 11; 210852; 50681; 6

"Size 10000"; "relative %"; "s/op"; "min/op"; "max/op"; "error %"; "measurements"; "instructions/op"; "branch/op"; "branch misses/op"
"std::vector<int>"; 1; 1.3676e-05; 1.3515e-05; 5.9338e-05; 0.0119126896041435; 11; 95563; 21184; 17
"std::deque<int>"; 0.715646258503401; 1.911e-05; 1.8968e-05; 3.4645e-05; 0.00162482310393628; 11; 151539; 26863; 112
"std::list<int>"; 0.105427073697194; 0.00012972; 0.000127737; 0.000339994; 0.0134216652864799; 11; 2110977; 507431; 14

When I run ./nb -tc=example_csv_md I get this output:

ns/op	op/s	err%	ins/op	cyc/op	IPC	bra/op	miss%	total	Size 100
262.93	3,803,258.47	0.1%	2,435.16	839.80	2.900	570.00	0.0%	0.00	std::vector<int>
265.24	3,770,237.30	0.3%	1,531.17	846.90	1.808	272.00	0.4%	0.00	std::deque<int>
1,730.58	577,839.84	0.2%	20,804.92	5,530.58	3.762	4,999.00	0.0%	0.00	std::list<int>

ns/op	op/s	err%	ins/op	cyc/op	IPC	bra/op	miss%	total	Size 1000
1,882.10	531,321.40	0.2%	11,887.09	6,009.18	1.978	2,623.00	0.2%	0.00	std::vector<int>
2,553.75	391,581.01	0.2%	13,930.62	8,146.25	1.710	2,413.00	0.4%	0.00	std::deque<int>
17,509.00	57,113.48	0.3%	210,852.00	55,993.00	3.766	50,681.00	0.0%	0.00	std::list<int>

ns/op	op/s	err%	ins/op	cyc/op	IPC	bra/op	miss%	total	Size 10000
18,681.00	53,530.32	0.2%	95,562.00	59,717.00	1.600	21,183.00	0.1%	0.00	std::vector<int>
26,248.00	38,098.14	0.7%	151,836.00	83,923.00	1.809	26,970.00	0.4%	0.00	std::deque<int>
176,193.00	5,675.59	0.4%	2,110,977.00	562,590.00	3.752	507,431.00	0.0%	0.00	std::list<int>

[e4lam]

Thank you, looks good! I think the only other request here is some way to choose the separator to output as I can's just run "start foo.csv" and have Excel import it automatically.