Add broad range of inputs for leap benchmark?

[exercism]

Add broad range of inputs for leap benchmark?

[soniakeys]

Or maybe benchmark single numbers of the different cases -- not divisible by 4, 4 but not 100, and so on. It's a little crazy to benchmark a bunch of stuff all together. I think it's okay for comparing one implementation to another, but less meaningful for understanding the time of an individual operation.

[kytrinyx]

Ah, I was thinking of the benchmark as comparing implementations.

Yeah, we could separate them out by case.

[soniakeys]

So how about single number tests plus a test of a 400 year period. The 400 year period would be good for comparing implementations but representing the frequency of occurrence of the cases. The individual cases might help understanding the 400 year results.

You know this pattern of benchmarking all the test cases together is just something that's easy as a comparable time. (I think the old pattern of starting and stopping the timer doesn't really work.) More interesting or more meaningful benchmarks might be devised for a number of the exercises.

[kytrinyx]

So how about single number tests plus a test of a 400 year period.

That sounds very good.

[soniakeys]

leap $ go test -tags example -bench .
PASS
Benchmark2015Once-8 1000000000           2.24 ns/op
Benchmark2014Once-8 1000000000           2.26 ns/op
Benchmark1900Once-8 500000000            3.54 ns/op
Benchmark1600Once-8 500000000            3.54 ns/op
Benchmark2015x400-8  2000000           924 ns/op
Benchmark2014x400-8  2000000           929 ns/op
Benchmark1900x400-8  1000000          1401 ns/op
Benchmark1600x400-8  1000000          1405 ns/op
BenchmarkRange400-8  1000000          1049 ns/op
ok      github.com/exercism/xgo/leap    18.751s

I'm not sure what it means. I would guess complexities of the CPU architecture and the the compiler code generator make this hard to understand. Maybe it's best to just use the 400 year period and not include the other tests.

[kytrinyx]

Yeah, agreed. The 400 year range will at least give people a number that would simulate a varied set of inputs.