arkoniak/benchmarkext.jl

A benchmarking framework for the Julia language

benchmarkext.jl's Issues

Full memory in Trial

Trial now only support lowest number of allocations and memory.
Should save all data

Benchmark sets

While judge is a nice feature, but it require some setup. It would be nice to have @benchmarkset f() g() to combine results together.
Some inspiration can be found here: JuliaCI#239

Split histogram and general info in benchmark

While it's nice to see histograms in benchmarks, it's much better to be able to turn them off.
Define .toml file, where user can set this option on and off.

GC test is failing

"Interpolated values are garbage-collectable" in Execution tests is failing. It is somehow related to ReTest. Should find it out and fix the test.

Colorschemes

Yep. That's it. We need colorschemes. And nice color printing.

Proposal:

Make a macro which accepts something like this

@colorprint io colorscheme "Hello:hello {world!}:world"

where colorscheme is like Dict(:hello => :green, :world => :yellow)

Make "classic" / "fancy" switch

I want to return original show for benchmarks.
Add switch in toml file, to choose what you want.

Switch to Preferences.jl

Currently I am using custom code to work with preferences. There is a project https://github.com/JuliaPackaging/Preferences.jl which should make it more reliable and stable.

Add ReTest

Move tests to ReTest.jl

Compact fancy mode

Now, that we have configuration file for the benchmark output, it make sense to try and make better fancy output, for example something like this:

JuliaCI#236

Add support for Tables.jl

It would be nice to save Trials in DataFrames with single command

DataFrame(@benchmark foo())

Add support for JSON3

Current serialization can be improved, by using JSON3 facilities.

Histograms support for IJulia/Pluto

Is is possible to define show so histograms is shown differently in this environments? That would be really nice.