ocaml-bench/bench

Placeholder repo for issues

bench

Placeholder repo for issues

bench's Issues

bench.ocamllabs.io: can we point it at pull-requests

Is there a way for bench.ocamllabs.io to point at a pull-request with some opam config variables?

bench.ocamllabs.io: dropdown for selecting changes is not scalable

The dropdown on the changes tab of bench.ocamllabs.io will not handle a very large number of commits.

More generally we don't seem to be able to browse every commit easily from the front end.

self-testing of a benchmark run for soundness of a result

Are there ways to highlight that a given benchmarking run is 'not reliable' using the statistics of the run?

(NB: this can be hard in general if a given run is clean but randomness is outside of it, for example what we saw with ASLR)

bench.ocamllabs.io: permalinks from standard cut and paste

Having to hit a permalink button before cut-and-paste is pretty old-school.

Are we able to fix it so that you can cut-and-paste the link you see and send it to other people? Right now people are sometimes sending links where others will not be able to view the data they are seeing.

bench.ocamllabs.io: is there an interface to download the raw data

Is it possible to download the raw data from bench.ocamllabs.io so that people can make their own tools that use the data?

bench.ocamllabs.io: integration of sandmark benchmarks into website

can we integrate the sandmark macro benchmark output into the website?

bench.ocamllabs.io: addition of arm64 and ppc machines

Can we get benchmarking data for arm64 and ppc machines integrated into the website?

bench.ocamllabs.io: Can we visualize multidimensional data

Want to be able to visualize the GC output (minor collections, major collections, compactions, etc.) and other timeseries data (e.g. perf cycles, perf branch miss predicts, perf L2 data miss, etc.)

bench.ocamllabs.io: git log changes tab layout is horrid

The git log output in the changes tab is overwriting the table making things hard to see unless you have a large screen.

bench.ocamllabs.io: log comparison bars

When comparing multiple versions on the compare page, the normalisation becomes an arbitrary choice that will affect your interpretation.

Is it possible to add a log-mode plot where we fix this (stedolan has ideas here)

bench.ocamlabs.io: can we track compilation times

Is it possible to add compilation times to bench.ocamllabs.io?

This looks like it will need:

• a selection of examples we want to compile
• a way to measure the compile time accurately with the minimum of noise
• collation of the stats
• uploading to a frontend (codespeed?)

bench.ocamllabs.io: way to sort the comparison data by slowest to fastest speedup

It would be useful to be able to sort the comparison data over all benchmarks to list them by speedup/slowdown.

One option is to make the javascript graphs sortable.

An alternative option is to output an HTML table of benchmark_name x executable_normalized_value
which can then be sorted independent of the javascript bar chart.

data format specification

Are we able to specify the data format that all benchmarking experiments should produce which allows the decoupling of the downstream analysis and visualisation from benchmarking production?

bench.ocamllabs.io: addition of flambda-OClassic & flambda-O3

For 4.08 can we track flambda-OClassic & flambda-O3