The bench from liufengyun

change daily test to immediate test on merge

move repo to lampepfl

Move repo to lampepfl. This is not a priority, but it's something good to do.

bench points to the wrong PR

The latest speedup most likely comes from scala/scala3#2896 -- due to some reordering of the commits, the graph doesn't point to the correct PR.

Improvements

Copied from scala/scala3#2884

Suggestions:

measure in milliseconds;
graph multiple lines:
- average
- median [50% percentile]
- 5% percentile
- 95% percentile
- 99% percentile
- trendline for all listed above
store raw data for runs, so that we don't need to rerun benchmarks.
add a test that is supposed to always be flat, will be used as indicator of stability of machine.
make the Y axis start from 0
One more suggestion: could you align the x axis across multiple benchmarks? so that one vertical always relates to the same PR on all benchmarks.

More todos

add all tests in the tests folder and one normal exhaustivity checks
fix test for the dotty project

make benchmarks work from any repo

Currently it only works with the dotty-staging/dotty repo, it would be good to make benchmarks work from any repo.

retrospectively test stdlib

The script to do the job:

set -e

cp bin/stdlib-plan bin/plan
bin/run -f 1850  -t 1889

cp bin/stdlib-plan bin/plan
bin/run -f 1889  -t 1910

cp bin/stdlib-plan bin/plan
bin/run -f 2312 -t 2353

cp bin/stdlib-plan bin/plan
bin/run -f 2353 -t 2384

cp bin/stdlib-plan bin/plan
bin/run -f 2384 -t 2389

cp bin/stdlib-plan bin/plan
bin/run -f 2509 -t 2538

cp bin/stdlib-plan bin/plan
bin/run -f 2547 -t 2523

cp bin/stdlib-plan bin/plan
bin/run -f 2523 -t 2587

cp bin/stdlib-plan bin/plan
bin/run -f 2587 -t 2606

cp bin/stdlib-plan bin/plan
bin/run -f 2606 -t 2619

cp bin/stdlib-plan bin/plan
bin/run -f 2619 -t 2643

cp bin/stdlib-plan bin/plan
bin/run -f 2674 -t 2694

cp bin/stdlib-plan bin/plan
bin/run -f 2694 -t 2719

The script bin/stdlib-plan:

measure "tests/scala-library" -language:Scala2 $(find tests/scala-library -name *.scala)

give different URLs to different test command for the same PR

Unable to find the resource: /META-INF/BenchmarkList

On some new machines, we encounter the exception:

fliu@lamppc34:/localhome/fliu/bench$ target/pack/bin/bench tests/implicit_cache.scala
Exception in thread "main" java.lang.RuntimeException: ERROR: Unable to find the resource: /META-INF/BenchmarkList
	at org.openjdk.jmh.runner.AbstractResourceReader.getReaders(AbstractResourceReader.java:98)
	at org.openjdk.jmh.runner.BenchmarkList.find(BenchmarkList.java:122)
	at org.openjdk.jmh.runner.Runner.internalRun(Runner.java:256)
	at org.openjdk.jmh.runner.Runner.run(Runner.java:206)
	at dotty.tools.benchmarks.Bench$.main(Benchmarks.scala:37)
	at dotty.tools.benchmarks.Bench.main(Benchmarks.scala)

weird exceptions on server

The tests crash from time to time due to following exception:

Could not create file /data/workspace/bench/dotty/sbt-bridge/target/streams/$global/projectDescriptors/$global/streams/outjava.io.IOException: No such file or directory

Simply restart the testing solves the problem.

support bench specified commits of a PR

Ability to zoom in or set a start date for the graphs ?

There are now a lot of points on http://dotty-bench.epfl.ch/ this is good :), but it makes a bit hard to see what's happening in the last few points. It'd be great if we could ignore points before a start date, or zoom in to some points.