Benchmark graph improvement

[sverx]

Benchmark graph improvement

[maxim-zhao]

Hmm, PR welcome :) I guess somewhere near https://github.com/maxim-zhao/bmp2tilecompressors/blob/master/benchmark/benchmark.py#L133, I am not a Pythonista so I don't know how to do it easily.

I think the fact that the points are so scattered means the mean+SD isn't very meaningful though.

The graph was inspired by the one at https://github.com/emmanuel-marty/lzsa but it seems impossible to distil my benchmarks into a single point.

[sverx]

PR welcome :)

I feared that.

The mean (and the SD circle/ellipsis around it) would help identify strengths and weaknesses more easily. It's that distilled single point you wanted.

I don't know how to graph from Python but average and SD should not be too hard to calculate using some statistics package?

[maxim-zhao]

Yeah, it's the interaction with matplotlib that gets me confused. Just getting it to not repeat colours was hard work.

A mean+SD might mean more if my corpus was larger. It's a matter of building up representative tilesets, which is extra tricky if I try to not use copyrighted material.

[sverx]

I see you're already using copyrighted material (and I suspect it's fair use anyway). How many items do you think are needed to make a good corpus?

[maxim-zhao]

It's hard to say. The giant spread you see on all the compressors suggest there's a lot of variation in performance and speed depending on the artwork.

Anyway, how about this?

I think it's a bit hard to deal with the overlapping ellipses, maybe I can push them to the bottom of the Z order but they still obscure each other a bit...

[maxim-zhao]

I tried adding trendlines (quadratic best fit, but the X axis is a log scale):

I don't think it helps much, especially as some seem pretty non-linear...
This also has Z ordering such that the SD ellipses are at the bottom, then trendlines, then dots and "big dots" for means on the top.

[maxim-zhao]

Solid ellipse lines instead of transparent fills, the PS Gaiden pink one is still hard to see:

Anyway, clearly the PS Gaiden fast compressor is the best option...

[sverx]

Nice!!! 🤩

Probably the third one is the best one IMHO. I would just remove the average and the ellipses on those two (the uncompressed ones) that have zero SD, as it's useless there.

As for the PSGaiden fast, sometimes I wonder if I should try to improve it more, now that I have (or... should have) more experience with Z80 asm. Well, this is off-topic anyway.

[maxim-zhao]

One final go with the trendlines as it turned out the above was linear best fit rather than quadratic (curvature was just due to the log scale):

I'm fairly convinced that trendlines mean nothing here so I'll remove them.

[sverx]

Well, the trendlines aren't completely useless IMHO. They sort of describe how the decoding speed and the compression level relate to each other for each compression scheme. So for instance in some cases better compression brings faster decode times and in some other cases the opposite is true instead... and a few pretty weird cases in between.