Maximum compression

[ebiggers]

Maximum compression

[ebiggers]

The main problem with that file is that libdeflate creates too many blocks and wastes too much space sending Huffman codes. Case in point: if you increase OPTIM_BLOCK_LENGTH in deflate_compress.c from 16384 to 65536 and recompile, then libdeflate gzip -12 compresses enwiki8 to 35,056,842 bytes, beating 7-Zip -mx9.

I am aware that there is work needed to improve the block splitting, especially at high compression modes. Currently I have spent the most effort optimizing the decompressor and the compressor at low to medium compression modes, which is where libdeflate is primarily intended to compete. It should also be understood that if you want the "optimum" gzip compression ratio you should probably use Zopfli, which was specifically designed for that purpose.

[ebiggers]

libdeflate v0.2 gzip -12 compresses enwiki8 to 35,100,553 bytes.