concatenated_zlib

this is a playground repository to experiment with an zlib and zlibng decompressor that can handle zlib compressed concatenated chunks.

The code here is basically copied from the excellent imagecodecs package: https://github.com/cgohlke/imagecodecs/ and modified to support concatenated input.

description

The below fails to decompress concatenated zlib compressed blocks:

import zlib
c0 = zlib.compress(b"abc")
c1 = zlib.compress(b"efg")

print(zlib.decompress(c0 + c1))  # returns b"abc"

This repo provides a zlib and zlibng based decompressor that supports concatenated input:

import zlib
from concatenated_zlib import zlib_concat_decode
from concatenated_zlib import zlibng_concat_decode
c0 = zlib.compress(b"abc")
c1 = zlib.compress(b"efg")

print(zlib_concat_decode(c0 + c1))  # returns b"abcefg"
print(zlibng_concat_decode(c0 + c1))  # returns b"abcefg"

benchmarks

To reproduce:

git clone --recursive https://github.com/zlib-ng/zlib-ng.git
cd zlib-ng/ && mkdir build && cd build
cmake -DCMAKE_INSTALL_PREFIX=/opt/zlib-ng ..
make -j $( nproc ) install
export CPATH=/opt/zlib-ng/include
export LD_LIBRARY_PATH="/opt/zlib-ng/lib:$LD_LIBRARY_PATH"
export LIBRARY_PATH="/opt/zlib-ng/lib:$LIBRARY_PATH"

python3 -m pip install --force-reinstall 'git+https://github.com/mxmlnkn/indexed_bzip2.git@zlib-support#egginfo=rapidgzip&subdirectory=python/rapidgzip'

git clone --recursive https://github.com/ap--/concatenated-zlib.git
cd concatenated-zlib/
python3 -m pip install --user .
cd bench
python time_load_chunks.py

To reproduce:

# ON A MAC YOU CAN INSTALL THE DEPENDECIES VIA homebrew
brew install zlib zlib-ng libdeflate
# ^^^ install the corresponding *-dev packages on your OS accordingly

git clone --recursive https://github.com/ap--/concatenated-zlib.git
cd concatenated-zlib/
python3 -m venv venv
source venv/bin/activate
python3 -m pip install .
cd bench; python time_load_chunks.py

Run cd bench; python time_load_chunks.py to get some crude timing for the different approaches... Weirdly enough the cython implementation doesn't currently beat the plain python stdlib decompression of individual chunks. Needs more investigation...

related info

https://stackoverflow.com/a/60208718 decompress concatenated zlib chunks

mxmlnkn / concatenated-zlib Goto Github PK