maayanorner/compress_pickle

Standard python pickle, thinly wrapped with standard compression libraries

compress_pickle

Standard python pickle, thinly wrapped with standard compression libraries

The standard pickle package provides an excellent default tool for serializing arbitrary python objects and storing them to disk. Standard python also includes broad set of data compression packages. compress_pickle provides an interface to the standard pickle.dump, pickle.load, pickle.dumps and pickle.loads functions, but wraps them in order to direct the serialized data through one of the standard compression packages. This way you can seemlessly serialize data to disk or to any file-like object in a compressed way.

compress_pickle supports python >= 3.6. If you must support python 3.5, install compress_pickle==v1.1.1.

Supported compression protocols:

• gzip
• bz2
• lzma
• zipfile

Furthermore, compress_pickle supports the lz4 compression protocol, that isn't part of the standard python compression packages. This is provided as an optional extra requirement that can be installed as:

pip install compress_pickle[lz4]

Please refer to the package's documentation for more information