How can i decompress value, if i don't know maxDestLen?

[lz4]

How can i decompress value, if i don't know maxDestLen?

[eyalroth]

I am closing this one since this is not really an issue

Seriously? this just makes the API more cumbersome and adds work for the user. Why not simply add a header to the compressed data with the decompressed length? sounds to me like an easy way of encapsulating this logic from the user.

Moreover, the documentation in the main README.md of the project suggests that there's a way of decompressing while only knowing the compressed data length:

// - method 2: when the compressed length is known (a little slower)
// the destination buffer needs to be over-sized
LZ4SafeDecompressor decompressor2 = factory.safeDecompressor();
int decompressedLength2 = decompressor2.decompress(compressed, 0, compressedLength, restored, 0);

In this example, one must know what size to initialize restored with; i.e, the decompressed size must be known in this method as well. Well, not exactly: One could also do with the maximum potential decompressed size (if there's such a thing for one's application), but then it would be inefficient to allocate a byte array with the maximum size on each decompression.

[andresilva]

The decompress method is overloaded with a definition that doesn't need the maxDestLen parameter.

https://github.com/jpountz/lz4-java/blob/master/src/java/net/jpountz/lz4/LZ4UnknownSizeDecompressor.java#L25

[i-davydov]

But, What size do i have to set for byte[] dest ?

[jpountz]

@Acvilon Indeed, this API is low-level and the original size of the compressed byte[] is not encoded in the compressed stream so you need to get this information by storing it somewhere else. This is done this way for maximum flexibility: although some users might want to encode the original length before the compressed bytes to be able to know the original length at decompression time, other users may know this from their application logic, for example if they always compress blocks of exactly 16KB.

For example, if you use lz4 to compress files on disk, you could reserve the first 8 bytes to store the original length of the file and the following bytes to store the compressed file.

I am closing this one since this is not really an issue but feel free to ask more questions in this thread if the API still feels unclear to you.

[kaushalhakani]

encode the original length before the compressed bytes to be able to know the original length at decompression time

This should be provided as one of the implementations rather than asking most users to implement it. If you would like it I can contribute that piece of code.

[odaira]

This functionality was implemented by LZ4CompressorWithLength and LZ4DecompressorWithLength (#119) in the master branch. It will be included in the next release.

[gerritjvv]

Could you update the README.md so that LZ4CompressorWithLength is known while reading the readme? It took me quite a while to actually find that this class exists.
thanks!.