failed to decompress: Unknown frame descriptor If padding is attached after compression

[datadog]

failed to decompress: Unknown frame descriptor If padding is attached after compression

[anshulchaurasia]

@Viq111 I hope you can help with this

[Viq111]

Hi @anshulchaurasia

It looks like you are not consuming after doing a zr.Close(). When you close it actually flush the frame finish so you need to consume the rest via reading remaining data fromzr.

It looks like it would be a common pitfall so I'll add to the documentation the exact code snippet

[anshulchaurasia]

hi @Viq111

Could you please post the snippet here it will be very helpful.
As I am getting the error on the following line:

buf, err = ioutil.ReadAll(zr)

it's before calling Close().

[Viq111]

Oh sorry, I read it incorrectly and though you were compressing and not decompressing.
For decompression, zstd will try to interpret your padding as compressed data and will fail. You have to provide to the reader exactly the compressed data.

In your case (unrelated to zstd), you would probably want to prefix the data block by how much padding you are adding.

Something like: https://play.golang.org/p/7muxEHDJDTl

[anshulchaurasia]

Hi @Viq111
As a consumer I am not aware how much padding is added so not able to remove it.
According to your previous comment I have to provide exactly the compressed data whereas decompression is working fine with padding upto 4 bytes.(verified with a sample code)
Can you please explain how it is working and is there any possible way to work with any numbers of padded bytes?

Thanks

[anshulchaurasia]

Hi @Viq111

Any suggestion from your side?

Thanks

[Viq111]

I haven't looked too much at the internals of the compression algorithm of zstd unfortunately ( https://github.com/facebook/zstd/issues would probably be a better place to ask about it) so I'm not sure about the guarantee of the end of frame (i.e: can a frame finish with zero bytes).
It's usually pretty common for systems adding padding to provide the size of it because you cannot know if the extra pad is real data or not