Using MultiGzDecoder for file with garbage after gzip data

[jinschoi]

Using MultiGzDecoder for file with garbage after gzip data

[Byron]

Thanks for bringing this up! Could you provide a testfile along with a test-program that shows the current behaviour?
My assumption is that it's not actually possible to decode the last member of such a file as it emits an error on the last read, but it would be great to validate. Maybe it will also decode the last member successfully, but fail on the next read as it can't decode garbage, which would mean all zipped data could be read with the current API, and it's really a question on how to indicate a 'garbage tail' occurred.

With such example, it would be possible to setup a test-case and eventually find a solution.

[jinschoi]

Sure. Here is a test file I created with the following commands:

echo "Hello, world" | gzip > hello.gz
echo x >> hello.gz

hello.gz

Running gzcat on the file:

% gzcat hello.gz
Hello, world
gzcat: hello.gz: trailing garbage ignored

A short program that demonstrates the issue:

use flate2::read::MultiGzDecoder;
use std::fs::File;
use std::io::{BufReader, Read, Write};

fn main() {
    let file = File::open("hello.gz").unwrap();
    let mut buf_reader = BufReader::new(MultiGzDecoder::new(file));
    let mut buf = [0_u8; 100];
    loop {
        match buf_reader.read(&mut buf) {
            Ok(0) => break,
            Ok(n) => {
                println!("read {} bytes:", n);
                std::io::stdout().write_all(&buf[0..n]).unwrap();
                std::io::stdout().flush().unwrap();
            }
            Err(e) => {
                println!("{:?}", e);
                break;
            }
        }
    }
}

And the output:

read 13 bytes:
Hello, world
Kind(UnexpectedEof)

[Byron]

Thanks a lot for your help!

Thoughts

It looks like the question here is of UnexpectedEOF could also be another kind of error which would indicate more clearly that indeed, there was nothing to read, or that the stream it tried to read (x here) isn't actually a deflated stream.

Another guess is that if the trailing bytes are longer, the error message will be a different, maybe more specific one. After all, the magic signature seems to be 2 bytes long, so we'd run out of data before reading this even.

Next Steps

I think it would be worth setting up a couple of test cases with varying length of trailing garbage to get an idea of how it can be detected on application level right now.

From there, we might be able to figure out how to optionally adjust or enhance the API in a backward compatible manner to make detecting this situation easier.