picohz/lzzzz

Full-featured liblz4 binding for Rust

lzzzz's Issues

WriteCompressor generates invalid lz4f frame if flush() called before any write()

WriteCompressor generates an invalid lz4f frame when flushed() before any calls to write().

let mut comp = WriteCompressor::new(Vec::new(), Preferences::default())?;
comp.flush()?;

let buf = comp.into_inner();
let mut decomp = ReadDecompressor::new(buf.as_slice())?;
let mut out = Vec::new();
assert!(matches!(decomp.read_to_end(&mut out), Err(_)));

Writing explicitly an empty slice before flushing, seems to work fine.

let mut comp = WriteCompressor::new(Vec::new(), Preferences::default())?;
comp.write(&[])?;
comp.flush()?;

let buf = comp.into_inner();
let mut decomp = ReadDecompressor::new(buf.as_slice())?;
let mut out = Vec::new();
assert!(matches!(decomp.read_to_end(&mut out), Ok(0)));

WriteCompressor does not implement Send

It seems like all the compressors and decompressors are not Send because of CompressionContext and DecompressionContext respectively.

Here is the minimal reproduction code.

use lzzzz::{lz4f::*};

fn is_send<T: Send>(_: &T) {}

fn main() {
    let mut buf = vec![];
    let w = WriteCompressor::new(&mut buf, Preferences::default());
    is_send(&w);
}

Here is the backtrace for the Send bound.

error[E0277]: `std::ptr::NonNull<lzzzz::lz4f::binding::LZ4FCompressionCtx>` cannot be sent between threads safely
  --> tests/lz4f_stream.rs:20:9
   |
13 |     fn is_send<T: Send>(t: &T) {}
   |        -------    ---- required by this bound in `write_compressor::is_send`
...
20 |         is_send(&w);
   |         ^^^^^^^ `std::ptr::NonNull<lzzzz::lz4f::binding::LZ4FCompressionCtx>` cannot be sent between threads safely
   |
   = help: within `std::result::Result<lzzzz::lz4f::stream::comp::write::WriteCompressor<&mut std::vec::Vec<u8>>, lzzzz::lz4f::error::Error>`, the trait `std::marker::Send` is not implemented for `std::ptr::NonNull<lzzzz::lz4f::binding::LZ4FCompressionCtx>`
   = note: required because it appears within the type `lzzzz::lz4f::api::CompressionContext`
   = note: required because it appears within the type `lzzzz::lz4f::stream::comp::Compressor`
   = note: required because it appears within the type `lzzzz::lz4f::stream::comp::write::WriteCompressor<&mut std::vec::Vec<u8>>`
   = note: required because it appears within the type `std::result::Result<lzzzz::lz4f::stream::comp::write::WriteCompressor<&mut std::vec::Vec<u8>>, lzzzz::lz4f::error::Error>`

Is this a bug or a fundamental limitation of the liblz4?

AsyncReadDecompressor reads `Ok(0)` before EOF is reached

Basically if the read / write buffer is small enough (less than 54 bytes), AsyncReadCompressor::poll_read returns Poll::Ready(Ok(0)), which is the sentinel for EOF, even though we haven't actually reached end of file.

Here is the example. It uses my async_std branch because it makes the code simpler, but its also an issue on master.

Remove async from lzzzz

Allow linking non-vendored lz4 and put vendored sources behind a feature

If this is already possible, how do I do that?

Implement Encode trait for `async-compression`

As discussed in #4, the async-io part of the crate could be moved to async-compression directly.

I looked into it and this would work, but we would need to expose the raw Compressor & Decompressor so that they can be used to fit the following traits.

pub trait Encode {
    fn encode(
        &mut self,
        input: &mut PartialBuffer<impl AsRef<[u8]>>,
        output: &mut PartialBuffer<impl AsRef<[u8]> + AsMut<[u8]>>,
    ) -> Result<()>;

    /// Returns whether the internal buffers are flushed
    fn flush(&mut self, output: &mut PartialBuffer<impl AsRef<[u8]> + AsMut<[u8]>>)
        -> Result<bool>;

    /// Returns whether the internal buffers are flushed and the end of the stream is written
    fn finish(
        &mut self,
        output: &mut PartialBuffer<impl AsRef<[u8]> + AsMut<[u8]>>,
    ) -> Result<bool>;
}

pub trait Decode {
    /// Reinitializes this decoder ready to decode a new member/frame of data.
    fn reinit(&mut self) -> Result<()>;

    /// Returns whether the end of the stream has been read
    fn decode(
        &mut self,
        input: &mut PartialBuffer<impl AsRef<[u8]>>,
        output: &mut PartialBuffer<impl AsRef<[u8]> + AsMut<[u8]>>,
    ) -> Result<bool>;

    /// Returns whether the internal buffers are flushed
    fn flush(&mut self, output: &mut PartialBuffer<impl AsRef<[u8]> + AsMut<[u8]>>)
        -> Result<bool>;

    /// Returns whether the internal buffers are flushed
    fn finish(
        &mut self,
        output: &mut PartialBuffer<impl AsRef<[u8]> + AsMut<[u8]>>,
    ) -> Result<bool>;
}

So I'll work on a fork that expose this enough functions to fit this API, and I'll use it in async-compression crate.

AsyncReadCompressor is never waked up when buffer too small

It seems that when the read / write buffer is smaller than the total size of the lz4f frame, the AsyncReadCompressor is not waked up properly.

Here is my repoduction example. I'm using my async_std branch for simplicity.

Unable to set compression level?

It doesn't appear Preferences supports setting the compression level. Both the field compression_level and method set_compression_level are private; is there an alternative way I'm missing?