picohz / lzzzz Goto Github PK
View Code? Open in Web Editor NEWFull-featured liblz4 binding for Rust
License: MIT License
Full-featured liblz4 binding for Rust
License: MIT License
WriteCompressor generates an invalid lz4f frame when flushed() before any calls to write().
let mut comp = WriteCompressor::new(Vec::new(), Preferences::default())?;
comp.flush()?;
let buf = comp.into_inner();
let mut decomp = ReadDecompressor::new(buf.as_slice())?;
let mut out = Vec::new();
assert!(matches!(decomp.read_to_end(&mut out), Err(_)));
Writing explicitly an empty slice before flushing, seems to work fine.
let mut comp = WriteCompressor::new(Vec::new(), Preferences::default())?;
comp.write(&[])?;
comp.flush()?;
let buf = comp.into_inner();
let mut decomp = ReadDecompressor::new(buf.as_slice())?;
let mut out = Vec::new();
assert!(matches!(decomp.read_to_end(&mut out), Ok(0)));
It seems like all the compressors and decompressors are not Send
because of CompressionContext
and DecompressionContext
respectively.
Here is the minimal reproduction code.
use lzzzz::{lz4f::*};
fn is_send<T: Send>(_: &T) {}
fn main() {
let mut buf = vec![];
let w = WriteCompressor::new(&mut buf, Preferences::default());
is_send(&w);
}
Here is the backtrace for the Send
bound.
error[E0277]: `std::ptr::NonNull<lzzzz::lz4f::binding::LZ4FCompressionCtx>` cannot be sent between threads safely
--> tests/lz4f_stream.rs:20:9
|
13 | fn is_send<T: Send>(t: &T) {}
| ------- ---- required by this bound in `write_compressor::is_send`
...
20 | is_send(&w);
| ^^^^^^^ `std::ptr::NonNull<lzzzz::lz4f::binding::LZ4FCompressionCtx>` cannot be sent between threads safely
|
= help: within `std::result::Result<lzzzz::lz4f::stream::comp::write::WriteCompressor<&mut std::vec::Vec<u8>>, lzzzz::lz4f::error::Error>`, the trait `std::marker::Send` is not implemented for `std::ptr::NonNull<lzzzz::lz4f::binding::LZ4FCompressionCtx>`
= note: required because it appears within the type `lzzzz::lz4f::api::CompressionContext`
= note: required because it appears within the type `lzzzz::lz4f::stream::comp::Compressor`
= note: required because it appears within the type `lzzzz::lz4f::stream::comp::write::WriteCompressor<&mut std::vec::Vec<u8>>`
= note: required because it appears within the type `std::result::Result<lzzzz::lz4f::stream::comp::write::WriteCompressor<&mut std::vec::Vec<u8>>, lzzzz::lz4f::error::Error>`
Is this a bug or a fundamental limitation of the liblz4?
Basically if the read / write buffer is small enough (less than 54 bytes), AsyncReadCompressor::poll_read
returns Poll::Ready(Ok(0))
, which is the sentinel for EOF
, even though we haven't actually reached end of file.
Here is the example. It uses my async_std
branch because it makes the code simpler, but its also an issue on master.
If this is already possible, how do I do that?
As discussed in #4, the async-io part of the crate could be moved to async-compression
directly.
I looked into it and this would work, but we would need to expose the raw Compressor
& Decompressor
so that they can be used to fit the following traits.
pub trait Encode {
fn encode(
&mut self,
input: &mut PartialBuffer<impl AsRef<[u8]>>,
output: &mut PartialBuffer<impl AsRef<[u8]> + AsMut<[u8]>>,
) -> Result<()>;
/// Returns whether the internal buffers are flushed
fn flush(&mut self, output: &mut PartialBuffer<impl AsRef<[u8]> + AsMut<[u8]>>)
-> Result<bool>;
/// Returns whether the internal buffers are flushed and the end of the stream is written
fn finish(
&mut self,
output: &mut PartialBuffer<impl AsRef<[u8]> + AsMut<[u8]>>,
) -> Result<bool>;
}
pub trait Decode {
/// Reinitializes this decoder ready to decode a new member/frame of data.
fn reinit(&mut self) -> Result<()>;
/// Returns whether the end of the stream has been read
fn decode(
&mut self,
input: &mut PartialBuffer<impl AsRef<[u8]>>,
output: &mut PartialBuffer<impl AsRef<[u8]> + AsMut<[u8]>>,
) -> Result<bool>;
/// Returns whether the internal buffers are flushed
fn flush(&mut self, output: &mut PartialBuffer<impl AsRef<[u8]> + AsMut<[u8]>>)
-> Result<bool>;
/// Returns whether the internal buffers are flushed
fn finish(
&mut self,
output: &mut PartialBuffer<impl AsRef<[u8]> + AsMut<[u8]>>,
) -> Result<bool>;
}
So I'll work on a fork that expose this enough functions to fit this API, and I'll use it in async-compression
crate.
It seems that when the read / write buffer is smaller than the total size of the lz4f frame, the AsyncReadCompressor
is not waked up properly.
Here is my repoduction example. I'm using my async_std
branch for simplicity.
It doesn't appear Preferences
supports setting the compression level. Both the field compression_level
and method set_compression_level
are private; is there an alternative way I'm missing?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.