Comments (19)
Could you let me know:
- which compressor impl you are using (running System.out.println on the compressor would be enough)
- if you are using direct or heap byte buffers as inputs and outputs
In addition, if you have a reproducible test failure, this would be great to understand what is happening here. :-)
from lz4-java.
I really appreciate your quick response and willingness to help.
I m using LZ4JNICompressor to compress and LZ4JNISafeDecompressor to decompress.
I have also tried LZ4JNIFastDecompressor toi decompress but same result.
Caused by: net.jpountz.lz4.LZ4Exception: Error decoding offset 39 of input buffer
at net.jpountz.lz4.LZ4JNIFastDecompressor.decompress(LZ4JNIFastDecompressor.java:66) ~[lz4-1.3.0.jar:na]
I also have tried decompression creating a temporary byteArray for byteArray to byteArray decompression , it also works fine.
And yes we have a reproducing situation, i will assist you with any information.
from lz4-java.
To confirm the issue is with the JNI impl, could you make sure the problem does not reproduce if you use one of the Java impls?
from lz4-java.
Java impls ? I didnt get you.
If i do the decompression using a intermediate bytearray, its works fine.
If i do decompression using a temp ByteBuffer as a source ByteBuffer, it works fine.
It appears that the source ByteBuffer gets altered while decompressing while decompressing directly.
Please feel free to ask if you want me to carry out any other tests .
from lz4-java.
By Java impls, I meant using eg LZ4Factory.safeInstance instead of LZ4Factory.fastestInstance.
from lz4-java.
I will test it with LZ4Factory.safeInstance and will let you know the result tomorrow.
from lz4-java.
Hi Adrien,
I tried using LZ4Factory.safeInstance for both compressor and decompressor, i got following error while decompressing. Surprisingly, with safest Instance I got error for the first message itself unlike fastestInstance where I used to get error for second messahge onwards.
But when I tried using temporary buffer(copy of Source) as source buffer todecompress with safest Instance, it worked fine.
Caused by: net.jpountz.lz4.LZ4Exception: Malformed input at 15
at net.jpountz.lz4.LZ4JavaSafeSafeDecompressor.decompress(LZ4JavaSafeSafeDecompressor.java:81) ~[lz4-1.3.0.jar:na]
at net.jpountz.lz4.LZ4JavaSafeSafeDecompressor.decompress(LZ4JavaSafeSafeDecompressor.java:116) ~[lz4-1.3.0.jar:na]
at com.sungard.Core.Protocol.NGOProtocolWithCompression.deCompress(NGOProtocolWithCompression.java:154) ~[bin/:na]
at com.sungard.Core.Protocol.NGOProtocolWithCompression.decomposeBuffer(NGOProtocolWithCompression.java:101) ~[bin/:na]
from lz4-java.
If the safe impl fails too then I'm wondering that your code might be misusung the API, because it should really never modify the source content. However, if you use the compress/decompress methods that don't take offset, the offset of the byte buffers will be updated, maybe this is the cause of your issue? Can you try to build a reproducible unit test that demonstrates the problem?
from lz4-java.
Hi Adrien,
I would try to give you a simple scenario which reproduces issue for me.
Lets say
- I have a byteBuffer with limit = y
- I am compressing this bytebuffer from offset x to offset y-1 into a temporary Bytebuffer.
- Then we copy the temporary ByteBuffer content to the Original Bytebuffer.
- We sent the Original ByteBuffer Over the network.
- Upon receiving the compressed bytebuffer , we try to decompress bytebuffer from offset x to offset (y-1), We get error
"error decoding offset " - However, if we copy the received bytebuffer into a temporary bytebuffer and then try to decompress the temporary bytebuffer from offset x to offset (y-1), decompression get successful.
- Alternatively, Even if we convert the received bytebuffer into a bytearray and try decompression on the bytearray, it works.
Using a temporary ByteBuffer is not a sustainable solution for me, Please see if you can help me in any way.
from lz4-java.
Hi Adrien,
Strangely, Decompression works for me fine when i use the bytearray API of decompressor.
int returnVal = deCompressor.decompress(src.array(), src.arrayOffset() + srcOffset,srcLength, target.array(),target.arrayOffset()+ targetOffset, targetLength);
I have one more query,
I m getting quite good speed in Decompression around 1200MB/s.
However, I m not getting good speed in compression around 100 MB/s
Can you please suggest some ways to SPEED up COMPRESSION.
I m using following code to compress :
static LZ4Factory factory = LZ4Factory.fastestInstance();
public static LZ4Compressor compressor = factory.fastCompressor();
public static LZ4SafeDecompressor deCompressor = factory.safeDecompressor();
int returnVal = compressor.compress(src, srcOffset, srcLength, Target, targetOffset, targetLength);
from lz4-java.
Hi Guys,
Any update on this ?
I'm experiencing exactly the same error :-/
Cheers,
Francesco
from lz4-java.
The following test shows what actually fails :
public class DeCompressionTest {
private static LZ4Factory lz4Factory;
private static LZ4Compressor lz4Compressor;
private static LZ4FastDecompressor lz4Decompressor;
static {
lz4Factory = LZ4Factory.fastestInstance();
lz4Compressor = lz4Factory.fastCompressor();
lz4Decompressor = lz4Factory.fastDecompressor();
}
@Test
public void testFastDeCompressionSpecialChars () {
String toCompress = "2ο Χέρι. Τοποθετήστε το Στοίχημα Εδώ.";
byte[] compressedBuffer = lz4Compressor.compress(toCompress.getBytes());
byte[] decompressedBuffer = new byte[toCompress.length()];
lz4Decompressor.decompress(compressedBuffer, decompressedBuffer);
Assert.assertArrayEquals(toCompress.getBytes(), decompressedBuffer);
}
@Test
public void testDeCompressionStandardChars () {
String toCompress = "This is a standard translation string";
byte[] compressedBuffer = lz4Compressor.compress(toCompress.getBytes());
byte[] decompressedBuffer = new byte[toCompress.length()];
lz4Decompressor.decompress(compressedBuffer, decompressedBuffer);
Assert.assertArrayEquals(toCompress.getBytes(), decompressedBuffer);
}
@Test
public void testSafeDeCompressionSpecialChars () {
LZ4SafeDecompressor lz4Decompressor;
lz4Factory = LZ4Factory.safeInstance();
lz4Compressor = lz4Factory.fastCompressor();
lz4Decompressor = lz4Factory.safeDecompressor();
String toCompress = "2ο Χέρι. Τοποθετήστε το Στοίχημα Εδώ.";
byte[] compressedBuffer = lz4Compressor.compress(toCompress.getBytes());
byte[] decompressedBuffer = new byte[toCompress.length()];
lz4Decompressor.decompress(compressedBuffer, decompressedBuffer);
Assert.assertArrayEquals(toCompress.getBytes(), decompressedBuffer);
}
}
from lz4-java.
Hi,
I've just realised the test were bugged because of the following line 👍
byte[] decompressedBuffer = new byte[toCompress.length()];
replacing with :
byte[] decompressedBuffer = new byte[toCompress.getBytes().length];
fixed the issue :)
from lz4-java.
Hey thanks,
I will try to check this solution to see if it works
On Dec 10, 2015 4:17 PM, "fflatorre" [email protected] wrote:
Hi,
I've just realised the test were bugged because of the following line [image:
👍]byte[] decompressedBuffer = new byte[toCompress.length()];
replacing with :
byte[] decompressedBuffer = new byte[toCompress.getBytes().length];
fixed the issue :)
—
Reply to this email directly or view it on GitHub
#68 (comment).
from lz4-java.
Hi Adrien,
I was studying the LZ4 algorithm. I have a query, How do we identify if a
LZ4 sequence has ended and next LZ4 sequence is started?
Can you please answer the query, I will be really helpful to me.
On Thu, Dec 10, 2015 at 4:17 PM, fflatorre [email protected] wrote:
Hi,
I've just realised the test were bugged because of the following line [image:
👍]byte[] decompressedBuffer = new byte[toCompress.length()];
replacing with :
byte[] decompressedBuffer = new byte[toCompress.getBytes().length];
fixed the issue :)
—
Reply to this email directly or view it on GitHub
#68 (comment).
Vaibhav Hajela
from lz4-java.
Hi @vaibhavhajela check the generated files: build\java\net\jpountz\lz4\LZ4JavaSafeFastDecompressor.java, of cause after the building.
it's very clear about the lz4 sequence what you want.
A simple description as follows:
lz4compressed=[token-data][token-data]....
token-data=(RunLength,MatchLegth,Offset)=[half-half][run length additional bytes....][data][short little endian offset][match length additional bytes...]
if RunLength or MatchLength<15(half byte) there are no additional bytes followed
The additional bytes used while(x-=255) loop, that's 255*times+remainder, Indeed this part can be replaced with other shceme, such as varint.
When decompressed sequence clear, then the compressor part is clear too.
the original document at here http://fastcompression.blogspot.in/2011/05/lz4-explained.html
from lz4-java.
Was the original problem of modified source ByteBuffer solved?
from lz4-java.
from lz4-java.
Let me close this issue now. If you encounter the same problem again, please reopen this with a reproducible test case.
from lz4-java.
Related Issues (20)
- Premature end of data while decompressing HOT 3
- CVE-2021-3520 Whether the vulnerability affects LZ4-Java HOT 2
- why lz aarch64 version is 1.9.1
- How to decopress large LZ4 files? HOT 1
- the value don't equal xxhsum,the value length is 15 HOT 2
- how to build 'arm64/liblz4-java.so' ? HOT 1
- Data larger than 4194304 Bytes is not decompressed correctly HOT 2
- lz4 legacy format support (typically for Linux Kernel compression) HOT 1
- LZ4 java with double buffering
- LZ4 decompression exception Malformed input at
- Exception in thread "main" net.jpountz.lz4.LZ4Exception: Malformed input at 16 HOT 3
- Add riscv support
- Stream ended prematurely when trying to readAllBytes
- Compile with frame pointers?
- Another release? HOT 5
- Enable support for riscv64 HOT 1
- JVM crashes at insertAndFindBestMatch
- Add more maintainers to `lz4-java` repo HOT 1
- add option to prefer java version when a library is hardcoded to LZ4Factory.fastestInstance
- Include generated source files in sources JAR deployed to Maven Central
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lz4-java.