Comments (2)
A bit late, but the reason this input was so slow is because it starts new "static Huffman" blocks very frequently. In fact the worst case of "all empty static Huffman blocks" was almost 10x slower than this reproducer even: only 0.36 MB/s vs. several hundred MB/s for typical data.
It turned out a simple optimization is possible. So with commit 57cab07 now it's much faster: on the same system, 175 MB/s for the all empty blocks case, and 219 MB/s for this reproducer.
Now, the worst known case is "all empty dynamic Huffman blocks", which is still only 5 MB/s on the same system. Unfortunately that is much harder to improve due to how the DEFLATE format works, and even zlib is only 10 MB/s for it...
Anyway, thanks for the useful report, which resulted in a big improvement!
from libdeflate.
Awesome fix, thanks for looking into it!
from libdeflate.
Related Issues (20)
- Build type does not default to Release with CMake 3.10 and earlier HOT 5
- Version `1.8` is able to decompress data that version `1.17` cannot, but only on Windows HOT 9
- Comparison to zlib and its forks HOT 2
- POSITION_INDEPENDENT_CODE ON for static compilations HOT 2
- v1.14 - v1.18 deflate compatibility HOT 5
- Decompressing Microsoft DOCX Files in Unreal Engine 5 with Libdeflate HOT 4
- Windows Explorer cannot unpack some files generated with libdeflate (reward offered) HOT 9
- Tests fail on Windows in Release mode with VS platform toolset v141 HOT 1
- Please also generate a static source code archive for release HOT 2
- Libdeflate compressed stream in kunzip fail to decompress HOT 12
- Slow level 0 "compression" HOT 2
- I added stream & multi-thread support for libdeflate HOT 16
- ESA matchfinder HOT 4
- crc32 is twice as fast in zlib-ng HOT 19
- how to change sliding window size at runtime? HOT 1
- arm_acle.h compiler errors when building for iOS with Xcode 15.3 HOT 4
- analyzer warning regression: The left operand of '+' is a garbage value due to array index out of bounds HOT 1
- Unaligned access on PowerPC: forgotten macros in `common_defs.h` HOT 11
- 32-bit build error when using cctools 949.0.1 with assembler that doesn't support AVX instructions HOT 7
- VNNI support seems to require something later than GCC 11.1 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from libdeflate.