r-lyeh-archived/bundle

:package: Bundle, an embeddable compression library: DEFLATE, LZMA, LZIP, BZIP2, ZPAQ, LZ4, ZSTD, BROTLI, BSC, CSC, BCM, MCM, ZMOLLY, ZLING, TANGELO, SHRINKER, CRUSH, LZJB and SHOCO streams in a ZIP file (C++03)(C++11)

bundle's Issues

memory in-place encoding/decoding

@todo

fse.c included twice

since db55633 (last commit :)), when building for clang/gcc, "fse.c" is included twice, causing symbols defined twice

stream support

@todo
write zip as stream rather than single memory archive block
also, flush to disk on the fly or support std::i/o-streams at least

consider snappy

It would be great to have snappy as part of the benchmark.

try also...

lz4-hc
snappy
lzturbo or tornado
libbsc
FiniteStateEntropy

from wikipedia

MPQs used in Blizzard's games generally contain a game's data files, including graphics, sounds, and level data. The format's capabilities include compression, encryption, file segmentation, extensible file metadata, cryptographic signature and the ability to store multiple versions of the same file for internationalization and platform-specific differences. MPQ archives can use a variety of compression algorithms which may also be combined.

bring back bzip2 support (by popular demand)

to consider

richox/zmolly

to consider

https://github.com/kmar/mlz
https://github.com/ebiggers/xpack
http://encode.ru/threads/2427-GLZA
balz, lz4x, crush11, bcm120, jq,
lzhl, ppmd_sh, wflz, zlite,
smaz,
brieflz, liblzg, sflate, smallz4, unlzw, zx7

new contenders (enwik8 tests)

// tangelo 2.4, faster zpaq, less filters
~/> tangelo c enwik8 enwik8.tangelo
Compressed from 100000000 -> 21031051 bytes. 97.24 secs

// bcm
~/> bcm enwik8 enwik8.bcm
enwik8: 100000000 -> 22531851 in 26.419s
26.5055 s.

// zmolly 0.0.1
~/> zmolly-x86_64 -e enwik8 enwik8.zmolly
encode-block: 16777216 => 4074603
encode-block: 16777216 => 3940888
encode-block: 16777216 => 3935657
encode-block: 16777216 => 3935740
encode-block: 16777216 => 3924577
encode-block: 16113920 => 3752436
10000000 -> 23.563.956 
16.9817 s.

// libzling (fast)
~/> zling e0 enwik8 enwik8.zling0
encode: 100000000 => 32378861, time=3.732 sec, speed=26.795 MB/sec

// libzling (extra)
~/> zling e4 enwik8 enwik8.zling4
encode: 100000000 => 30707658, time=6.457 sec, speed=15.487 MB/sec

// MCM v0.83 (mode=6)
mcm -t ..\data\enwik8 ..\data\enwik8.mcmt
100,000,000->21,020,366 in 39.107s
mcm -f ..\data\enwik8 ..\data\enwik8.mcmf
100,000,000->19,922,132 in 42.336s
mcm -m ..\data\enwik8 ..\data\enwik8.mcmm
100,000,000->19,201,503 in 50.718s
mcm -h ..\data\enwik8 ..\data\enwik8.mcmh
100,000,000->19,166,857 in 60.895s
mcm -x ..\data\enwik8 ..\data\enwik8.mcmx
100,000,000->19,057,576 in 69.599s

// MCM v0.83 (mode=9)
mcm -t9 ..\data\enwik8 ..\data\enwik8.mcmt
100,000,000->20,180,965 in 44.387s
mcm -f9 ..\data\enwik8 ..\data\enwik8.mcmf
100,000,000->19,305,599 in 47.988s
mcm -m9 ..\data\enwik8 ..\data\enwik8.mcmm
100,000,000->18,624,824 in 53.92700s
mcm -h9 ..\data\enwik8 ..\data\enwik8.mcmh
100,000,000->18,501,425 in 70.372s
mcm -x9 ..\data\enwik8 ..\data\enwik8.mcmx
100,000,000->18,377,349 in 77.48600s

// discarded

// crush (extra)
~/> crush cx enwik8 enwik8.crush
Compressing enwik8...
100000000 -> 31731711 in 489.281s
489.361 s.

// crush (fast)
~/> crush cf enwik8 enwik8.crushf
Compressing enwik8...
100000000 -> 37308893 in 5.235s
5.33109 s.

consider pre-pass of filters

like fse, rolz or xwrt

lib & license directives

Basically,

BUNDLE_NO_BSD2
BUNDLE_NO_BSD3
BUNDLE_NO_MIT
BUNDLE_NO_APACHE2

BUNDLE_NO_BROTLI
BUNDLE_NO_EASYLZMA
BUNDLE_NO_LZMA
BUNDLE_NO_LZIP
BUNDLE_NO_ZPAQ
BUNDLE_NO_BSC
BUNDLE_NO_LZ4
BUNDLE_NO_MINIZ
BUNDLE_NO_SHOCO
BUNDLE_NO_ZSTD
BUNDLE_NO_SHRINKER
BUNDLE_NO_CSC

where's a makefile?

Hi,

Is this a command line utility or a library ? I'm missing a makefile somewhere, any plans of using one ?

Segmentation fault with v2.0.1

I'm getting a segfault after having switched to the current master with my compression benchmark. I'm on OSX 10.10 with Clang 3.7. Here's a backtrace:

(lldb) process launch -i data/pcap/2009-M57-day11-18-10k.pcap
Process 45472 launched: '/Users/mavam/Dropbox/code/compbench/benchmark-debug' (x86_64)
Algorithm   Raw Packed  Unpacked    Compression Decompression
RAW 2772093 2772093 2772093 3743    1162
LZ4F    2772093 1643265 2772093 4821    837
MINIZ   2772093 1519520 2772093 150800  7096
LZIP    2772093 949119  2772093 526206  60906
LZMA20  2772093 952561  2772093 471057  58925
ZPAQ    2772093 943286  2772093 8675433 8734077
LZ4 2772093 1540706 2772093 59946   903
BROTLI9 2772093 963956  2772093 263584  6107
ZSTD    2772093 1003409 2772093 215553  2119
LZMA25  2772093 949055  2772093 571734  58700
BSC 2772093 1072946 2772093 347131  299024
BROTLI11    2772093 936971  2772093 5798909 10169
SHRINKER    2772093 1598614 2772093 6424    1650
CSC20   2772093 957509  2772093 454960  67649
ZSTDF   2772093 1223905 2772093 7042    2316
BCM 2772093 1220156 2772093 398050  530257
ZLING   2772093 1061986 2772093 72908   20161
MCM 2772093 928892  2772093 1263611 1157556
TANGELO 2772093 990425  2772093 1658184 1796580
benchmark-debug was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 45472 stopped
* thread #1: tid = 0x11f6df, 0x00000001000e2553 benchmark-debug`ppm_model_t::current_o4(this=0x00007fff5fb3f170) + 419 at bundle.cpp:108648, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00000001000e2553 benchmark-debug`ppm_model_t::current_o4(this=0x00007fff5fb3f170) + 419 at bundle.cpp:108648 [opt]
   108645               }
   108646           } else {  // not found -- create new node for context
   108647               o4 = (sparse_model_t*)m_o4_allocator.alloc(sizeof(sparse_model_t));
-> 108648               memset(o4->m_symbols, 0, sizeof(o4->m_symbols));
   108649               o4->m_context = m_context;
   108650               o4->m_sum = 0;
   108651               o4->m_cnt = 0;
(lldb) bt
* thread #1: tid = 0x11f6df, 0x00000001000e2553 benchmark-debug`ppm_model_t::current_o4(this=0x00007fff5fb3f170) + 419 at bundle.cpp:108648, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame #0: 0x00000001000e2553 benchmark-debug`ppm_model_t::current_o4(this=0x00007fff5fb3f170) + 419 at bundle.cpp:108648 [opt]
    frame #1: 0x00000001000ced23 benchmark-debug`ppm_model_t::encode(this=0x00007fff5fb3f170, coder=0x00007fff5fb3efe0, c=212) + 35 at bundle.cpp:108665 [opt]
    frame #2: 0x0000000100094702 benchmark-debug`zmolly_encode(fdata=0x00007fff5fbbcbc0, fcomp0=0x00007fff5fbbcb20, block_size=<unavailable>) + 1602 at bundle.cpp:108871 [opt]
    frame #3: 0x0000000100097106 benchmark-debug`bundle::pack(q=<unavailable>, in=<unavailable>, inlen=2772093, out=<unavailable>, outlen=0x00007fff5fbfeac0) + 4854 at bundle.cpp:109853 [opt]
    frame #4: 0x0000000100001279 benchmark-debug`main + 222 at bundle.hpp:189 [opt]
    frame #5: 0x000000010000119b benchmark-debug`main [inlined] std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > bundle::pack<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(q=20) + 18 at bundle.hpp:220 [opt]
    frame #6: 0x0000000100001189 benchmark-debug`main + 393 at benchmark.cpp:63 [opt]
    frame #7: 0x00007fff968ff5c9 libdyld.dylib`start + 1
(lldb) 

After switching to master, I only updated the existing libraries:

--- a/benchmark.cpp
+++ b/benchmark.cpp
@@ -49,9 +49,11 @@ auto main() -> int {
 #endif
   std::string buffer{std::istreambuf_iterator<char>{std::cin},
                       std::istreambuf_iterator<char>{}};
-  std::vector<unsigned> libs{RAW, SHOCO, LZ4F, MINIZ, LZIP, LZMA20, ZPAQ, LZ4,
-                             BROTLI9, ZSTD, LZMA25, BSC, BROTLI11, SHRINKER,
-                             CSC20};
+  std::vector<unsigned> libs{
+    RAW, SHOCO, LZ4F, MINIZ, LZIP, LZMA20, ZPAQ,
+    LZ4, BROTLI9, ZSTD, LZMA25, BSC, BROTLI11, SHRINKER,
+    CSC20, ZSTDF, BCM, ZLING, MCM, TANGELO, ZMOLLY
+  };
   auto to_mus = [](std::chrono::high_resolution_clock::duration d) {
     return std::chrono::duration_cast<std::chrono::microseconds>(d).count();
   };

Since I get output up to TANGELO, this could be an issue with ZMOLLY.

Compiler warnings

When compiling bundle (with GCC) you'll get a lot of warnings. I simply wanted to know if you'd like to reduce these or just keep the "embedded" external library sources exactly the same?

What's the optimum minimum length for compression for brotli and shoco?

What's the optimum minimum length for compression for brotli and shoco?

i think optimum min length also should be tested.

how to compile .bun and .zip

Hi, I generated the bundle.dll and bundle.lib. I want to use archive.bun or archive.zip to write a file.
But, the program report link error 2019. because could not find .bun & .zip in lib and dll.
My IDE is VS2017.