powturbo/turbobench

Compression Benchmark

TurboBench:Compressor Benchmark

• High performance precision, zero overhead. One might call it the compression lie detector
• The only benchmark program including LzTurbo
• 100% in-memory benchmark, no I/O overhead
• Include (>70) allmost all popular, latest or fastest compressors in one compiled package
• Benchmarking Entropy Coders, Lz77, Rolz, BWT and Context mixing compressors
• Multiple input files with recursive directories
• Concatenate multiple small files into one multiblock file
• Avoid cache szenario and cpu throttling found in other benchmarks
• Automatic sort, update and merge of result files
• Text, html, csv, markdown output
• ✨ Transfer speed sheet for different connections or devices: GPRS,2G,3G,4G,DSL,Network,HDD,SSD,RAM
• 👍 Html plot: Speedup + Speed/Ratio
• All in one executable, no hassless installing of additional packages, graphic libraries, python,...
• 👍 build in peak memory usage reporting for compression and decompression in html output

Release:

• Build or Download turbobench from releases

• List all plugins: "./turbobench -l2"
  
• List all compiled codecs: "./turbobench -l1"
  
• type "./turbobench -h" for help

Benchmark:

• see also: Compression Benchmark

CPU: Sandy bridge i7-2600k at 4.4GHz, gcc 6.3, single thread.

• Realistic and practical benchmark with large files

• No PURE cache benchmark

• All compressors with latest versions (2017-05-10)

• File app3.tar binary Portable Apps Suite

(bold = pareto) MB=1.000.000

C Size	ratio%	C MB/s	D MB/s	Name
32823983	32.8	3.40	67.92	lzma 9
32872154	32.8	0.31	315.27	brotli 11d27
32925079	32.9	1.70	70.67	lzturbo 49
33936389	33.9	2.57	1701.35	lzturbo 39
34105370	34.1	3.32	952.59	zstd 22
36751363	36.7	48.30	1701.59	lzturbo 32
36920708	36.7	2.98	2355.32	lzturbo 29
46546059	46.5	163.77	1489.57	lzturbo 31
46805879	46.8	44.66	940.64	zstd 9
48152545	48.1	52.94	349.62	brotli 4
49497505	49.4	2.48	2299.20	lizard 49
49773790	49.7	38.08	1952.73	lzturbo 22
49860700	49.8	16.94	295.99	zlib 9
49962678	49.9	35.70	294.24	zlib 6
50278958	50.2	282.43	1372.91	lzturbo 30
52509931	52.5	290.96	347.16	brotli 1
52549655	52.5	239.35	2153.41	lzturbo 21
52928477	52.9	69.17	276.75	zlib 1
52983490	52.9	393.67	984.00	zstd 1
54251482	54.2	2.60	4367.15	lzturbo 19
54410769	54.4	46.37	3305.22	lz4 9
55923645	55.9	188.40	4200.23	lzturbo 12
57606731	57.6	386.90	3948.64	lzturbo 11
59085723	59.0	698.39	2196.24	lzturbo 20
61455711	61.4	800.71	4003.54	lzturbo 10
61938605	61.9	730.46	3330.40	lz4 1
100098564	100.0	8647.84	8408.10	memcpy

---

Hardware: ODROID C2 - ARM 64 bits - 2Ghz CPU, OS: Ubuntu 16.04, gcc 5.3
Compressors with versions 16.08.2016
pd3d.tar - 3D Test Set (RAD Game Tools)

C Size	ratio	C MB/s	D MB/s	Name
8052040	25.2	0.53	23.23	lzma 9
9092280	28.4	0.08	52.61	brotli 11
9159574	28.7	0.52	119.76	lzturbo 39
9691094	30.3	0.68	94.02	zstd 22
9826984	30.7	3.24	136.91	lzturbo 32
10264073	32.1	26.15	142.28	lzturbo 30
10427322	32.6	4.90	108.76	zstd 9
10938385	34.2	9.46	110.38	lzfse
10966870	34.3	8.92	101.96	zstd 5
11059511	34.6	1.74	98.16	zlib 9
11121480	34.8	7.63	97.47	zlib 6
12649309	39.6	0.61	366.17	lzturbo 29
13302907	41.6	19.07	435.28	lzturbo 21
14237494	44.5	0.66	500.67	lzturbo 19
14283317	44.7	10.04	329.14	lz4 9
14723054	46.1	103.21	483.81	lzturbo 20
14814049	46.4	8.14	484.09	lzturbo 12
16069593	50.3	121.12	365.08	lz4 1
16166867	50.6	111.43	475.66	lzturbo 10
31952896	100.0	1676.10	1704.00	memcpy

- Groups FASTEST,FAST,EFFICIENT,MAX,OPTIMAL,BWT:

• test all fast compressors in the lz4, lzturbo, zlib class
  (additional groups can be defined in the "turbobench.ini" file)

  ./turbobench -eFAST file
  

• Some codecs can accept additional parameters (see option -l1 or file plugins.cc) (colon ':' as separator)

  ./turbobench -elzma,9mt2:d29:a1:fb273:mf=bt4:mc999:lc8:lp0:pb2 file
  

- Codecs:

• individual codec test (output to screen & file.tbb)
  

  ./turbobench -elzturbo,19,29,39/brotli,6/zlib,6 file
  

• retest or test other compressors and merge the results to file.tbb
  

  ./turbobench -eFAST/bzip2 file
  

- Print + Plot

• Print result file + "transfer+decompression speedup" plot to file.html for browsing

  ./turbobench -p2 -S2 file.tbb

Compile:

	git clone --depth=1 --recursive https://github.com/powturbo/TurboBench.git
    cd TurboBench
	make

Turbobench mini: compile (only popular codecs)

	make NCODEC2=1

Turbobench incl. Entropy Coder codecs

    make EC=1

Environment:

OS/Compiler (32 + 64 bits):

• Windows: MinGW-w64 makefile
• Linux amd/intel: GNU GCC (>=4.6)
• Linux amd/intel: Clang (>=3.2)
• Linux arm: aarch64
• MaxOS: XCode (>=9)
• PowerPC ppc64le
• IBM Z series s390x

Plugins:

Compressor Lz77,Rolz,Bwt,zpaq:

• LzTurbo
• Blosc
• BriefLz
• Brotli
• Bzip2
• Chameleon
• CSC
• Density
• Doboz
• FastLz
• Fast-lzma
• glza
• heatshrink
• Intel(R) Intelligent Storage Acceleration Library
• bsc
• Libdeflate
• LibLZF
• LibLz
• LibSLZ
• Lz4
• Lz4ultra
• lzjody 🆕2023.06
• lizard
• Lzfse
• Lzham v1.1
• Lzlib
• Lzmat
• Lzma
• Lzo
• Lzoma
• LZSSE
• LZSA
• Miniz
• ms-compress
• Nakamichi Washigan
• Oodle (windows + linux) 🆕2023.05
• Quantile Compression
• Pithy
• Quicklz
• sap
• shoco
• Shrinker
• Smallz4
• Snappy
• Snappy-c
• Tornado
• Unishox
• wfLZ
• yalz77
• yappy v2011
• zlib
• zlib-ng
• zlib cloudflare
• xpack
• zopfli
• zstd
• zpaq

Entropy coder:

bitwise range coder

• TurboRC-Range Coder
• Bitwise RC v2010
• Bitwise vector RC v2012
• AOMedia AV1 entropy coder
• Daala entropy coder
• FastAri v15-10

bytewise range coder

• TurboAC
• subotin range coder v2000
• Fast AC v2006
• FQZ/PPMD Range Coder v15-03
• PPMD Range Coder v15-03
• sserangecoding 🆕2023.05

ABS: Asymmetric binary systems

• Fpaqc:Asymmetric Binary Coder v07-12

ANS: Asymmetric Numeral Systems

• TurboANX-ANS
• TurborANS
• Finite State Coder v15-05
• Finite State Entropy v16-08

Huffman Coding

• TurboHF-Huffmann v1.3
• Tornado Huf v0.6a
• zlib Huffmann v1.2.8
• Fast HF v2006
• FPC
• FSE Huff
• Polar Codes v10-07

memcpy

• inline memcpy
• library memcpy

Encoding:

• TurboRLE Turbo Run Length Encoding
• tcobs short messages compression with COBS framing 🆕2023.06
• TurboBase64 Turbo Base64 Encoding/Decoding
• fastbase64 Base64 Encoding
• base64 Fast Base64 stream encoder/decoder

Transform:

• bwt:libdivsufsort
• st: bsc schindler transform

Last update: 07 JUL 2023

turbobench's Issues

TurboBench: Dynamic/Static web content compression benchmark

file html8 : 100MB random html pages from a 1m Alexa Top sites corpus.
Number of pages = 1178
Average length = 84886 bytes
The pages (length + content) are concatenated into a single html8 file,
but compressed/decompressed separately using the multiblock mode in TurboBench.

compress: page1,page2,...pageN
decompress : Page1,page2,...pageN

In the speedup (see below) plots you can see the best compressors for content providers:

• brotli 11 is best for static content
• brotli 5 is best until 1MB/s network transfer speed
• libdeflate 6 is best from 1MB/s to 6MB/s (followed by brotli,4)
• igzip 1,2 is best for very fast networks > 10MB/s

Remarks:

• Note that for web-content, the difference in decompression speed between zlib, brotli and others is practically imperceptible to the user.
• Actually there are only web server support for zlib, zlib_ng, slz and brotli.
• zstd is included only as reference as it is not compatible with web content encoding.

Unlike other benchmarks on the net, this is pure memory benchmark without any additional (http) server overhead.

Page Statistics:
Page Length: Minimum = 16kb, Maximum = 128kb

bits histogram:
15:######################## 24%
16:####################################### 39%
17:##################################### 37%

Lenovo Ideapad Pro 5 / CPU 7840HS 3.8-5.1GHz, DDR5 6400MHz

C Size	R%	C MB/s	D MB/s	Name	C Mem	D Mem	C Stack	D Stack
16457986	16.5	1.41	612.34	brotli 11	10,629,680	247,672
18579017	18.6	60.35	703.56	brotli 5	10,711,032	199,384
18996682	19.0	0.53	1787.41	zstd 22	815,429,856	191,952
19615056	19.6	0.34	1003.78	zopfli	33,644,120	14,352
19766557	19.8	7.10	1661.60	libdeflate 12	18,027,032	23,152
19971457	20.0	135.89	716.37	brotli 4	10,153,192	198,904
19974358	20.0	5.64	2092.36	zstd 15	72,091,872	191,952
20282366	20.3	96.65	1700.68	libdeflate 9	1,336,592	23,152
20363869	20.4	55.00	687.49	zlib 9	274,096	14,320
20451000	20.5	84.95	1130.76	zlib_ng 9	36 GB	778,098,680
20485533	20.5	92.93	682.52	zlib 6	274,096	14,320
20502399	20.5	201.65	1702.04	libdeflate 6	1,340,024	23,152
20568950	20.6	78.85	2004.33	zstd 5	5,503,608	191,952
20592048	20.6	173.56	1141.23	zlib_ng 6	36 GB!	778,098,680
21624240	21.6	242.98	702.77	brotli 2	9,235,608	231,792
22159165	22.2	434.23	1733.07	libdeflate 1	407,544	23,152
22484852	22.5	370.72	1483.97	igzip 3	696,336	0	7,340,032	7,340,032
22867303	22.9	374.92	2064.67	zstd 1	1,374,840	191,952
23098360	23.1	766.27	1446.97	igzip 2	663,568	0	7,340,032	7,340,032
23153034	23.2	518.39	678.09	brotli 1	1,200,512	165,704
23287503	23.3	850.51	1444.42	igzip 1	569,336	0	7,340,032	7,340,032
23723266	23.7	238.93	629.48	zlib 1	274,096	14,320
25355327	25.4	982.05	1543.00	igzip 0	1,032	0	7,340,032	7,340,032
28214601	28.2	573.26	692.20	slz 6	0	14,320	65,536
28214601	28.2	576.88	684.35	slz 9	0	14,320	65,536
29476316	29.5	587.78	679.40	slz 1	1,032	14,320	65,536
30040886	30.0	569.15	1078.03	zlib_ng 1	25 GB	778,098,680

R: Compression ratio
Mem: Heap Memory usage in bytes
Stack: Stack Memory usage in bytes
Zstd is added only as indication. There is actually no zstd content-encoding for the web

speedup files

turbobench doesn't work

I have a i3-4005U cpu, running ubuntu 16.04. I just tried to clone and run this program, with the command ./turbobench -p2 -S2 file.tbb &> error_output.txt then the program aborts with exit code 255 and the output of error_output.txt is <86> <AD><FB> (less puts < and > with hexcode instead of showing raw characters). file.tbb was not written

Support ZSTD's dictionary compression

Does TurboBench support the ZSTD dictionary compression performance test? I want to try to compare the performance of ZSTD compression use the dictionary with other algorithms in small data mode.

Compile failed on MacBook Pro (M1)

I got Error message as follow when make project:

In file included from lzham_codec_devel/lzhamdecomp/lzham_core.h:296:
lzham_codec_devel/lzhamdecomp/lzham_platform.h:27:25: error: unrecognized instruction mnemonic
   __asm__ __volatile__("pause");
                        ^
<inline asm>:1:2: note: instantiated into assembly here
        pause
        ^....

Then I modified the makefile, set "LZHAM=0" and try again：

Error message as follow:

Turbo-Range-Coder/include_/sse_neon.h:232:85: error: invalid conversion between vector type 'uint64x2_t' (vector of 2 'uint64_t' values) and 'uint8x8_t' (vector of 8 'uint8_t' values) of different size
static ALWAYS_INLINE uint64_t  mm_movemask4_epu8(__m128i v) { return vgetq_lane_u64((uint64x2_t)vshrn_n_u16((uint8x16_t)v, 4), 0); } //uint8x16_t

zstd and brotli both produce incorrect output for big file

Hi. zstd and brotli both produce incorrect output for file https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd/debian-12.0.0-amd64-DVD-1.iso (3.7 G). Here is output:

# ./turbobench -D -B4G -ebrotli,1/zstd,1 /tmp/debian-12.0.0-amd64-DVD-1.iso 
           0     0.0      -0.00   -1023.18   brotli 1         debian-12.0.0-amd64-DVD-1.iso
ERROR at 868478448:3f, 66 file=debian-12.0.0-amd64-DVD-1.iso
  3677323496    93.5    -240.61    -689.08   zstd 1           debian-12.0.0-amd64-DVD-1.iso
ERROR at 869007360:34, cb file=debian-12.0.0-amd64-DVD-1.iso
TurboBench:  - Sun Jun 11 10:13:25 2023

      C Size  ratio%     C MB/s     D MB/s   SCORE      Name            File
           0     0.0       0.00   11053.89   167423219872854250310480057943336997924438274955232624522333477440436613541123027690304032184897808517832736441413296818951444073477260938629943455293343883926095357701695786860068785110851124808262911167419640293628520552910140667265102788183953006076438190605626183719130397942773345747264478380032.00   ?brotli 1       debian-12.0.0-amd64-DVD-1.iso
  3677323496    93.5    2599.46    7444.48    3679.89   ?zstd 1         debian-12.0.0-amd64-DVD-1.iso

Test was performed in docker container with debian sid in it. Using gcc 12.2.0 from debian. Using turbobench commit 1d17683 .

Here is full reproducer for docker:

FROM debian:sid-20230522
ENV LC_ALL C.UTF-8
RUN apt-get update && apt-get install -y apt-utils whiptail
RUN apt-get update && apt-get install -y git make gcc g++ wget ca-certificates
RUN git clone --recurse https://github.com/powturbo/TurboBench
WORKDIR TurboBench
RUN git checkout 1d17683531537ca953ae7200fb36198ad5ab2ae6
RUN make
RUN wget 'https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd/debian-12.0.0-amd64-DVD-1.iso'
RUN ./turbobench -D -B4G -ebrotli,1/zstd,1 debian-12.0.0-amd64-DVD-1.iso

ZStandard Presets

Your default levels for ZSTD seem a little wonky:

• TURBO: ZSTD is missing, it should be -5 | --fast=5 (the fastest level currently available). This should push it into LZ4 territory.
• EFFICIENT should probably include 3, ZSTD's default level.
• OPTIMAL (meaning best trade off?) should probably be 20, as 22 balloons memory usage and requires another flag in order to use it.

You might want to comment the .ini file, as it's not clear what each preset means (like BWT). Also, the README mentions includes a FASTEST preset the turbobench.ini file only has TURBO.

[Question] What's total difference between turbobench and lzbench?

Hi, just find this project, And not sure the really difference between turbobench and lzbench. Thanks

add codecs: xz, pixz, pigz

https://github.com/xz-mirror/xz

https://github.com/vasi/pixz

https://github.com/madler/pigz

TurboBench: Maximum Compression Benchmark - Skylake i7-6700 3.4GHz

Latest update: 2019.08.09

Binary:mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar

     C Size   ratio%     C MB/s     D MB/s   Name       (bold = pareto) MB=1.000.0000
    42179292     9.1       0.91     189.42   lzturbo 49      
    42457543     9.1       2.07     186.33   lzma 9d29:fb273:mf=bt4
    42988097     9.2       0.57     857.10   brotli 11d29    
    45871347     9.9       1.50    2206.52   lzturbo 39      
    46458069    10.0       2.66    1491.27   zstd 22d29      
    56248534    12.1       1.93    2966.68   lzturbo 29      
    58474096    12.6      16.25      27.38   bsc 0         
    58654705    12.6     125.48    2100.32   lzturbo 32      
    60760645    13.1      13.12      60.33   lzturbo 59      
    60765065    13.1      14.08      90.42   lzturbo 59t2    
    73060389    15.7       2.24    5717.94   lzturbo 19
    87416420    18.8      11.75    1397.49   zstd 15         
    88006994    18.9      45.49     514.92   brotli 5        
    88252775    19.0       1.78    2232.29   lizard 49       
    94672486    20.3       1.91    3073.42   lizard 29 
   111853281    24.0      12.56      39.18   bzip2           
   126614955    27.2       9.25     340.34   zlib 9          
   133921751    28.8       2.97    2230.60   lizard 39       
   143616390    30.9      38.77    3233.08   lz4 9           
   144311136    31.0       3.34    3519.31   lizard 19       
   145898393    31.3       1.51    3828.81   lzsse8 17       
   174569457    37.5     871.54    4503.78   lzturbo 10
   179181598    38.5     657.00    2866.63   lz4 1 
   465457156   100.0   13824.50   14242.87   memcpy          

English Text:Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar

     C Size   ratio%     C MB/s     D MB/s   Name        
    21905524    20.3       9.45      15.92   bsc 0         
    22279381    20.7       9.46      49.89   lzturbo 59      
    22280917    20.7      10.04      71.61   lzturbo 59t2    
    27070097    25.1       0.91      82.32   lzturbo 49      
    27526894    25.5       1.23      86.14   lzma 9d29:fb273:mf=bt4
    27600942    25.6       0.46     305.70   brotli 11d29    
    27945445    25.9       1.06     672.33   lzturbo 39      
    28008216    26.0       1.43     640.32   zstd 22d29      
    31570314    29.3       1.10    1124.56   lzturbo 29      
    31734097    29.4      12.91      29.08   bzip2           
    32696227    30.3      46.86     781.41   lzturbo 32      
    33249008    30.8       4.94     851.66   zstd 15         
    35077489    32.5       1.85     694.91   lizard 49       
    36277020    33.7      26.67     367.32   brotli 5        
    41420278    38.4       1.92     973.00   lizard 29       
    42104125    39.1       8.15    1145.53   lizard 39       
    42236588    39.2      12.32     243.79   zlib 9          
    43696364    40.5       9.52    3085.19   lzsse8 17       
    45341158    42.1       1.28    2958.75   lzturbo 19      
    47834366    44.4       8.36    1555.15   lizard 19       
    48214128    44.7      22.56    2825.87   lz4 9 
    67850583    63.0     438.16    3994.67   lzturbo 10
    68340992    63.4     358.85    2790.46   lz4 1          
   107784196   100.0   14225.18   14694.50   memcpy          

PDF: Learn-Hot-English_magazine_(18-issues).tar

      C Size  ratio%     C MB/s     D MB/s   Name 
   244285827    54.0       0.47     212.50   brotli 11d30    
   245265022    54.2       2.17      33.91   lzturbo 49      
   246117453    54.4       3.22    3616.72   lzturbo 39      
   246415683    54.5       3.48      34.90   lzma 9d30:fb273 
   246786277    54.6       3.84    2182.46   zstd 22d30      
   250400776    55.4      81.47    3913.83   lzturbo 32
   253265114    56.0       3.62    7728.73   lzturbo 19      
   253761470    56.1       3.53    6090.31   lzturbo 29      
   289049146    63.9       5.62    6516.19   lizard 29       
   289084982    63.9       5.30    5445.98   lizard 49       
   299881226    66.3      17.58    4348.64   zstd 15
   330124786    73.0      47.57    6174.21   lz4 9          
   330179011    73.0       8.09    6278.61   lizard 39       
   330769147    73.1      28.04     351.13   zlib 9
   330921620    73.2       9.18    6710.84   lizard 19       
   334380724    73.9    1940.57    7960.84   lzturbo 10
   349951390    77.4       0.69    4303.37   lzsse8 17       
   368898163    81.6       0.69    2528.31   lzsse4 17       
   403005235    89.1       0.69    1329.14   lzsse2 17  
   452223492   100.0   13956.22   13771.35   memcpy 

Question about colon and mf=

I saw in turbobench.ini the following line LZMA9Z lzma,9mt2:d29:a1:fb273:mf=bt4:mc999:lc8:lp0:pb2

What is the meaning of the colon : and mf=? Also where can i read about what things like pb mean and what numbers after that are accepted?

Testing against Facebook's zstandard

Would be interesting to see results:
https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/
Thanks!

GCC 10 Build Fails

I don't think Turbobench and GCC10 are friends

gcc version 10.2.1 20200723 releases/gcc-10.2.0-3-g677b80db41 (Clear Linux OS for Intel Architecture)

/usr/bin/ld: glza/GLZAformat.o:/root/TurboBench/glza/GLZA.h:6: multiple definition of `params'; glza/GLZAcomp.o:/root/TurboBench/glza/GLZA.h:6: first defined here
/usr/bin/ld: glza/GLZAcompress.o:/root/TurboBench/glza/GLZA.h:6: multiple definition of `params'; glza/GLZAcomp.o:/root/TurboBench/glza/GLZA.h:6: first defined here
/usr/bin/ld: glza/GLZAencode.o:/root/TurboBench/glza/GLZA.h:6: multiple definition of `params'; glza/GLZAcomp.o:/root/TurboBench/glza/GLZA.h:6: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:45: multiple definition of `bin_code_length'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:37: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:39: multiple definition of `sum_nbob'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:38: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:40: multiple definition of `max_code_length'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:36: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:38: multiple definition of `queue_size'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:39: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:43: multiple definition of `queue_offset'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:35: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:44: multiple definition of `queue'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:41: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:38: multiple definition of `queue_size_az'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:39: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:38: multiple definition of `queue_size_other'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:39: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:38: multiple definition of `queue_size_space'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:39: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:41: multiple definition of `prior_is_cap'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:35: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:41: multiple definition of `prior_end'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:35: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:46: multiple definition of `queue_miss_code_length'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:37: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:41: multiple definition of `use_mtf'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:35: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:40: multiple definition of `UTF8_compliant'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:35: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:40: multiple definition of `max_regular_code_length'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:36: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:37: multiple definition of `num_base_symbols'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:40: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:45: multiple definition of `symbol_lengths'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:37: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:42: multiple definition of `cap_symbol_defined'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:36: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:42: multiple definition of `cap_lock_symbol_defined'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:36: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:40: multiple definition of `cap_encoded'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:35: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZA.h:6: multiple definition of `params'; glza/GLZAcomp.o:/root/TurboBench/glza/GLZA.h:6: first defined here

Compile error

In file included from turbobench.c:66:
time_.h:34:26: error: conflicting types for ‘uint64_t’; have ‘long long unsigned int’
34 | typedef unsigned __int64 uint64_t;
| ^~~~~~~~
In file included from /usr/include/sys/types.h:46,
from /usr/include/stdio.h:61,
from turbobench.c:31:
/usr/include/sys/_stdint.h:60:20: note: previous declaration of ‘uint64_t’ with type ‘uint64_t’ {aka ‘long unsigned int’}
60 | typedef __uint64_t uint64_t ;
| ^~~~~~~~
make: *** [makefile:726: turbobench.o] Error 1

Consider dropping pithy

As part of my efforts to fuzz compression libraries, I recently tested pithy and found some vulnerabilities. The author has not responded to email about the issue for over a month, nor the pull request filed against the project, so AFAICT it's abandoned.

I'm not sure whether or not it is appropriate for this project to keep supporting pithy or not. It obviously shouldn't be used in production code, but turbobench isn't really intended for production code… I guess the distinction should be whether turbobench is intended to help people choose a compression library for their code (in which case it would be better not to include pithy), or as a tool for people writing compression libraries (in which case it would be better to include it).

bwt preset segfaults the program

Running ./turbobench -eBWT ascii.txt
We can see we can do the first two items in the test:
1576 8.9 42.84 261.86 bzip2 ascii.txt
1347 7.6 40.68 0.03 bzip3 ascii.txt

But after that, we get a segfault:
Segmentation fault (core dumped)

Running Arch and the latest release.

add codecs

It would be interesting to add codecs

libzlf
lzfx
lz88
aPlib
delta encoding
runlength encoding
zstandard

all but the last one are potentially useful in use on low power microcontrollers

Naka compilation error

When compiling from a pristine clone I'm getting the following error:

plugins.o: In function `codcomp':
plugins.cc:(.text+0x1ce1): undefined reference to `NakaCompress'
plugins.o: In function `coddecomp':
plugins.cc:(.text+0x32aa): undefined reference to `NakaDecompress'
collect2: error: ld returned 1 exit status
make: *** [makefile:436: turbobench] Error 1

Compilation of zpaq fails

Dear Developer,
I've tried to compile with gcc 7.1.1 and got this error:

zpaq/libzpaq.o: In function `libzpaq::Array<int>::resize(unsigned long, int) [clone .constprop.129]':
libzpaq.cpp:(.text+0xf65): undefined reference to `libzpaq::error(char const*)'
libzpaq.cpp:(.text+0xfe0): undefined reference to `libzpaq::error(char const*)'
zpaq/libzpaq.o: In function `libzpaq::Array<unsigned short>::resize(unsigned long, int) [clone .constprop.131]':
libzpaq.cpp:(.text+0x1050): undefined reference to `libzpaq::error(char const*)'
libzpaq.cpp:(.text+0x10d0): undefined reference to `libzpaq::error(char const*)'
zpaq/libzpaq.o: In function `libzpaq::Array<unsigned char>::resize(unsigned long, int) [clone .constprop.132]':
libzpaq.cpp:(.text+0x1188): undefined reference to `libzpaq::error(char const*)'
zpaq/libzpaq.o:libzpaq.cpp:(.text+0x11a0): more undefined references to `libzpaq::error(char const*)' follow
collect2: error: ld returned 1 exit status
make: *** [makefile:436: turbobench] Error 1

Feedback - Open Benchmark Directory (user testdatafiles/proposals allowed)

Thanks a lot for the nifty TurboBench, after wandering here and there, finally thought it is a good idea to post my endless quest (in form of console logs and tables) for giving a rich picture of [de]compressors of today.
Hope, you don't mind me posting on a weekly basis inhere.
My wish is we to have on a public site (GitHub is fully okay) as many as possible interesting datasets tested.

The older package used for tests:

The single PDF file unfinished (134 pages) overview of Textual Benchmarks (done on my other laptop with i5-7200u) is here:
https://drive.google.com/file/d/1DUVowDtp__WGiC3HIw1whE2dTKt7kvBD/view?usp=sharing

Wanted to be much more versatile and bigger, but it is what it is.

The newer package used for tests:

The testmachine will be (already acquired) the Toshiba laptop that I used to torture:

My wish was (has to wait) to buy a dedicated machine for heavy compression benchmarks with i3-8100 and 64GB DDR4: http://www.overclock.net/forum/158-laptops-netbooks/1619592-kaby-lake-intel-core-i5-7200u-how-fast.html#post26536756
The key thing inhere is the eventual running of any file (<1.5GB due to free RAM being ~14GB). It would be nice once and for all we to have one versatile roster. My preference towards textual data remains, so the files listed below will be run first.

E:\Textual_Madness_quickoverview>dir/og/on

02/13/2018  02:48 AM    <DIR>          .
02/13/2018  02:48 AM    <DIR>          ..
02/13/2018  03:40 AM    <DIR>          _Check_Integrity_Folder
02/13/2018  03:42 AM       193,367,552 Big_Soviet_Encyclopedia_in_30_volumes_(1239-HTMs).tar
02/13/2018  03:42 AM       681,979,392 book_serie_SPETSNAZ_803_novels_(Russian).tar
02/13/2018  03:42 AM     1,000,000,000 enwik9
02/13/2018  03:42 AM       648,260,096 gcc-6.3.0_(96398_Files_5502_Folders).tar
02/13/2018  03:42 AM     2,037,880,832 INTERNET_SACRED_TEXT_ARCHIVE_DVD-ROM_9_(English_140479_htm_files).tar
02/13/2018  03:42 AM     1,640,759,296 Machine-Learning_amazon_review_full_csv_(35_million_reviews).tar
02/13/2018  03:42 AM     4,680,140,800 Machine-Learning_British-National-Corpus_XML-edition.tar
02/13/2018  03:42 AM       405,610,647 Machine-Learning_Douban_Movie_Short_Comments_(Chinese).csv
02/13/2018  03:42 AM       150,950,913 Machine-Learning_Global_Terrorism_Database_(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
02/13/2018  03:42 AM     1,917,822,288 Machine-Learning_Urban_Dictionary_Definitions_Corpus_(1999_-_May-2016).words.json
02/13/2018  03:42 AM       310,654,639 Machine-Learning_Wikipedia_Article_Titles_(September-20-2017).txt
02/13/2018  03:42 AM       133,901,432 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
02/13/2018  03:42 AM       630,339,584 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
02/13/2018  03:42 AM       203,288,144 Machine-Learning_www.kaggle.com_examine-the-examiner_(headlines_of_3_million_articles).csv
02/13/2018  03:42 AM       385,286,656 Machine-Learning_www.kaggle.com_japaneseenglish-bilingual-corpus_(500000_pairs_of_manually-translated_sentences).tar
02/13/2018  03:42 AM       282,218,054 Machine-Learning_www.kaggle.com_opencorpora-russian_(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
02/13/2018  03:42 AM       560,714,120 Oxford_English_Dictionary_2nd_Edition_Version_4_(En-En).dsl
02/13/2018  03:42 AM       325,071,872 Star_Trek_-_737_Ebooks.tar
02/13/2018  03:42 AM       630,349,312 Stephan_Kaze_http_unbound.biola.edu_103-bibles.tar
02/13/2018  03:42 AM     1,028,290,560 Stephan_Kaze_windows_nt_4_source_code.tar
02/13/2018  03:42 AM     1,382,122,496 TEXTFILES.COM_(58096_files).tar
02/13/2018  03:42 AM     1,499,100,672 the-anarchist-library-2016-01-18-en.tar
02/13/2018  03:42 AM     1,036,155,727 Urban_Dictionary_2015_(Eng-Eng)_utf8.dsl
02/13/2018  03:42 AM       681,378,816 webdevdata.org_8000_home_pages_from_the_top_10000_most_popular_web_sites.tar
02/13/2018  03:42 AM       686,991,360 www.kernel.org_linux-4.8.4.tar
02/13/2018  03:42 AM       975,021,056 www.ncbi.nlm.nih.gov_Dragonfly_(Ladona_fulva)_whole_genome_shotgun.tar

E:\Textual_Madness_quickoverview>type _Check_Integrity_Folder\_Check_Integrity.ORIGINAL
d32999b84d3a0c1395c4b5ed9200a248ce6f3d38  ..\Big_Soviet_Encyclopedia_in_30_volumes_(1239-HTMs).tar
ad49660290b680d759238ab0c5a8d15307080b68  ..\book_serie_SPETSNAZ_803_novels_(Russian).tar
2996e86fb978f93cca8f566cc56998923e7fe581  ..\enwik9
c103fbe221bfb384c2417e27fcb7c6420fd114f1  ..\gcc-6.3.0_(96398_Files_5502_Folders).tar
7c2e32a76716e184d302e5542b96c16e95047002  ..\INTERNET_SACRED_TEXT_ARCHIVE_DVD-ROM_9_(English_140479_htm_files).tar
05a382d4d82a2b81f954f17a3cbe8950e36c3a55  ..\Machine-Learning_amazon_review_full_csv_(35_million_reviews).tar
e199cd3f606db268cdd0be5eef1c4932f37acd13  ..\Machine-Learning_British-National-Corpus_XML-edition.tar
c3a9a9116551646c04317abb1c1b4d612cdeb3e9  ..\Machine-Learning_Douban_Movie_Short_Comments_(Chinese).csv
7db6ec6256ac5346e8f98ca52076d12193ae84d7  ..\Machine-Learning_Global_Terrorism_Database_(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
fd0567e4a5d800f1880c6efa459125d4256646f4  ..\Machine-Learning_Urban_Dictionary_Definitions_Corpus_(1999_-_May-2016).words.json
89d5905943237c81ce0c6ff0c08a001e4f0b6355  ..\Machine-Learning_Wikipedia_Article_Titles_(September-20-2017).txt
56ffb512c8e055a5b3bb959c0210521dca13d178  ..\Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
b5e3ce3ec31d7088709fde1a6465088beeaf4c91  ..\Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
ab83ab51bb94b6ddeac80639fc83cbfe7ed35743  ..\Machine-Learning_www.kaggle.com_examine-the-examiner_(headlines_of_3_million_articles).csv
6014d55046c734a5be1c006c136432203bab0c3a  ..\Machine-Learning_www.kaggle.com_japaneseenglish-bilingual-corpus_(500000_pairs_of_manually-translated_sentences).tar
6a4dfc9b77af3c15ba58c1032f549448e4da2dcb  ..\Machine-Learning_www.kaggle.com_opencorpora-russian_(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
31ae7c9ecdbfbf79221bab5db963268acec3f77a  ..\Oxford_English_Dictionary_2nd_Edition_Version_4_(En-En).dsl
b8e5c4b1932300dee41c1e9655951599d4f7a036  ..\Star_Trek_-_737_Ebooks.tar
199769ffd4c11c8d01fa0bac00b65edf830455bd  ..\Stephan_Kaze_http_unbound.biola.edu_103-bibles.tar
4b42a93f4cb3952dc878f1c3ee600bf9162b4f8b  ..\Stephan_Kaze_windows_nt_4_source_code.tar
f8cd4ad35c75f3f40df1658cd54bdc14eb8d952e  ..\TEXTFILES.COM_(58096_files).tar
76287ba461260fb11bfd6d4f6acc44a1d5857bee  ..\the-anarchist-library-2016-01-18-en.tar
d7827d7458bcd5457aa1499a8479fd05a574a591  ..\Urban_Dictionary_2015_(Eng-Eng)_utf8.dsl
ef845f6b1a57d69228999664e000e2e678fe501a  ..\webdevdata.org_8000_home_pages_from_the_top_10000_most_popular_web_sites.tar
b569d3709706ff240ade45b276f5ed78b931a417  ..\www.kernel.org_linux-4.8.4.tar
25cc7d3b4f18e63f5452127f2b915adc834f1f67  ..\www.ncbi.nlm.nih.gov_Dragonfly_(Ladona_fulva)_whole_genome_shotgun.tar

E:\Textual_Madness_quickoverview>type Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
id;budget;genres;imdb_id;original_language;original_title;overview;popularity;production_companies;production_countries;release_date;revenue;runtime;spoken_languages;status;tagline;title;vote_average;vote_count;production_companies_number;production_countries_number;spoken_languages_number
2;0;Drama|Crime;tt0094675;fi;Ariel;Taisto Kasurinen is a Finnish coal miner whose father has just committed suicide and who is framed for a crime he did not commit. In jail, he starts to dream about leaving the country and starting a new life. He escapes from prison but things don't go as planned...;0.823904;Villealfa Filmproduction Oy;Finland;21/10/1988;0;69;suomi;Released;;Ariel;7.1;40;2;1;2
3;0;Drama|Comedy;tt0092149;fi;Varjoja paratiisissa;An episode in the life of Nikander, a garbage man, involving the death of a co-worker, an affair and much more.;0.47445;Villealfa Filmproduction Oy;Finland;16/10/1986;0;76;English;Released;;Shadows in Paradise;7.0;32;1;1;3
5;4000000;Crime|Comedy;tt0113101;en;Four Rooms;It's Ted the Bellhop's first night on the job...and the hotel's very unusual guests are about to place him in some outrageous predicaments. It seems that this evening's room service is serving up one unbelievable happening after another.;1.698;Miramax Films;United States of America;25/12/1995;4300000;98;English;Released;Twelve outrageous guests. Four scandalous requests. And one lone bellhop, in his first day on the job, who's in for the wildest New year's Eve of his life.;Four Rooms;6.5;485;2;1;1
6;0;Action|Thriller|Crime;tt0107286;en;Judgment Night;"While racing to a boxing match, Frank, Mike, John and Rey get more than they bargained for. A wrong turn lands them directly in the path of Fallon, a vicious, wise-cracking drug lord. After accidentally witnessing Fallon murder a disloyal henchman, the four become his unwilling prey in a savage game of cat &amp; mouse as they are mercilessly stalked through the urban jungle in this taut suspense drama";1.32287;Universal Pictures;Japan;15/10/1993;12136938;110;English;Released;Don't move. Don't whisper. Don't even breathe.;Judgment Night;6.5;69;3;2;1
8;42000;Documentary;tt0825671;en;Life in Loops (A Megacities RMX);"Timo Novotny labels his new project an experimental music documentary film, in a remix of the celebrated film Megacities (1997), a visually refined essay on the hidden faces of several world ""megacities"" by leading Austrian documentarist Michael Glawogger. Novotny complements 30 % of material taken straight from the film (and re-edited) with 70 % as yet unseen footage in which he blends original shots unused by Glawogger with his own sequences (shot by Megacities cameraman Wolfgang Thaler) from Tokyo. Alongside the Japanese metropolis, Life in Loops takes us right into the atmosphere of Mexico City, New York, Moscow and Bombay. This electrifying combination of fascinating film images and an equally compelling soundtrack from Sofa Surfers sets us off on a stunning audiovisual adventure across the continents. The film also makes an original contribution to the discussion on new trends in documentary filmmaking. Written by KARLOVY VARY IFF 2006";0.054716;inLoops;Austria;01/01/2006;0;80;English;Released;A Megacities remix.;Life in Loops (A Megacities RMX);6.4;4;1;1;5
9;0;Drama;tt0425473;de;Sonntag im August;;0.001647;none;Germany;02/09/2004;0;15;Deutsch;Released;;Sunday in August;5.3;2;0;1;1
11;11000000;Adventure|Action|Science Fiction;tt0076759;en;Star Wars;Princess Leia is captured and held hostage by the evil Imperial forces in their effort to take over the galactic Empire. Venturesome Luke Skywalker and dashing captain Han Solo team together with the loveable robot duo R2-D2 and C-3PO to rescue the beautiful princess and restore peace and justice in the Empire.;10.492614;Lucasfilm;United States of America;25/05/1977;775398007;121;English;Released;A long time ago in a galaxy far, far away...;Star Wars;8.0;6168;2;1;1
12;94000000;Animation|Family;tt0266543;en;Finding Nemo;Nemo, an adventurous young clownfish, is unexpectedly taken from his Great Barrier Reef home to a dentist's office aquarium. It's up to his worrisome father Marlin and a friendly but forgetful fish Dory to bring Nemo home -- meeting vegetarian sharks, surfer dude turtles, hypnotic jellyfish, hungry seagulls, and more along the way.;9.915573;Pixar Animation Studios;United States of America;30/05/2003;940335536;100;English;Released;There are 3.7 trillion fish in the ocean, they're looking for one.;Finding Nemo;7.6;5531;1;1;1
13;55000000;Comedy|Drama|Romance;tt0109830;en;Forrest Gump;A man with a low IQ has accomplished great things in his life and been present during significant historic events - in each case, far exceeding what anyone imagined he could do. Yet, despite all the things he has attained, his one true love eludes him. 'Forrest Gump' is the story of a man who rose above his challenges, and who proved that determination, courage, and love are more important than ability.;10.351236;Paramount Pictures;United States of America;06/07/1994;677945399;142;English;Released;The world will never be the same, once you've seen it through the eyes of Forrest Gump.;Forrest Gump;8.2;7204;1;1;1
14;15000000;Drama;tt0169547;en;American Beauty;Lester Burnham, a depressed suburban father in a mid-life crisis, decides to turn his hectic life around after developing an infatuation with his daughter's attractive friend.;8.191009;DreamWorks SKG;United States of America;15/09/1999;356296601;122;English;Released;Look closer.;American Beauty;7.9;2994;2;1;1
15;839727;Mystery|Drama;tt0033467;en;Citizen Kane;Newspaper magnate, Charles Foster Kane is taken from his mother as a boy and made the ward of a rich industrialist. As a result, every well-meaning, tyrannical or self-destructive move he makes for the rest of his life appears in some way to be a reaction to that deeply wounding event.;3.82689;RKO Radio Pictures;United States of America;30/04/1941;23217674;119;English;Released;It's Terrific!;Citizen Kane;7.9;1110;2;1;1
16;12800000;Drama|Crime|Music;tt0168629;en;Dancer in the Dark;Selma, a Czech immigrant on the verge of blindness, struggles to make ends meet for herself and her son, who has inherited the same genetic disorder and will suffer the same fate without an expensive operation. When life gets too difficult, Selma learns to cope through her love of musicals, escaping life's troubles - even if just for a moment - by dreaming up little numbers to the rhythmic beats of her surroundings.;2.106217;Fine Line Features;Argentina;17/05/2000;40031879;140;English;Released;You don't need eyes to see.;Dancer in the Dark;7.6;348;26;12;1
17;0;Horror|Thriller|Mystery;tt0411267;en;The Dark;Adèle and her daughter Sarah are traveling on the Welsh coastline to see her husband James when Sarah disappears. A different but similar looking girl appears who says she died in a past time. Adèle tries to discover what happened to her daughter as she is tormented by Celtic mythology from the past.;1.253999;Constantin Film;Germany;26/01/2006;0;87;English;Released;One of the living for one of the dead.;The Dark;5.6;69;4;2;2
18;90000000;Adventure|Fantasy|Action|Thriller|Science Fiction;tt0119116;en;The Fifth Element;In 2257, a taxi driver is unintentionally given the task of saving a young girl who is part of the key that will ensure the survival of humanity.;9.233786;Columbia Pictures;France;07/05/1997;263920180;126;English;Released;There is no future without it.;The Fifth Element;7.2;3629;2;1;3
19;92620000;Drama|Science Fiction;tt0017136;de;Metropolis;In a futuristic city sharply divided between the working class and the city planners, the son of the city's mastermind falls in love with a working class prophet who predicts the coming of a savior to mediate their differences.;3.669986;Paramount Pictures;Germany;10/01/1927;650422;153;No Language;Released;There can be no understanding between the hands and the brain unless the heart acts as mediator.;Metropolis;8.0;614;2;1;1
20;0;Drama|Romance;tt0314412;en;My Life Without Me;A Pedro Almodovar production in which a fatally ill mother with only two months to live creates a list of things she wants to do before she dies with out telling her family of her illness.;0.911462;El Deseo;Canada;07/03/2003;9726954;106;English;Released;;My Life Without Me;7.2;75;2;2;1
21;0;Documentary;tt0060371;en;The Endless Summer;The Endless Summer, by Bruce Brown, is one of the first and most influential surf movies of all times. The film documents American surfers Mike Hynson and Robert August as they travel the world during California’s winter (which back in 1965 was off-season for surfing) in search of the perfect wave and an endless summer.;0.144179;Bruce Brown Films;United States of America;15/06/1966;0;95;English;Released;;The Endless Summer;7.8;20;1;1;1
22;140000000;Adventure|Fantasy|Action;tt0325980;en;Pirates of the Caribbean: The Curse of the Black Pearl;Jack Sparrow, a freewheeling 17th-century pirate who roams the Caribbean Sea, butts heads with a rival pirate bent on pillaging the village of Port Royal. When the governor's daughter is kidnapped, Sparrow decides to help the girl's love save her. But their seafaring mission is hardly simple.;28.769026;Walt Disney Pictures;United States of America;09/07/2003;655011224;143;English;Released;Prepare to be blown out of the water.;Pirates of the Caribbean: The Curse of the Black Pearl;7.4;6368;2;1;1
24;30000000;Action|Crime;tt0266697;en;Kill Bill: Vol. 1;An assassin is shot at the altar by her ruthless employer, Bill and other members of their assassination circle – but 'The Bride' lives to plot her vengeance. Setting out for some payback, she makes a death list and hunts down those who wronged her, saving Bill for last.;7.891837;Miramax Films;United States of America;10/10/2003;180949000;111;English;Released;Go for the kill.;Kill Bill: Vol. 1;7.7;4486;3;1;3
25;72000000;Drama|War;tt0418763;en;Jarhead;Jarhead is a film about a US Marine Anthony Swofford’s experience in the Gulf War. After putting up with an arduous boot camp, Swafford and his unit are sent to the Persian Gulf where they are earger to fight but are forced to stay back from the action. Meanwhile Swofford gets news of his girlfriend is cheating on him. Desperately he wants to kill someone and finally put his training to use.;2.41718;Universal Pictures;Germany;04/11/2005;96889998;125;English;Released;Welcome to the suck.;Jarhead;6.5;722;4;2;4
26;1400000;Drama;tt0352994;en;LaLehet Al HaMayim;Eyal, an Israeli Mossad agent, is given the mission to track down and kill the very old Alfred Himmelman, an ex-Nazi officer, who might still be alive. Pretending to be a tourist guide, he befriends his grandson Axel, in Israel to visit his sister Pia. The two men set out on a tour of the country during which, Axel challenges Eyal's values.;0.455665;Lama Films;Israel;05/02/2004;0;103;العربية;Released;He was trained to hate until he met the enemy.;Walk on Water;6.4;18;2;2;6
27;1000000;Drama|Music|Romance;tt0411705;en;9 Songs;Matt, a young glaciologist, soars across the vast, silent, icebound immensities of the South Pole as he recalls his love affair with Lisa. They meet at a mobbed rock concert in a vast music hall - London's Brixton Academy. They are in bed at night's end. Together, over a period of several months, they pursue a mutual sexual passion whose inevitable stages unfold in counterpoint to nine live-concert songs.;2.939728;Revolution Films;United Kingdom;16/07/2004;1574623;66;English;Released;2 lovers, one summer, and the 9 songs that defined them.;9 Songs;5.1;95;1;1;1
28;31500000;Drama|War;tt0078788;en;Apocalypse Now;"At the height of the Vietnam war, Captain Benjamin Willard is sent on a dangerous mission that, officially, ""does not exist, nor will it ever exist."" His goal is to locate - and eliminate - a mysterious Green Beret Colonel named Walter Kurtz, who has been leading his personal army on illegal guerrilla missions into enemy territory.";7.620077;United Artists;United States of America;15/08/1979;89460381;153;;Released;This is the end...;Apocalypse Now;8.0;1869;2;1;4
30;0;Animation|Science Fiction;tt1530535;ja;彼女の想いで;Koji Morimato’s animated science fiction short story about how the boarder between reality and illusion on a space station become blurry.;0.811221;Studio 4°C;Japan;23/12/1995;0;44;日本語;Released;;Magnetic Rose;7.7;10;1;1;1
31;0;Action|Animation|Comedy;tt6266826;ja;最臭兵器;Tensai Okamura’s animated action packed short story with lots of humorous elements in which a person transforms into a weapon of mass destruction without themselves being aware.;1.281042;Studio 4°C;Japan;23/12/1995;0;40;日本語;Released;;Stink Bomb;5.3;3;1;1;1
32;0;Animation|History;tt6264824;ja;大砲の街;Otomo Katsuhiro’s short anime story;0.838219;Studio 4°C;Japan;23/12/1995;0;21;日本語;Released;;Cannon Fodder;5.3;3;1;1;1
...

H:\smashshop_2018-Feb-12_Judaica>_BENCH_a_file.BAT Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv

H:\smashshop_2018-Feb-12_Judaica>lzbench173 -c4 -i1,15 -o3 -ebrotli24,1,5,11/tornado,16/blosclz,9/brieflz/crush,2/csc,5/density,3/fastlz,2/gipfeli/zstd24,12,22/zstd24LDM,12,22/lzo1b,999/lzham,4/lzham24,4/libdeflate,1,12/lz4fast,1,99/lz4/lz4hc,10,12/lizard,19,29,39,49/lzf,1/lzfse/lzg,9/lzham,1/lzjb/lzlib,9/lzma,9/lzrw,5/lzsse2,17/lzsse4,17/lzsse8,17/lzvn/pithy,9/quicklz,3/snappy/slz_zlib,3/ucl_nrv2b,9/ucl_nrv2d,9/ucl_nrv2e,9/xpack,9/xz,9/yalz77,12/yappy,99/zlib,1,5,9/zling,4/shrinker/wflz/lzmat Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzbench 1.7.3 (64-bit Windows)   Assembled by P.Skibinski
Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
memcpy                   6365 MB/s  6365 MB/s   133901432    133901432 100.00 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
...
The results sorted by column number 4:
Compressor name         Compress. Decompress.  Orig. size  Compr. size  Ratio Filename
csc 2016-10-13 -5        1.65 MB/s    45 MB/s   133901432     34749865  25.95 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzma 16.04 -9            0.78 MB/s    60 MB/s   133901432     34816931  26.00 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
xz 5.2.3 -9              0.82 MB/s    55 MB/s   133901432     34817747  26.00 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzham 1.0 -d26 -4        0.58 MB/s   122 MB/s   133901432     35117612  26.23 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzlib 1.8 -9             0.74 MB/s    40 MB/s   133901432     35346969  26.40 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
tornado 0.6a -16         0.87 MB/s   119 MB/s   133901432     36088796  26.95 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
brotli24 2017-12-12 -11  0.33 MB/s   172 MB/s   133901432     36237727  27.06 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzham24 1.0 -4           0.71 MB/s   122 MB/s   133901432     36536959  27.29 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
zstd24 1.3.3 -22         0.89 MB/s   335 MB/s   133901432     36966062  27.61 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
zstd24LDM 1.3.3 -22      0.91 MB/s   337 MB/s   133901432     37129453  27.73 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
zling 2016-01-10 -4        18 MB/s   103 MB/s   133901432     41708378  31.15 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzham 1.0 -d26 -1        1.43 MB/s   125 MB/s   133901432     42404312  31.67 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
zstd24 1.3.3 -12         5.02 MB/s   344 MB/s   133901432     44018384  32.87 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
zstd24LDM 1.3.3 -12      4.88 MB/s   344 MB/s   133901432     44019143  32.87 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
crush 1.0 -2             0.20 MB/s   219 MB/s   133901432     45171471  33.73 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
brotli24 2017-12-12 -5     13 MB/s   246 MB/s   133901432     45354869  33.87 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
xpack 2016-06-02 -9      8.36 MB/s   448 MB/s   133901432     49434209  36.92 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lizard 1.0 -49           0.83 MB/s   664 MB/s   133901432     52052053  38.87 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
libdeflate 0.7 -12       5.71 MB/s   380 MB/s   133901432     52203890  38.99 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lizard 1.0 -29           0.87 MB/s   776 MB/s   133901432     52980228  39.57 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzfse 2017-03-08           35 MB/s   367 MB/s   133901432     53217444  39.74 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
zlib 1.2.11 -9             13 MB/s   192 MB/s   133901432     54103240  40.41 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzsse2 2016-05-14 -17    5.90 MB/s  1808 MB/s   133901432     54676447  40.83 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
zlib 1.2.11 -5             21 MB/s   190 MB/s   133901432     54783631  40.91 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
ucl_nrv2e 1.03 -9        0.92 MB/s   188 MB/s   133901432     55218581  41.24 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzsse4 2016-05-14 -17    6.82 MB/s  1876 MB/s   133901432     55450256  41.41 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzsse8 2016-05-14 -17    6.41 MB/s  1846 MB/s   133901432     55598759  41.52 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
ucl_nrv2d 1.03 -9        0.94 MB/s   190 MB/s   133901432     55700314  41.60 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
brotli24 2017-12-12 -1     91 MB/s   200 MB/s   133901432     56499193  42.19 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
ucl_nrv2b 1.03 -9        0.94 MB/s   188 MB/s   133901432     56995941  42.57 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
libdeflate 0.7 -1          88 MB/s   349 MB/s   133901432     58406821  43.62 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzo1b 2.09 -999            10 MB/s   393 MB/s   133901432     60063312  44.86 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lz4hc 1.8.0 -12          8.74 MB/s  1610 MB/s   133901432     60256681  45.00 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lizard 1.0 -19           5.38 MB/s  1781 MB/s   133901432     60301678  45.03 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lz4hc 1.8.0 -10            15 MB/s  1627 MB/s   133901432     60525615  45.20 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lizard 1.0 -39           5.43 MB/s  1626 MB/s   133901432     61587571  45.99 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzg 1.0.8 -9             0.62 MB/s   403 MB/s   133901432     61803020  46.16 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
zlib 1.2.11 -1             51 MB/s   190 MB/s   133901432     62234317  46.48 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzmat 1.01                 22 MB/s   193 MB/s   133901432     62428884  46.62 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
yalz77 2015-09-19 -12      11 MB/s   174 MB/s   133901432     62982598  47.04 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
brieflz 1.1.0              70 MB/s   108 MB/s   133901432     63919589  47.74 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
density 0.12.5 beta -3    104 MB/s   196 MB/s   133901432     63987540  47.79 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
quicklz 1.5.0 -3           31 MB/s   532 MB/s   133901432     66622450  49.75 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzvn 2017-03-08            30 MB/s   546 MB/s   133901432     66786297  49.88 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
gipfeli 2016-07-13        102 MB/s   208 MB/s   133901432     68309192  51.01 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzrw 15-Jul-1991 -5        83 MB/s   279 MB/s   133901432     70927828  52.97 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
pithy 2011-12-24 -9       162 MB/s   815 MB/s   133901432     73138033  54.62 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
shrinker 0.1              193 MB/s   577 MB/s   133901432     75857895  56.65 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzf 3.6 -1                167 MB/s   346 MB/s   133901432     78638419  58.73 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
fastlz 0.1 -2             164 MB/s   302 MB/s   133901432     80024724  59.76 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
slz_zlib 1.0.0 -3         126 MB/s   167 MB/s   133901432     80345979  60.00 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
yappy 2014-03-22 -99       61 MB/s  1431 MB/s   133901432     81157014  60.61 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
blosclz 2015-11-10 -9     140 MB/s   470 MB/s   133901432     81334487  60.74 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
snappy 1.1.4              192 MB/s   652 MB/s   133901432     84587804  63.17 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lz4fast 1.8.0 -1          273 MB/s  1774 MB/s   133901432     86683715  64.74 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lz4 1.8.0                 273 MB/s  1773 MB/s   133901432     86683715  64.74 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
wflz 2015-09-16           123 MB/s   508 MB/s   133901432     90616277  67.67 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzjb 2010                 155 MB/s   300 MB/s   133901432    101511464  75.81 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lz4fast 1.8.0 -99        3442 MB/s  5355 MB/s   133901432    133073240  99.38 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv

H:\smashshop_2018-Feb-12_Judaica>"turbobenchs_Official_v17.04_-_build_07_Apr_2017.exe" Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv -elibdeflate,12/oodle,19,49,89,112,114,116,118,129/lzsse2,17/lzturbo,19,12,29,22,39,32,49,59/zstd,12,22/lizard,19,29,39,49/brotli,11/lzma,9/bzip2/xpack,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -g -I1 -J31 -k1 -B2G
      C Size  ratio%     C MB/s     D MB/s   Name            File
    27751984    20.7       6.28      13.79   lzturbo 59                       Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    28626045    21.4       0.29       0.29   zpaq 5                           Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    29365736    21.9      12.30       5.76   bsc 6                            Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    33800605    25.2       0.60      61.26   lzturbo 49                       Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    34816935    26.0       0.79      60.60   lzma 9                           Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    34897942    26.1       0.59     122.19   lzham 4                          Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    35231273    26.3       0.69     314.19   lzturbo 39                       Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    35447440    26.5       0.67     324.30   zstd 22                          Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    36219098    27.0       0.34     342.69   oodle 129                        Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    36219098    27.0       0.39     342.40   oodle 89                         Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    37551395    28.0       0.20     206.53   oodle 19                         Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    38197262    28.5       0.35     212.81   brotli 11                        Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    38824278    29.0      15.90      10.57   bsc 3                            Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    39475083    29.5       9.44      18.91   bzip2                            Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    39775117    29.7       0.73     516.82   lzturbo 29                       Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    41317955    30.9      16.96     351.28   lzturbo 32                       Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    42616730    31.8       0.40     224.70   xpack 9                          Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    44018388    32.9       4.99     381.61   zstd 12                          Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    44786082    33.4       2.71      51.32   zpaq 2                           Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    52052057    38.9       0.85     680.56   lizard 49                        Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    52203900    39.0       5.74     360.87   libdeflate 12                    Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    52980232    39.6       0.87     795.93   lizard 29                        Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    53378419    39.9      16.99     603.31   lzturbo 22                       Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    54676451    40.8       5.62    1800.68   lzsse2 17                        Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    56151529    41.9       0.34    1621.35   oodle 116                        Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    56763285    42.4       0.45    1607.96   oodle 118                        Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    60271701    45.0       0.84    2400.20   lzturbo 19                       Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    60290973    45.0       2.73    1716.35   oodle 49                         Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    60301682    45.0       5.30    1850.90   lizard 19                        Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    61519812    45.9       8.27    1765.82   oodle 114                        Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    61587575    46.0       5.38    1738.74   lizard 39                        Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    63905083    47.7      29.59    2402.07   lzturbo 12                       Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    63987386    47.8     114.56     205.91   density 3                        Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    67517585    50.4      29.13    2244.17   oodle 112                        Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
    82491467    61.6    1043.42    1581.51   chameleon 2                      Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
   133625257    99.8     104.70    1420.42   trle                             Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb

H:\smashshop_2018-Feb-12_Judaica>timer64 PPMd_varI_rev2_Intel15_32bit.exe e -o6 -m256 -fMachine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.O6.PPMd_varI Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
Fast PPMII compressor for textual data, variant I, Apr  3 2016
Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv:133901432

Kernel  Time =     0.156 =    0%
User    Time =    28.797 =   97%
Process Time =    28.953 =   97%    Virtual  Memory =    258 MB
Global  Time =    29.555 =  100%    Physical Memory =    203 MB

H:\smashshop_2018-Feb-12_Judaica>timer64 PPMd_varI_rev2_Intel15_32bit.exe e -o16 -m256 -fMachine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.O16.PPMd_varI Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
Fast PPMII compressor for textual data, variant I, Apr  3 2016
Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv:133901432

Kernel  Time =     0.234 =    0%
User    Time =    48.719 =   98%
Process Time =    48.953 =   99%    Virtual  Memory =    258 MB
Global  Time =    49.416 =  100%    Physical Memory =    259 MB

H:\smashshop_2018-Feb-12_Judaica>timer64 "7za_v16.04_x64.exe" a -tgzip -mx9 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MX9.zip Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv

7-Zip (a) [64] 16.04 : Copyright (c) 1999-2016 Igor Pavlov : 2016-10-04

Scanning the drive:
1 file, 133901432 bytes (128 MiB)

Creating archive: Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MX9.zip

Items to compress: 1

Files read from disk: 1
Archive size: 52319186 bytes (50 MiB)
Everything is Ok

Kernel  Time =     0.093 =    0%
User    Time =   143.130 =   99%
Process Time =   143.224 =   99%    Virtual  Memory =      6 MB
Global  Time =   143.710 =  100%    Physical Memory =      7 MB

H:\smashshop_2018-Feb-12_Judaica>timer64 "7za_v16.04_x64.exe" a -t7z -mx9 -md=29 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MX9Dict512.7z Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv

7-Zip (a) [64] 16.04 : Copyright (c) 1999-2016 Igor Pavlov : 2016-10-04

Scanning the drive:
1 file, 133901432 bytes (128 MiB)

Creating archive: Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MX9Dict512.7z

Items to compress: 1

Files read from disk: 1
Archive size: 34610398 bytes (34 MiB)
Everything is Ok

Kernel  Time =     1.372 =    1%
User    Time =   180.368 =  136%
Process Time =   181.741 =  137%    Virtual  Memory =   1356 MB
Global  Time =   132.258 =  100%    Physical Memory =   1293 MB

H:\smashshop_2018-Feb-12_Judaica>timer64 "xz_v5.2.3_x64.exe" -z -k -f -9 -e -v -v --lzma2=dict=512MiB --threads=1 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
xz_v5.2.3_x64: Filter chain: --lzma2=dict=512MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
xz_v5.2.3_x64: 5,378 MiB of memory is required. The limiter is disabled.
xz_v5.2.3_x64: Decompression will need 513 MiB of memory.
Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv (1/1)
  100 %        33.0 MiB / 127.7 MiB = 0.258   750 KiB/s       2:54

Kernel  Time =     0.811 =    0%
User    Time =   171.975 =   98%
Process Time =   172.786 =   99%    Virtual  Memory =   5389 MB
Global  Time =   174.423 =  100%    Physical Memory =   1525 MB

H:\smashshop_2018-Feb-12_Judaica>timer64 "bsc_v3.1.0_x64.exe" e Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.ST6Block512.bsc -b512 -m6 -cp -Tt
This is bsc, Block Sorting Compressor. Version 3.1.0. 8 July 2012.
Copyright (c) 2009-2012 Ilya Grebnov <[email protected]>.

Machine-Learning_www.kaggle.com_350000-movies-from-them compressed 133901432 into 28865952 in 11.420 seconds.

Kernel  Time =     0.374 =    3%
User    Time =    10.623 =   87%
Process Time =    10.998 =   90%    Virtual  Memory =    715 MB
Global  Time =    12.197 =  100%    Physical Memory =    709 MB

H:\smashshop_2018-Feb-12_Judaica>timer64 "zpaq_v7.05_x64.exe" add Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.method29.zpaq Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv -method 29 -threads 1
zpaq v7.05 journaling archiver, compiled Apr 17 2015
Adding 133.901432 MB in 1 files -method 29 -threads 1 at 2017-12-02 08:53:06.
100.00% 0:00:00 + Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv 133901432
100.00% 0:00:00 [1..1898] 133909032 -method 29,121,1
1 +added, 0 -removed.

0.000000 + (133.901432 -> 133.901432 -> 41.293302) = 41.293302 MB
62.728 seconds (all OK)

Kernel  Time =     0.234 =    0%
User    Time =    60.559 =   96%
Process Time =    60.793 =   96%    Virtual  Memory =   1190 MB
Global  Time =    62.769 =  100%    Physical Memory =    774 MB

H:\smashshop_2018-Feb-12_Judaica>timer64 "zpaq_v7.05_x64.exe" add Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.method59.zpaq Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv -method 59 -threads 1
zpaq v7.05 journaling archiver, compiled Apr 17 2015
Adding 133.901432 MB in 1 files -method 59 -threads 1 at 2017-12-02 08:54:09.
100.00% 0:00:00 + Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv 133901432
100.00% 0:00:00 [1..1898] 133909032 -method 59,121,1
1 +added, 0 -removed.

0.000000 + (133.901432 -> 133.901432 -> 26.153739) = 26.153739 MB
547.049 seconds (all OK)

Kernel  Time =     0.483 =    0%
User    Time =   544.100 =   99%
Process Time =   544.583 =   99%    Virtual  Memory =   2064 MB
Global  Time =   547.053 =  100%    Physical Memory =   1656 MB

H:\smashshop_2018-Feb-12_Judaica>timer64 lz5 -49 -B7 --no-frame-crc Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
using blocks of size 262144 KB
Compressed filename will be : Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.lz5
Compressed 133901432 bytes into 45174349 bytes ==> 33.74%

Kernel  Time =     0.202 =    0%
User    Time =   161.554 =   98%
Process Time =   161.757 =   98%    Virtual  Memory =    676 MB
Global  Time =   163.938 =  100%    Physical Memory =    463 MB

H:\smashshop_2018-Feb-12_Judaica>timer64 "nanozip-0.09a.win64.exe" a -t1 -cc Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.cc.nz Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
NanoZip 0.09 alpha/Win64  (C) 2008-2011 Sami Runsas  www.nanozip.net
  Intel(R) Core(TM) i5-2430M CPU @ 2.40GHz|35487 MHz|#2+HT|14832/16332 MB
Archive: Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.cc.nz
Threads: 1, memory: 512 MB
Compressor #0: nz_cm [524 MB]
Compressed 133 901 432 into 26 762 899 in 3m 16.03s, 667 KB/s
IO-in: 0.08s, 1502 MB/s. IO-out: 0.01s, 1702 MB/s

Kernel  Time =     0.390 =    0%
User    Time =   200.008 =   99%
Process Time =   200.398 =   99%    Virtual  Memory =    539 MB
Global  Time =   200.953 =  100%    Physical Memory =    474 MB

H:\smashshop_2018-Feb-12_Judaica>timer64 "nanozip-0.09a.win64.exe" a -t1 -co Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.co.nz Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
NanoZip 0.09 alpha/Win64  (C) 2008-2011 Sami Runsas  www.nanozip.net
  Intel(R) Core(TM) i5-2430M CPU @ 2.40GHz|49756 MHz|#2+HT|14843/16332 MB
Archive: Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.co.nz
Threads: 1, memory: 512 MB
Compressor #0: nz_optimum1 [540 MB]
Compressed 133 901 432 into 27 087 613 in 16.23s, 8055 KB/s
IO-in: 0.10s, 1228 MB/s. IO-out: 0.13s, 187 MB/s

Kernel  Time =     0.858 =    4%
User    Time =    15.927 =   89%
Process Time =    16.785 =   94%    Virtual  Memory =    559 MB
Global  Time =    17.731 =  100%    Physical Memory =    407 MB

CABARC, Microsoft (R) Cabinet Tool - Version 5.1.2600.0, Copyright (c) Microsoft Corporation.

H:\smashshop_2018-Feb-12_Judaica>timer64 cabarc.exe -m LZX:21 N Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.LZX21.cab Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv

Microsoft (R) Cabinet Tool - Version 5.1.2600.0
Copyright (c) Microsoft Corporation. All rights reserved..

Creating new cabinet 'Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.LZX21.cab' with compression 'LZX:21':
  -- adding Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv

Completed successfully

Kernel  Time =     0.436 =    0%
User    Time =   135.206 =   98%
Process Time =   135.642 =   98%    Virtual  Memory =     20 MB
Global  Time =   137.652 =  100%    Physical Memory =     21 MB

H:\smashshop_2018-Feb-12_Judaica>timer64 cabarc.exe -m MSZIP N Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MSZIP.cab Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv

Microsoft (R) Cabinet Tool - Version 5.1.2600.0
Copyright (c) Microsoft Corporation. All rights reserved..

Creating new cabinet 'Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MSZIP.cab' with compression 'MSZIP':
  -- adding Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv

Completed successfully

Kernel  Time =     0.280 =    1%
User    Time =    15.007 =   86%
Process Time =    15.288 =   88%    Virtual  Memory =      1 MB
Global  Time =    17.293 =  100%    Physical Memory =      3 MB

Compress, version: (N)compress 4.2.4.4, compiled: Fri, Aug 23, 2013 11:56:09. Authors: Peter Jannesen, Dave Mack, Spencer W. Thomas, Jim McKie, Steve Davies, Ken Turkowski, James A. Woods, Joe Orost.

H:\smashshop_2018-Feb-12_Judaica>timer64 compress.exe -c Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv  1>Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.Z

H:\smashshop_2018-Feb-12_Judaica>timer64 zstd-v1.3.3-win64.exe --ultra -22 --zstd=wlog=29,clog=30,hlog=30,slog=26 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv : 26.40%   (133901432 => 35354048 bytes, Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.zst)

Kernel  Time =     1.965 =    0%
User    Time =  1353.917 =   99%
Process Time =  1355.882 =   99%    Virtual  Memory =   8339 MB
Global  Time =  1359.043 =  100%    Physical Memory =   8340 MB

H:\smashshop_2018-Feb-12_Judaica>timer64 rz_1.01.exe a -d 512M Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.512M.rz Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv

 *** RAZOR Archiver 1.01 (2017-09-14) - DEMO/TEST version ***
 *** (c) Christian Martelock ([email protected]) ***

 Scanning h:\smashshop_2018-feb-12_judaica\machine-learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
 Found 0 dirs, 1 files, 133901432 bytes.

 Creating archive Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.512M.rz
 Window : 130764K (512M..128G)
 Header : 98
 Size   : 29922927

 Archive ok. Added 0 dirs, 1 files, 133901432 bytes.
 CPU time = 639.230s / wall time = 477.908s

Kernel  Time =     2.620 =    0%
User    Time =   639.229 =  133%
Process Time =   641.850 =  134%    Virtual  Memory =   1600 MB
Global  Time =   478.017 =  100%    Physical Memory =   1494 MB

12/02/2017  01:03 AM        26,153,739 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.method59.zpaq
12/02/2017  01:09 AM        26,762,899 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.cc.nz
12/02/2017  01:09 AM        27,087,613 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.co.nz
12/02/2017  12:53 AM        28,865,952 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.ST6Block512.bsc
12/02/2017  12:44 AM        29,087,569 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.O6.PPMd_varI
12/02/2017  01:42 AM        29,922,927 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.512M.rz
12/02/2017  12:45 AM        30,120,412 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.O16.PPMd_varI
12/02/2017  12:49 AM        34,610,398 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MX9Dict512.7z
11/13/2017  12:08 AM        34,610,800 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.L9Dict512.xz
11/13/2017  12:08 AM        35,354,048 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.zst
12/02/2017  01:11 AM        40,245,342 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.LZX21.cab
12/02/2017  12:54 AM        41,293,302 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.method29.zpaq
11/13/2017  12:08 AM        45,174,349 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.M49Block256.lz5
12/02/2017  12:47 AM        52,319,186 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MX9.zip
12/02/2017  01:12 AM        54,300,130 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MSZIP.cab
12/02/2017  01:12 AM        58,780,239 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.Z
11/13/2017  12:08 AM       133,901,432 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
...

Above dump is only a candidate, I am open for suggestions as to what additional compressors/options to be added.
The package used is this:
smashshop_2018-Feb-13_Judaica.zip 226 MB (237,723,608 bytes)

Is Oodle benchmarking supported?

Oodle is on the plugin list, but when I try to use it error occurs:

$ ./turbobench.exe ./data/big_building.ppm -eoodle,89
codec 'oodle' not found

I have dlls in turbobench.exe directory:

oo2core_4_win64.dll
oo2core_6_win64.dll
oo2core_7_win64.dll
oo2core_8_win64.dll

Windows 10 x64, Turbobench latest release.
Thanks.

Help artifacts compress test

I would like to know if you could use your tool to evaluate genomic file compression metrics such as compression rate, speed, and especially data loss?

Could I adapt your script to parse specific genomics compression tools?

Tnx

Feedback of Dostoyevski

Thanks Hamid for sharing your binaries.
A quick run on i7-3630QM @3.4GHz 16GB DDR3 @1600MHz:

F:\S>turbobench_build_19_Feb_2020.exe "Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar" -eOPTIMAL/lzo1b,999/bsc,0:e2/lzfse/zlib,9
      C Size  ratio%     C MB/s     D MB/s   Name            File
     5315936    13.2       6.11      10.68   bsc 0:e2        Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar

     5446644               8.06     ?        bwtturbo -59 -t0 -b1024 "%1" "%1.zv"

     6942920    17.2       1.28     120.86   lzma 9          Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
     6996677    17.4       0.92     278.81   lzham 4         Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
     7022275    17.4       0.36     312.06   brotli 11       Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
     7038628    17.5       1.40     703.90   zstd 22         Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
     8061821    20.0       1.40     815.14   lizard 49       Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar

     8851573               1.38     ?        BriefLZ_130_Intel_v19_64bit.exe --optimal -b400m "%1" "%1.blz" 
     9177782               0.05     930      Satanichi_GCC730_64bit.exe "%1" "%1.Nakamichi" 24 4000 i

    10060030    25.0       0.25     365.81   zopfli          Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
    10771354    26.7       4.74     325.28   zlib 9          Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
    11728605    29.1       5.02     651.98   lzo1b 999       Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
    12073794    30.0       9.23    2629.03   lz4 9           Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
    12239442    30.4      38.69     435.75   lzfse           Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
    40303104   100.0    9216.35    9159.80   memcpy          Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar

40303104/774=52,071 B/s
40303104/29=1,389,762 B/s @jibsen
40303104/5=8,060,620 B/s

F:\S>timer64.exe Satanichi_GCC730_64bit.exe "Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar" "Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.Nakamichi" 24 4000 i
...
Leprechaun: Total Searches-n-Inserts Per Second: 41,639,014 SNIPS = 26,898,803,684 keys in 646 seconds
Leprechaun: RAM needed to house B-trees (relative to the file being ripped): 51N = 1,994MB

Compressing 40,303,104 bytes ...
-; Each rotation means 64KB are encoded; Speed: 0,338,681 B/s; Done 100%; Compression Ratio: 4.39:1; Matches(16/24/48): 752,724/380,713/20,519; 128[+] long matches: 0; ETA: 0.00 hours
NumberOfFullLiterals (lower-the-better): 1294
Tsuyo_HEURISTIC_APPLIED_thrice_back-to-back: 0
NumberOf(Tiny)Matches[Micro]Window (4)[16B]: 9672
NumberOfMatches[Bheema]Window [128GB window]: 11405
RAM-to-RAM performance: 338681 B/s.
Compressed to 9,177,782 bytes.
Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0xcd46,fbc3
Target-file-Hash(FNV1A_YoshimitsuTRIAD) = 0xb7ac,545f
Decompressing 9,177,782 (being the compressed stream) bytes ...
RAM-to-RAM performance: 930 MB/s.
Verification (input and output sizes match) OK.
Verification (input and output blocks match) OK.

Kernel  Time =     1.390 =    0%
User    Time =   772.437 =   99%
Process Time =   773.828 =   99%    Virtual  Memory =   5952 MB
Global  Time =   774.604 =  100%    Physical Memory =   3935 MB

F:\S>timer64.exe BriefLZ_130_Intel_v19_64bit.exe --optimal -b400m "Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar" "Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.blz"

Kernel  Time =     0.343 =    1%
User    Time =    29.000 =   98%
Process Time =    29.343 =   99%    Virtual  Memory =   8869 MB
Global  Time =    29.352 =  100%    Physical Memory =   1180 MB

F:\S>TIMER64 bwtturbo -59 -t0 -b1024 "Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar" "Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.zv"

Kernel  Time =     0.359 =    7%
User    Time =     4.484 =   92%
Process Time =     4.843 =   99%    Virtual  Memory =   2564 MB
Global  Time =     4.850 =  100%    Physical Memory =   1219 MB

09/15/2018  04:41           231,424 BriefLZ_130_Intel_v19_64bit.exe
02/07/2020  01:38           555,806 bwtturbo.exe
02/19/2020  13:42           193,517 Satanichi_GCC730_64bit.exe
02/15/2020  22:24             6,144 timer64.exe
01/09/2020  12:51             1,079 turbobench.ini
02/19/2020  18:39         4,536,493 turbobench_build_19_Feb_2020.exe
06/30/2018  20:35         5,446,644 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.zv
06/30/2018  20:32         8,851,573 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.blz
                          9,177,782 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.Nakamichi
07/28/2018  07:22        40,303,104 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar

Printing to other formats with `-p' doesn't seem to work.

I tried to print to cvs or markdown (with `-p7') but it didn't work. Actually any format makes it to exit without any action.
I tried to fix it but didn't get far. All I got to is that `printfile()' exits at if(!k) return; as `plugread()' returns 0.

Compilation Problem

After compiling TurboBench, the executable is not generated. How do I fix this?

Flag to print list of available compressors

There is in the readme a list of compressors https://github.com/powturbo/TurboBench#compressor-lz77rolzbwtzpaq i assume these can be tested with turbobench and also that they can be specified in turbobench.ini.

I also assume that a compressor like Intel(R) Intelligent Storage Acceleration Library must be specified with a different name in the turbobench.ini

Could there be added a option to list all possible compressors? At first i thought -l1 or -l2 would print this list. But they print less entries than the 57 which can be found in the readme.

Segmentation fault in turbobench.c:249 stackpeak

version 2023-03 fails on linux, with compiler gcc (GCC) 12.2.0

$ gdb --args turbobench test-files.tar 
(gdb) r
   15.66  3_01
Program received signal SIGSEGV, Segmentation fault.
stackpeak (_sp=_sp@entry=0x0) at turbobench.c:249
249       for(sp = _sp - STACK_SIZE; *sp == STACK_MAGIC; sp++);
(gdb) bt
#0  stackpeak (_sp=_sp@entry=0x0) at turbobench.c:249
#1  0x0000000000576cff in plugfile (plug=plug@entry=0x6e8940 <plug>, finame=finame@entry=0x7fffffff9989 "test-files.tar", filenmax=filenmax@entry=1073741824, bsize=bsize@entry=1073741824, plugr=plugr@entry=0x6db980 <plugr>, tid=<optimized out>, krep=0) at turbobench.c:1249
#2  0x0000000000403365 in main (argc=2, argv=<optimized out>) at turbobench.c:1510

TurboBench/turbobench.c

Lines 238 to 251 in 51e2961

	#define STACK_MAGIC 0x79a53fb6
	static const size_t STACK_SIZE = ((1024*1024*7)/sizeof(unsigned));
	unsigned *stackini(void) {
	unsigned stack[STACK_SIZE],*sp;
	for(sp = stack; sp != &stack[STACK_SIZE]; sp++) *sp = STACK_MAGIC;
	return sp;
	}
	size_t stackpeak(unsigned *_sp) {
	unsigned *sp;
	for(sp = _sp - STACK_SIZE; *sp == STACK_MAGIC; sp++);
	return (_sp - sp)*sizeof(unsigned);
	}

fix

replace -O3 with -O1

TurboBench claims to have tons of codecs, but I only see a few

Unlike lzbench, TurboBench is quite useless. Doesn't have an easy way to test all possible codecs to see which one compresses best or decompresses faster.

It absolutely HAS TO HAVE -eall mode like lzbench. I absolutely wasn't able to even test lzturbo.

a bug about sserangecoding, in plugin.cc:ssercenc & ssercdec

Test data: 0x0000
during encoding, the cum prob table is: 0, 2731
However, when decoding, the cum prob table is: 0, 4096 (because of scaled_cum_prob[a] = (1<<SSE_BITS))

Therefore, we should save the whole cum prob table and read it back when decoding, i.e.,
plugin.cc:891, useless now
plugin.cc:906, i < a + 1
plugin.cc:921, i < a + 1
plugin.cc:922, delete.

Hope this helps.

Option - b is invalid for fse and fsehuf

1K：-b1k
./turbobench ../dataset/book -efsehuf -b1k
851499254 79.3 681.55 1003.72 fsehuf

4k：-b4k
./turbobench ../dataset/book -efsehuf -b4k
851499254 79.3 681.85 1002.65 855.22 fsehuf

Suggestions for Codecs

Are you interested in adding new codecs?
I've been testing lately a lot of compressors specific for short strings, some suggestions

• shoco
• smaz
• unishox

If you are interested in more of these I can dig my archives.

Turbobench is marvellous testing tool, many thanks for creating it!

[INCOMPLETE] Windows static binary

Dear @powturbo,
Could you be so kind to generate static .exe for the rest of us who have no various lib*.dll libraries?