how do u know if terark-zip table is working?

[bytedance]

how do u know if terark-zip table is working?

[skyzh]

You can create a breakpoint on TerarkZipTable creation. Or you may use sst_dump tool to get information.

[superdolt]

just to confirm. does this mean it is using terarkzip correctly? coz zstd is stated

./sst_dump --file=005967.sst --show_properties
from [] to []
Process 005967.sst
Sst file format: block-based
Table Properties:is:issue is:open 
------------------------------
  # data blocks: 19124
  # entries: 870547
  # deletions: 0
  # merge operands: 0
  # range deletions: 0
  raw key size: 49783841
  raw average key size: 57.186850
  raw value size: 36562974
  raw average value size: 42.000000
  data block size: 60045443
  index block size (user-key? 0, delta-value? 0): 718092
  filter block size: 2315629is:issue is:open 
  (estimated) table size: 63079164
  filter policy name: rocksdb.BuiltinBloomFilter
  prefix extractor name: nullptr
  column family ID: 6
  column family name: f
  comparator name: leveldb.BytewiseComparator
  merge operator name: nullptr
  value meta extractor name: nullptr
  property collectors names: []
  SST file compression algo: ZSTD
  creation time: 1628963688
  time stamp of earliest key: 0
  Raw user collected properties
------------------------------
  # rocksdb.block.based.table.index.type: 0x00000000
  # rocksdb.block.based.table.prefix.filtering: 0x30
  # rocksdb.block.based.table.whole.key.filtering: 0x31
  # rocksdb.deleted.keys: 0x00
  # rocksdb.merge.operands: 0x00

[skyzh]

Sst file format: block-based

Seems to be a Block-based Table.

[superdolt]

can u show what it means to be terarkzipped? how will it look like? what special setting i need to put to use terarkzip?

[superdolt]

by the way, your documentation is very old. it doesnt seem to work even with the example compilation (doesnt work at all)

[skyzh]

You may use db_bench to fill a TerarkDB database with TerarkZipTable. Here is an example command:

./db_bench \
        --benchmarks=fillrandom \
        --use_existing_db=0 \
        --statistics=0 \
        --stats_per_interval=1 \
        --stats_interval_seconds=60 \
        --max_background_flushes=6 \
        --max_background_compactions=15 \
        --enable_lazy_compaction=0 \
        --level0_file_num_compaction_trigger=4 \
        --sync=1 \
        --allow_concurrent_memtable_write=1 \
        --bytes_per_sync=32768 \
        --wal_bytes_per_sync=32768 \
        --delayed_write_rate=419430400 \
        --enable_write_thread_adaptive_yield=1 \
        --threads=1 \
        --num_levels=7 \
        --key_size=36 \
        --value_size=8192 \
        --level_compaction_dynamic_level_bytes=true \
        --mmap_read=false \
        --compression_type=zstd \
        --memtablerep=patricia_trie \
        --blob_size=1024 \
        --blob_gc_ratio=0.0625 \
        --write_buffer_size=268435456 \
        --max_write_buffer_number=10 \
        --target_file_size_base=134217728 \
        --target_blob_file_size=134217728 \
        --blob_file_defragment_size=33554432 \
        --max_dependence_blob_overlap=128 \
        --optimize_filters_for_hits=true \
        --optimize_range_deletion=true \
        --num=60000000 \
        --db=test_db_1 \
        --benchmark_write_rate_limit=20971520 \
        --use_terark_table=1

--use_terark_table=1 enables TerarkZipTable on disk, and --memtablerep=patricia_trie enables in-memory trie format.

sst_dump doesn't support reading a TerarkZipTable. We have created a separate issue #108 for that.