zehaowei/risebench

A benchmark tool for RisingWave

License

The workload of this benchmark is derived from the TPC-H Benchmark and as such is not comparable to published TPC-H Benchmark results, as the workload results do not comply with the TPC-H Benchmark.

Differences from TPC-H Benchmark

• This benchmark is not intended to be used for benchmark results that are "comparable to TPC Benchmark Results".
• This benchmark uses the same table schemas as TPC-H Benchmark but modify the queries to make them compatible with the system under test.
• The data of tables is not generated by dbgen but we use dists.dss as the seed of data generators.
• This benchmark is not for public use.

We recommend that you download TPC-H from the official website for the purpose of getting "TPC-H Benchmark Results".

RiseBench User Guide

This work is still in progress and the workflow would be optimized in the future.

1.Download & Set up Kafka Cluster

# Download Kafka
https://kafka.apache.org/downloads

# Launch Kafka
# Kafka broker default port: 9092
cd ${KafkaDir}
bin/zookeeper-server-start.sh config/zookeeper.properties

# Launch only one Kafka broker
bin/kafka-server-start.sh config/server.properties

We recommend that to launch 2-3 kafka brokers. For launching multiple servers, please replicate file server.properties and change the following config in the file.

broker.id=1
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs-1
zookeeper.connect=localhost:2181

2.Configue and launch RisingWave

# Uncomment 30 and 31 line of `riselab.yml` to open Prometheus & Grafana
# Grafana for RisingWave link: localhost:3001
./risedev configure ## check Enable release mode
./risedev d

3.Start benchmark

# Build the project
cd ./scripts
./build.sh

# Run the benchmark tool
cd ..

# For detailed explanation please see the appendix

# frontend: frontend ip address of RisingWave
# kafka-addr: Kafka address that will be sent to RisingWave frontend 
# qps: data producing rate of MainTable
# scale: tpch dataset scale (ex: for lineitem, 1.0 = 6,000,000)
# query: tpch query id
# i: query the results of the MV every `i` seconds

# Run this cmd to execute a query, after you have run a query, please run the clean cmd below
# If you want to run self-define query, add new sql file named as "q%d" (ex: q22) under fold "assets/data"
./bin/bench --frontend 127.0.0.1 --kafka-addr localhost:9092 --partition 4 --type=tpch-std --qps=300000 --scale 10.0 --query 1 --i 5

# Clean RisingWave and Kafka after the bench (make query id the same as the above one)
./bin/bench --frontend 127.0.0.1 --type=tpch-clean --query 1

# Only send data to Kafka 
./bin/bench --type=tpch-k --partition 4 --qps=300000 --scale 1.0 --query 1

# Only create source and MV in RisingWave
./bin/bench --frontend 127.0.0.1 --kafka-addr localhost:9092 --type=tpch-q --query 1

Note

The default frontend is rust frontend, if you want to use legacy frontend, please add --legacy-frontend to the cmd

Parameter

1.Benchmark type --type

• tpch-std: create 8 topics(one topic for one TPC-H table) in Kafka, create 8 Kafka source tables in RisingWave, send data rows of small tables to Kafka, create MV in RisingWave according to query, finally send data of MainTable(usually lineitem) to Kafka in real time.

• tpch-clean: delete 8 topics in Kafka, if giving query, it will drop MV and source tables in RisingWave.

• tpch-k: create 8 topics and send all data rows to Kafka.

• tpch-q: create 8 Kafka source tables in RisingWave and create MV in RisingWave according to query.

2.Kafka config

• --kafka-addr
  Kafka broker address that will be sent to RisingWave. For example: localhost:9092

• --partition
  We recommend that users set patition = 2 * ComputeNodeNums WARN: When running some queries using partition > 1, RisingWave might panic due to its deficits in old implementation of source.

3.RisingWave config

• --frontend
  Frontend ip address of RisingWave
• --frontend port
  Frontend port of RisingWave
• --legacy-frontend
  Enable Java frontend

4.Benchmark config

• --qps
  Data producing rate of MainTable
• --scale
  TPCH dataset scale (ex: for lineitem, 1.0 = 6,000,000 ≈ 2GB)
• --query
  TPCH query id
• --i
  Query the results of the MV every i seconds