Giter Site home page Giter Site logo

Comments (12)

hit-lacus avatar hit-lacus commented on August 16, 2024

Cluster Resource

Key Value
Node Memory 64GB
Node Core 16
Node (Instance Type) m5.4xlarge
Node Disk 400GB * 2, SSD
Node Num 4 Worker + 1 Master
EMR Version emr-5.31.0
Kylin Version 4.0.0
Yarn Memory 204.80 GB
Yarn Core 52
  • Scripts
  aws emr create-cluster --applications Name=Hadoop Name=Hive Name=Spark Name=ZooKeeper Name=Tez Name=Ganglia \
    --tags 'Project=Kylin_Benchmark' 'Owner=xiaoxiang.yu' \
    --ec2-attributes '{"KeyName":"?","AdditionalSlaveSecurityGroups":["?"],"InstanceProfile":"?","SubnetId":"?","EmrManagedSlaveSecurityGroup":"?","EmrManagedMasterSecurityGroup":"?","AdditionalMasterSecurityGroups":["?"]}' \
    --release-label emr-5.31.0 \
    --instance-groups '[{"InstanceCount":4,"EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"SizeInGB":400,"VolumeType":"gp2"},"VolumesPerInstance":2}]},"InstanceGroupType":"CORE","InstanceType":"m5.4xlarge","Name":"Hadoop Workers"},{"InstanceCount":1,"InstanceGroupType":"MASTER","InstanceType":"m5.4xlarge","Name":"Hadoop Master"}]' \
    --configurations '[{"Classification":"hdfs-site","Properties":{"dfs.replication":"1"}},{"Classification":"mapred-site","Properties":{"mapreduce.map.memory.mb":"3584","mapreduce.reduce.memory.mb":"8192","mapreduce.map.java.opts":"-Xmx3072m","mapreduce.reduce.java.opts":"-Xmx7168m"}},{"Classification":"yarn-site","Properties":{"yarn.nodemanager.resource.cpu-vcores":"13","yarn.nodemanager.resource.memory-mb":"52428","yarn.scheduler.maximum-allocation-mb":"52428","yarn.app.mapreduce.am.resource.mb":"2048"}}]' \
    --auto-scaling-role EMR_AutoScaling_DefaultRole \
    --ebs-root-volume-size 100 \
    --service-role EMR_DefaultRole \
    --name 'Kylin4 Benchmark' \
    --scale-down-behavior TERMINATE_AT_TASK_COMPLETION \
    --region cn-northwest-1

from kylin-tpch.

hit-lacus avatar hit-lacus commented on August 16, 2024

Cubing duration

Cube Value
lineitem 98.39 min
partsupp 5.10 min
customer_vorder 2.70 min
customer 1.42 min

from kylin-tpch.

hit-lacus avatar hit-lacus commented on August 16, 2024

Storage Size

Cube Value
lineitem 257.28 GB
partsupp 4.92 GB
customer_vorder 1.19 GB
customer 0.038 GB

from kylin-tpch.

hit-lacus avatar hit-lacus commented on August 16, 2024

Response Time (thread = 1)

image

from kylin-tpch.

hit-lacus avatar hit-lacus commented on August 16, 2024

Kylin Configuration

kylin.metadata.url=benchmark_kylin312@jdbc,url=jdbc:mysql://ip-172-31-11-46.cn-northwest-1.compute.internal:3306/hive,username=hive,password=nzSqiiWPGj5Gqzp3,maxActive=10,maxIdle=10,driverClassName=org.mariadb.jdbc.Driver
kylin.env.zookeeper-connect-string=localhost:2181

## Disable retry
kylin.engine.max-retry-time=1

## Build Engine Resource
kylin.engine.spark-conf.spark.executor.cores=2
kylin.engine.spark-conf.spark.executor.instances=25
kylin.engine.spark-conf.spark.executor.memory=7GB
kylin.engine.spark-conf.spark.executor.memoryOverhead=1GB

## Query Engine Resource
kylin.query.spark-conf.spark.master=yarn
kylin.query.spark-conf.spark.driver.cores=1
kylin.query.spark-conf.spark.driver.memory=8GB
kylin.query.spark-conf.spark.driver.memoryOverhead=1G
kylin.query.spark-conf.spark.executor.cores=1
kylin.query.spark-conf.spark.executor.instances=40
kylin.query.spark-conf.spark.executor.memory=4G
kylin.query.spark-conf.spark.executor.memoryOverhead=1G
kylin.query.spark-conf.spark.sql.parquet.filterPushdown=false

## Disable canary
kylin.canary.sparder-context-canary-enabled=false


## Shard setting
kylin.storage.columnar.shard-size-mb=75
kylin.storage.columnar.shard-rowcount=1200000
kylin.storage.columnar.shard-countdistinct-rowcount=600000
kylin.storage.columnar.repartition-threshold-size-mb=75
kylin.engine.spark-conf.spark.hadoop.parquet.block.size=268435456

from kylin-tpch.

hit-lacus avatar hit-lacus commented on August 16, 2024

Comment

from kylin-tpch.

hit-lacus avatar hit-lacus commented on August 16, 2024

Screenshots

from kylin-tpch.

hit-lacus avatar hit-lacus commented on August 16, 2024

Screenshots of Kylin

Cube Meta

image

image

Job Meta

image

HDFS

image

image

image

Yarn UI

image

Executor tab of Spark UI

image

from kylin-tpch.

hit-lacus avatar hit-lacus commented on August 16, 2024

Metrics of Cube Building

First Segment of LineitemCube

image

image

image

image

image

image

Second Segment of LineitemCube

image

image

image

from kylin-tpch.

hit-lacus avatar hit-lacus commented on August 16, 2024

Storage (Size of each shard under specific cuboid dir)

  • Base cuboid of lineitem_cube_optimized
[hadoop@ip-172-31-11-46 apache-kylin-4.0.0-SNAPSHOT-bin]$ hadoop fs -du -h /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00000-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00001-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00002-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00003-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00004-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.4 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00005-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00006-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00007-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.4 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00008-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00009-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00010-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00011-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00012-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00013-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00014-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00015-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00016-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00017-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00018-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00019-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00020-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00021-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00022-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00023-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.6 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00024-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00025-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00026-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00027-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00028-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00029-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00030-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00031-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00032-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00033-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00034-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00035-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00036-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.5 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00037-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet
88.4 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/4294967295/part-00038-eb2c2b7f-6cf1-4f11-99ae-8bee690afbaf-c000.snappy.parquet

[hadoop@ip-172-31-11-46 apache-kylin-4.0.0-SNAPSHOT-bin]$
[hadoop@ip-172-31-11-46 apache-kylin-4.0.0-SNAPSHOT-bin]$ hadoop fs -du -h /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00000-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00001-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.2 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00002-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00003-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.2 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00004-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.2 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00005-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00006-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00007-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.2 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00008-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.2 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00009-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.2 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00010-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.2 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00011-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00012-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00013-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00014-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.2 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00015-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00016-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00017-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00018-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.2 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00019-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.2 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00020-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00021-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00022-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00023-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00024-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00025-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00026-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet
88.3 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19960101000000_19990101000000_964/4294967295/part-00027-15fe2a27-0f1a-4227-b7db-3ef61faf2bbd-c000.snappy.parquet

[hadoop@ip-172-31-11-46 apache-kylin-4.0.0-SNAPSHOT-bin]$ hadoop fs -du -h /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00000-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00001-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00002-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00003-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00004-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00005-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00006-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00007-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00008-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00009-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00010-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00011-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00012-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00013-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00014-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00015-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
68.0 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00016-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00017-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00018-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.8 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00019-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00020-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.8 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00021-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00022-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00023-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00024-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00025-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00026-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet
67.9 M  /kylin/benchmark_kylin312/tpch/parquet/lineitem_cube_optimized/19920101000000_19960101000000_SHX/3221352992/part-00027-30e5a6c9-abee-45ff-907f-0af3dbceb774-c000.snappy.parquet

[hadoop@ip-172-31-11-46 apache-kylin-4.0.0-SNAPSHOT-bin]$ hadoop fs -du -h /kylin/benchmark_kylin312/tpch/parquet///customer_vorder_cube/FULL_BUILD_AZC/31
94.7 M  /kylin/benchmark_kylin312/tpch/parquet/customer_vorder_cube/FULL_BUILD_AZC/31/part-00000-816fd3a3-b27f-4301-9c2c-a82ce663a7e8-c000.snappy.parquet
94.7 M  /kylin/benchmark_kylin312/tpch/parquet/customer_vorder_cube/FULL_BUILD_AZC/31/part-00001-816fd3a3-b27f-4301-9c2c-a82ce663a7e8-c000.snappy.parquet
94.7 M  /kylin/benchmark_kylin312/tpch/parquet/customer_vorder_cube/FULL_BUILD_AZC/31/part-00002-816fd3a3-b27f-4301-9c2c-a82ce663a7e8-c000.snappy.parquet
94.7 M  /kylin/benchmark_kylin312/tpch/parquet/customer_vorder_cube/FULL_BUILD_AZC/31/part-00003-816fd3a3-b27f-4301-9c2c-a82ce663a7e8-c000.snappy.parquet
94.7 M  /kylin/benchmark_kylin312/tpch/parquet/customer_vorder_cube/FULL_BUILD_AZC/31/part-00004-816fd3a3-b27f-4301-9c2c-a82ce663a7e8-c000.snappy.parquet
94.7 M  /kylin/benchmark_kylin312/tpch/parquet/customer_vorder_cube/FULL_BUILD_AZC/31/part-00005-816fd3a3-b27f-4301-9c2c-a82ce663a7e8-c000.snappy.parquet
94.7 M  /kylin/benchmark_kylin312/tpch/parquet/customer_vorder_cube/FULL_BUILD_AZC/31/part-00006-816fd3a3-b27f-4301-9c2c-a82ce663a7e8-c000.snappy.parquet
[hadoop@ip-172-31-11-46 apache-kylin-4.0.0-SNAPSHOT-bin]$
[hadoop@ip-172-31-11-46 apache-kylin-4.0.0-SNAPSHOT-bin]$ hadoop fs -du -h /kylin/benchmark_kylin312/tpch/parquet/customer_cube/FULL_BUILD_SMD/7
38.7 M  /kylin/benchmark_kylin312/tpch/parquet/customer_cube/FULL_BUILD_SMD/7/part-00000-42a8e12e-dbc2-4a04-a72a-dcc2c577e05d-c000.snappy.parquet

from kylin-tpch.

hit-lacus avatar hit-lacus commented on August 16, 2024

Screenshots of Load Test (thread=1, 20 minutes)

image

image

image

image

image

Node Level

image

image

image

from kylin-tpch.

hit-lacus avatar hit-lacus commented on August 16, 2024

Screenshots of Load Test (thread=4, 10 minutes)

image

image

image

image

image

image

image

from kylin-tpch.

Related Issues (7)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.