We introduce our state-of-the-art tool viz Getafix, that has been specifically designed for the effective tuning of Apache Druid clusters, focusing on three key factors: configuration parameters, query tuning, and data modeling based on usage patterns. However, Getafix puts particular emphasis on the first and third factors to provide a specialized, more targeted approach.
In the realm of configuration parameters, our tool ensures to optimize your Druid cluster to the best possible extent. These parameters include but are not limited to memory settings, thread pools, and query capacities. You can effortlessly adjust these parameters to the specific requirements of your Druid cluster, ensuring optimum performance and utilization.
While our tool doesn't directly deal with query tuning, the second factor, it's important to note that well-configured parameters can indirectly contribute to more efficient query execution, thus enhancing overall system performance.
The third factor, data modeling based on usage patterns, is another area where our tool excels. With the aid of Getafix, it comprehends your cluster's data dynamics and adaptively remodels the data based on changing usage patterns. This ensures your Druid cluster remains efficient and delivers top-notch performance, even with shifting data demands and usage trends.
By combining these functionalities, the tool delivers a streamlined approach to Apache Druid cluster tuning. It simplifies the intricate process of cluster optimization and provides valuable insights into data usage patterns, leading to an exceptionally well-tuned and high-performing Apache Druid environment.
##Approach## Its recommend to do a cluster health check on a regular interval to ensure that the cluster is always tuned even with the changing the query patterns.
A cluster review scorecard used to evaluate the configuration and operation of Apache Druid clusters.
Designed to be executed in a cumulative way, depending on your interest and ability to collect information.
- Run API queries against the cluster to get information about datasources, compaction etc. *
- Add node size info and and compare to Druid cluster guidelines for this
- Collect Broker logs to understand best practices for partitioning, tiering
- Collect metrics (Clarity / Logs / Prometheus) information to identify hotspots / trends
Currently collected
API query | Description |
---|---|
compaction | Configuration of compaction for each applicable datasource |
compaction status | Current status of compaction for each applicable datasource |
coordinator dynamic config | Current configuration of the coordinator dynamic configuration |
datasources | Detailed information for all segments contained in each datasource |
lookups | Configuration information for all configured lookups |
overlord dynamic config | Current configuration of the overlord dynamic configuration |
retention | Current retention policies for all configured datasources |
segments | Number of segments, average size and average number of rows for each datasource |
servers | List of servers (services) which comprise the cluster |
supervisors | Details of configured supervisors |
tasks | A snapshot of recently run tasks |
workers | Number of workers available |
To add
/status/properties
from all available services to understand Druid tuning parameters
Evaluate data based on cDMN - can be expressed in XLS or DMN file types
file:///Users/khoondert/Documents/Data/Dev/temp/Getafix.html