dut3062796s/deepflow

Automated observability using eBPF

DeepFlow is an automated observability platform for cloud-native developers.

---

English | 简体中文

What is DeepFlow

DeepFlow is a highly automated observability platform for cloud-native developers. Using new technologies such as eBPF, WASM, and OpenTelemetry, DeepFlow innovatively implements core mechanisms such as AutoTracing, AutoMetrics, AutoTagging, and SmartEncoding, which greatly avoids code instrumentation and significantly reduces the resource overhead of back-end data warehouses. With the programmability and open API of DeepFlow, developers can quickly integrate it into their own observability stack.

Key Features

• Any Stack: With the AutoMetrics mechanism implemented by eBPF and cBPF, DeepFlow can automatically collect RED (Request, Error, Delay) performance metrics of any application, down to every request, covering all software technologie stacks from application to infrastructure. In cloud-native environments, the AutoTagging mechanism automatically discovers the attributes of services, instances and APIs, and automatically injects rich tags into each observability data, thereby eliminating data silos and releasing data drill-down capabilities.
• End to End: DeepFlow innovatively implements the AutoTracing mechanism using eBPF technology. It automatically traces the distributed request of any application and infrastructure service in cloud-native environments. On this basis, by integrating and automatically correlating data from OpenTelemetry, DeepFlow implements a complete full-stack, full-path distributed tracing, eliminating all blind spots.
• High Performance: The innovative SmartEncoding tag injection mechanism can improve the storage performance by 10 times, no more high-cardinality and sampling anxiety. DeepFlow Agent is implemented in Rust for extreme processing performance and memory safety. DeepFlow Server is implemented in Golang, and rewrites standard library map and pool for a nearly 10x performance in data query and memory GC.
• Programmability: DeepFlow supports collect HTTP(S), Dubbo, MySQL, PostgreSQL, Redis, Kafka, MQTT and DNS at the moment, and will iterate to support more application protocols. In addition, DeepFlow provides a programmable interface based on WASM technology, allowing developers to add private protocols quickly, and can be used to construct business analysis capabilities for specific scenarios, such as 5GC signaling analysis, financial transaction analysis, vehicle computer communication analysis, etc.
• Open Interface: DeepFlow embraces the open source community, supports a wide range of observability data sources, and uses AutoTagging and SmartEncoding to provide high-performance, unified tag injection capabilities. DeepFlow has a plugable database interface, developers can freely add and replace the most suitable database. DeepFlow provides a unified standard SQL query capability for all observability data, which is convenient for users to quickly integrate into their own observability platform.
• Easy to Maintain: The core of DeepFlow only consists of two components, Agent and Server, hiding the complexity within the process and reduces the maintenance difficulty to the extreme. The DeepFlow Servers can manage Agents in multiple kubernetes clusters, legacy hosts and cloud hosts in a unified manner, and can achieve horizontal scaling and load balancing without any external components.

Documentation

For more information, please visit the documentation website.

Quick start

There are three editions of DeepFlow:

• DeepFlow Community: for developers
• DeepFlow Enterprise: for organizations, solving team collaboration problems
• DeepFlow Cloud: SaaS service, currently in beta

The DeepFlow Community Edition consists of the core components of the Enterprise Edition.

DeepFlow Community

Please refer to the deployment documentation.

At the same time, we have also built a complete DeepFlow Community Demo, welcome to experience it. Login account/password: deepflow/deepflow.

DeepFlow Cloud

DeepFlow Cloud is the fully-managed service of DeepFlow, currently in beta and only supports Chinese.

DeepFlow Enterprise

DeepFlow Enterprise supports full-stack and end-to-end monitoring of hybrid cloud, covering containers, cloud servers, hosts, and NFV gateways, currently only supports Chinese, welcome to contact us for experience.

Compile DeepFlow from Source

• compile deepflow-agent

Software Architecture

DeepFlow Community Edition consists of two components, Agent and Server. An Agent runs in each K8s node, legacy host and cloud host, and is responsible for AutoMetrics and AutoTracing data collection of all application processes on the host. Server runs in a K8s cluster and provides Agent management, tag injection, data ingest and query services.

Milestones

Here is our future feature plan. Issues and Pull Requests are welcome.

Join DeepFlow WeChat Group

Join the WeChat group，you can communicate with other users (in Chinese):

Acknowledgments

• Thanks eBPF, a revolutionary Linux kernel technology.
• Thanks OpenTelemetry, provides vendor-neutral APIs to collect application telemetry data.

Landscapes

• DeepFlow enriches the CNCF CLOUD NATIVE Landscape.
• DeepFlow enriches the eBPF Project Landscape.