kafbat/kafka-ui

Open-Source Web UI for managing Apache Kafka clusters

Kafbat UI

Versatile, fast and lightweight web UI for managing Apache Kafka® clusters.

Documentation • Quick Start • Community
AWS Marketplace • ProductHunt

Kafbat UI is a free, open-source web UI to monitor and manage Apache Kafka clusters.

Kafbat UI is a simple tool that makes your data flows observable, helps find and troubleshoot issues faster and deliver optimal performance. Its lightweight dashboard makes it easy to track key metrics of your Kafka clusters - Brokers, Topics, Partitions, Production, and Consumption.

Kafbat UI, developed by Kafbat*, proudly carries forward the legacy of the UI Apache Kafka project. Our dedication is reflected in the continuous evolution of the project, ensuring adherence to its foundational vision while adapting to meet modern demands. We extend our gratitude to Provectus for their past support in groundbreaking work, which serves as a cornerstone for our ongoing innovation and dedication.

* - The Kafbat team comprises key contributors from the project's inception, bringing a wealth of experience and insight to this renewed endeavor.

Interface

Features

• Multi-Cluster Management — monitor and manage all your clusters in one place
• Performance Monitoring with Metrics Dashboard — track key Kafka metrics with a lightweight dashboard
• View Kafka Brokers — view topic and partition assignments, controller status
• View Kafka Topics — view partition count, replication status, and custom configuration
• View Consumer Groups — view per-partition parked offsets, combined and per-partition lag
• Browse Messages — browse messages with JSON, plain text, and Avro encoding
• Dynamic Topic Configuration — create and configure new topics with dynamic configuration
• Configurable Authentification — secure your installation with optional Github/Gitlab/Google OAuth 2.0
• Custom serialization/deserialization plugins - use a ready-to-go serde for your data like AWS Glue or Smile, or code your own!
• Role based access control - manage permissions to access the UI with granular precision
• Data masking - obfuscate sensitive data in topic messages

Feature overview

Click here for the feature overview

The Interface

Kafbat UI wraps major functions of Apache Kafka with an intuitive user interface.

Topics

Kafbat UI makes it easy for you to create topics in your browser by several clicks, pasting your own parameters, and viewing topics in the list.

It's possible to jump from connectors view to corresponding topics and from a topic to consumers (back and forth) for more convenient navigation. connectors, overview topic settings.

Messages

Let's say we want to produce messages for our topic. With the Kafbat UI we can send or write data/messages to the Kafka topics without effort by specifying parameters, and viewing messages in the list.

Schema registry

There are 3 supported types of schemas: Avro®, JSON Schema, and Protobuf schemas.

Before producing avro/protobuf encoded messages, you have to add a schema for the topic in Schema Registry. Now all these steps are easy to do with a few clicks in a user-friendly interface.

Getting Started

To run Kafbat UI, you can use either a pre-built Docker image or build it (or a jar file) yourself.

Quick start (Demo run)

docker run -it -p 8080:8080 -e DYNAMIC_CONFIG_ENABLED=true ghcr.io/kafbat/kafka-ui

Then access the web UI at http://localhost:8080

The command is sufficient to try things out. When you're done trying things out, you can proceed with a persistent installation

Persistent installation

services:
  kafbat-ui:
    container_name: kafbat-ui
    image: ghcr.io/kafbat/kafka-ui:latest
    ports:
      - 8080:8080
    environment:
      DYNAMIC_CONFIG_ENABLED: 'true'
    volumes:
      - ~/kui/config.yml:/etc/kafkaui/dynamic_config.yaml

Please refer to our configuration page to proceed with further app configuration.

Some useful configuration related links

Web UI Cluster Configuration Wizard

Configuration file explanation

Docker Compose examples

Misc configuration properties

Helm charts

Quick start

Building from sources

Quick start with building

Liveliness and readiness probes

Liveliness and readiness endpoint is at /actuator/health.
Info endpoint (build info) is located at /actuator/info.

Configuration options

All the environment variables/config properties could be found here.

Contributing

Please refer to contributing guide, we'll guide you from there.

Support

As we're fully independent, team members contribute in their free time. Your support is crucial for us, if you wish to sponsor us, take a look here

kafka-ui's Issues

Wizard: No enum constant com.provectus.kafka.ui.model.ActionDTO.view

via provectus/kafka-ui#4308

java.lang.IllegalArgumentException: No enum constant com.provectus.kafka.ui.model.ActionDTO.view
        at java.base/java.lang.Enum.valueOf(Enum.java:273)
        Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyExceptio**

TODO: Come up with a way to treat these enums with ignoring the case (view = VIEW)

Impl details:

A config like this:

        - resource: clusterconfig
          value: ".*"
          actions: all

when accessed via wizard and mapped via PropertiesMapper in ApplicationConfigController fails with the aforementioned exception as it tried to map 1:1 without ignoring the case.

Also, it tried to map the "all" action as well which doesn't exist in the enum.

RBAC: Connect filter doesn't seem to work

provectus/kafka-ui#4253

BE: Increase default audit topic retention

provectus/kafka-ui#4257

RBAC: Improve role assigning logging

Wizard: Add an ability to delete clusters from UI

Publish docker images to github packages

Brokers: Configs: Improvements

• display friendly units (sizes, time units)

All keys with ".bytes" in the name are sizes
All keys with ".ms" in the name are time units

• replace enums with human friendly string for source tooltips

Just use the human readable values, like DEFAULT_CONFIG -> "Default config"

• Display "changed first" (based on source)

Order priority:

1. DYNAMIC_* source
2. STATIC_BROKER_CONFIG source
3. DEFAULT_CONFIG being last
4. UNKNOWN

• Allow manual sorting by source

• Mask sensitive params values by default (isSensitive: true)

• Make r/o fields (isReadOnly: true) uneditable (disable the button with "Property is read-only" tooltip)

Audit history

BE: Messages: Polling breaks when one broker is down

provectus/kafka-ui#3948

UI doesn't display headers duplicates

provectus/kafka-ui#4352

Also: support null headers? via provectus/kafka-ui#4394

Brokers: Configs - The selection is not being reset when clearing the search

Reset the search component text in broker configs and the selection is kept intact rather being reset to the full list

provectus/kafka-ui#4179

Wizard: Serde configs are lost when changes are made in wizard

provectus/kafka-ui#4048

Replace checkstyle plugin

We require a viable solution to fix the checkstyle errors right in IDE. Currently you have to compile and visit each class present in logs.
This should work something like eslint/prettier with visual indication of invalidly formatted lines.

Split build & upload jobs

Currently build and uploading (or other stuff) is always a one job within
a workflow. So if something fails, it has to be rebuilt. We need to split
build and upload functions. Use upload/dl artifact action to transfer the
build between the jobs.

BE: Topics: Validate ISR/replication upon creation

It's currently possible to edit a topic so the min iSR > replication factor.

Filters: Implement a filter evaluator

BE: Implement a mechanism to skip SSL verification

provectus/kafka-ui#4082
provectus/kafka-ui#4083
#268

Create a dockerfile to build an image (+jar) with maven inside

1. For third-party users to make it easier without instaling maven on the host machine.
2. Will need to disable (probly a separate mvn profile) docker plugin.

Topics: Azure: Unable to get topic configs if one topic does not have sufficient permissions

Kafka UI fails to display topics if its credentials have access to a subset of Azure Event Hubs existing in the same namespace.

Basically the use case is like this:
There are multiple Azure Event Hubs in the same namespace. There are used by different teams. Every team has access to a subset of Event Hubs (topics) and uses corresponding credentials. Everything works perfectly fine except Kafka UI. It displays no topics.

provectus/kafka-ui#4300

Heading font sizes don't match

Figma:

Implementation:

BE: FE: Topics: Allow custom topic params upon creation

provectus/kafka-ui#4247

provectus/kafka-ui#4367

Migrate to Gradle

After #90

Refactor maven profiles

Current issues:

1. Some users need an API build without a frontend. There's a "skipuibuild" property for that. Doesn't seem optimal.
2. Local build doesn't build frontend, which is usually required for a local run. Prod profile, on the other hand, builds a docker image, no way to easily disable this. Which is usually not require for a local run.

FE: Create connector button throws 404 if KC cluster is inaccessible

Frontend does POST to
http://localhost:8080/api/clusters/local/connects//connectors (not double slash (missing connect name))

with this RBAC configuration:

        - resource: connect
          value: "test-"
          actions: all

and local connect with name "local" present

FE: ACL management

Currently, we have r/o ACL management, this issue is aimed at implementing a r/w ability.

Add a sponsor list image in README

• Set up the github sponsorship first: @germanosin
• Add an image: @Haarolean

Replace docker hub images with github packages

• Adjust all workflows to replace publishing docker images to GitHub packages rather than to the docker hub

FE: Cluster name is not escaped

provectus/kafka-ui#4345

Online partition count is red

https://discord.com/channels/897805035122077716/1197881542052159608

brokers.json:

[
    {
        "id": 1,
        "host": "test-region-1-broker-1.kafka",
        "port": 9092,
        "bytesInPerSec": null,
        "bytesOutPerSec": null,
        "partitionsLeader": 18,
        "partitions": 110,
        "inSyncPartitions": 110,
        "partitionsSkew": 0.0,
        "leadersSkew": -1.8
    },
    {
        "id": 2,
        "host": "test-region-1-broker-2.kafka",
        "port": 9092,
        "bytesInPerSec": null,
        "bytesOutPerSec": null,
        "partitionsLeader": 18,
        "partitions": 110,
        "inSyncPartitions": 110,
        "partitionsSkew": 0.0,
        "leadersSkew": -1.8
    },
    {
        "id": 3,
        "host": "test-region-1-broker-3.kafka",
        "port": 9092,
        "bytesInPerSec": null,
        "bytesOutPerSec": null,
        "partitionsLeader": 18,
        "partitions": 110,
        "inSyncPartitions": 110,
        "partitionsSkew": 0.0,
        "leadersSkew": -1.8
    },
    {
        "id": 4,
        "host": "test-region-2-broker-1.kafka",
        "port": 9092,
        "bytesInPerSec": null,
        "bytesOutPerSec": null,
        "partitionsLeader": 19,
        "partitions": 110,
        "inSyncPartitions": 110,
        "partitionsSkew": 0.0,
        "leadersSkew": 3.6
    },
    {
        "id": 5,
        "host": "test-region-2-broker-2.kafka",
        "port": 9092,
        "bytesInPerSec": null,
        "bytesOutPerSec": null,
        "partitionsLeader": 18,
        "partitions": 110,
        "inSyncPartitions": 110,
        "partitionsSkew": 0.0,
        "leadersSkew": -1.8
    },
    {
        "id": 6,
        "host": "test-region-2-broker-3.kafka",
        "port": 9092,
        "bytesInPerSec": null,
        "bytesOutPerSec": null,
        "partitionsLeader": 19,
        "partitions": 110,
        "inSyncPartitions": 110,
        "partitionsSkew": 0.0,
        "leadersSkew": 3.6
    }  
]

stats.json:

{
    "brokerCount": 6,
    "zooKeeperStatus": null,
    "activeControllers": 3,
    "onlinePartitionCount": 110,
    "offlinePartitionCount": 0, 
    "inSyncReplicasCount": 660,
    "outOfSyncReplicasCount": 0,
    "underReplicatedPartitionCount": 0,
    "diskUsage": [
        {
            "brokerId": 1, 
            "segmentSize": 15296,
            "segmentCount": 110
        },
        {
            "brokerId": 2, 
            "segmentSize": 15296,
            "segmentCount": 110
        },
        {
            "brokerId": 3, 
            "segmentSize": 15296,
            "segmentCount": 110
        },
        {
            "brokerId": 4, 
            "segmentSize": 15608,
            "segmentCount": 130
        },
        {
            "brokerId": 5, 
            "segmentSize": 15892,
            "segmentCount": 131
        },
        {
            "brokerId": 6, 
            "segmentSize": 15608,
            "segmentCount": 132
        }
    ],
    "version": "2.7-IV2"
}

FE: Sorting connectors duplicates items

Screen.Recording.2024-02-01.at.20.09.49.mov

Partition reassignment management

Replace groovy filters with CEL

https://github.com/google/cel-java

Should auto fix #30

RBAC: Topic create permission issues

provectus/kafka-ui#4328

RBAC: Google: Support google groups

via provectus/kafka-ui#3279

Audit: ClassCastException when OAuth is enabled

2023-11-02 08:54:56,490 ERROR [parallel-5] r.c.p.Operators: Operator called default onErrorDropped
reactor.core.Exceptions$ErrorCallbackNotImplemented: java.lang.ClassCastException: Cannot cast
org.springframework.security.oauth2.core.oidc.user.DefaultOidcUser to
org.springframework.security.core.userdetails.UserDetails
Caused by: java.lang.ClassCastException: Cannot cast
org.springframework.security.oauth2.core.oidc.user.DefaultOidcUser to
org.springframework.security.core.userdetails.UserDetails
at java.base/java.lang.Class.cast(Class.java:3889)
at
reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)
at
reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
at reactor.core.publisher.MonoCacheTime.subscribeOrReturn(MonoCacheTime.java:151)
at reactor.core.publisher.Mono.subscribe(Mono.java:4480)
at reactor.core.publisher.Mono.subscribeWith(Mono.java:4561)
at reactor.core.publisher.Mono.subscribe(Mono.java:4323)
at com.provectus.kafka.ui.service.audit.AuditService.audit(AuditService.java:186)
at com.provectus.kafka.ui.controller.AbstractController.audit(AbstractController.java:30)
at
com.provectus.kafka.ui.controller.TopicsController.lambda$getTopicDetails$10(TopicsController.java:168)

Rebranding

1. Get rid of the ugly stuff
2. Get all the features under the spoiler
3. Update links
4. Other refactoring

first issue yolo!

Update color palette

Update the palette from blue-ish to the palette we've decided on

CVE: CVE-2023-52251, RCE w/ groovy filters

https://github.com/BobTheShoplifter/CVE-2023-52251-POC

KC: Invalid task state upon a restart

provectus/kafka-ui#1890

Determine why backend tests are skipped

https://github.com/kafbat/kafka-ui/actions/runs/7658545852/job/20871724134?pr=61
neither "build and analyze" has been triggered

FE: Rename "max size on disk" to "Max partition size"

provectus/kafka-ui#4037

TODO:

• Rename "max size on disk" to "Max partition size"

FE: Messages: Display a schema used to deserialize a message

provectus/kafka-ui#4325

Stream Topology visualizer

RBAC: Support Active Directory

LdapSecurityConfig:

• verify ActiveDirectoryLdapAuthenticationProvider does populate authorities properly (most likely, it doesn't)
• if it doesn't, copy-paste ActiveDirectoryLdapAuthenticationProvider
  and (re)implement a populator (unless #4490 is resolved)

via provectus/kafka-ui#3741

Broker not shown in a list if disk usage is unknown for it

#3615

branch issues/3615

RBAC: BE: Can't delete a connector on FE

Can't delete a connector due to the frontend waiting for a DELETE permission, but for deleting one on the backend EDIT is required and DELETE doesn't exist.

Messages V2: API overhaul

This refactoring should address all the issues related to messages including

• pagination
• sorting
• frontend complexity enshittification

SR: Support schema references

Implement an authentication page

• Get rid of AuthController, do that on the frontend instead.
• Get rid of the default spring security login page
• Leave the csrf endpoint public, call when needed from the frontend.
• Impl a custom auth page for all the auth methods (oauth, ldap, basic)

Should fix:

• provectus/kafka-ui#4178

Infra: Support r/o FS for the image

provectus/kafka-ui#4318