devjjo/kafka-zk-restapi

Kafka Zookeeper RESTful API to perform topic/consumer group administration/metric(offset\lag\message) collection and monitor

Kafka REST API

Overview

Kafka/ZK REST API is to provide the production-ready endpoints to perform some administration/metric task for Kafka and Zookeeper.

Following features are provided:

• Broker List

• Topic create/delete/describe

• Topic config create/update/delete/list

• Topic partition add and reassign

• Consumer group(old zookeeper based/new kafka based) list/describe

• Offset check/reset

• Consumer Group Lag check

• Collect JMX metrics from brokers that expose JMX metrics
  More details refer to JMXCollector API Specification

• Secure the REST API with Spring Security

Installation and run

You can download the release package at https://github.com/gnuhpc/Kafka-zk-restapi/releases

You can get it running by unzip/untaring the distribution package and run bin/start.sh

Java 8 jre/jdk needed.

How to build and run

You can build this restapi server using maven.

#git clone https://github.com/gnuhpc/Kafka-zk-restapi.git
#cd Kafka-zk-restapi/

Change the following settings of application-home.yml in src/main/resources to valid value: kafka.brokers zookeeper.uris

Change security related parameters in application config file: See below: 'How to config security'

server.security.check server.security.checkInitDelay server.security.checkSecurityInterval

If you want to use JMX Query Filter function, you can add your own custom filter files to JMXFilterTemplate directory in project root folder. More details refer to JMXCollector API Specification

#mvn clean package -Dmaven.test.skip=true+

You will find zip/tar under directory: Kafka-zk-restapi/target

You can get it running by unzip/untaring the distribution package and run bin/start.sh

Security

Public REST services without access control make the sensitive data under risk.Then we provide a simple authentication mechanism using Spring Security. In order to make the project lighter, we use yml file to store user information, not using database.

Follow the steps to enable security feature：

Step 1：Modify the application config file and set server.security.check to true.

• server.security.check:

  • True: Add security for the API. Clients can access the API with valid username and password stored in security.yml, or the Swagger UI(http://127.0.0.1:8121/api) is only allowed to access.

  • False: All the endpoints can be accessed without authentication.

• server.security.checkInitDelay: The number of seconds of init delay for the timing thread to check the security file.

• server.security.checkSecurityInterval: The number of seconds of check interval for the timing thread to check the security file.

Step 2: Make sure security/security.yml exist in application root folder.

Step 3: Use user controller API to add user to security file security/security.yml.
Notice:

• The first user should be added manually. Password need to be encoded using bcrypt before saving to the yml file.For convenience, we provide CommonUtils to encode the password.

• No need to restart server after adding new user or update user info. Timing thread introduced in Step 1 will refresh the user list according to your settings.

Support Kafka Version Information

Currently, this rest api (master branch) supports Kafka 0.10.x brokers. The master branch is the most active branch. We’re going to get down to the work of supporting the Kafka 1.x version.

For 0.11.x, please checkout the branch 0.11.x by calling the command:

git clone -b 0.11.x https://github.com/gnuhpc/Kafka-zk-restapi.git

URI scheme

Host : localhost:8121
BasePath : /

You can access Swagger-UI by accessing http://127.0.0.1:8121/api

API LIST for 0.10

API LIST for 0.11.x

• kafka-controller : Kafka Api

• zookeeper-controller : Zookeeper Api

• collector-controller : JMX Metric Collector Api

• user-controller : User management Api

Data Model Definitions for 0.10

Data Model Definitions for 0.11.x

Version information

Version : 0.1.0

Contact information

Contact : gnuhpc
Contact Email : [email protected]
Github : https://github.com/gnuhpc