foobaar/kafka-offset-manager

Move Consumer offsets as you please

Motivation

• The idea behind the app is to allow you to manually set offsets in kafka.
• You can move to a custom offset or move to the beginning or end of a topic

Pre req

• kafka version of 0.10.1.0 or greater
• You need to make sure that all consumers that belong to a consumer group should be stopped before changing offsets.

Installation

./gradlew bootRun

Usage

Setting Custom Offsets

POST localhost:8080/custom-offset
POST BODY:
{
  "consumerGroupId" : "consumer1",
  "kafkaBroker" : "localhost:9092",
  "topic": "test",
  "partitionOffsets": [{"partition": "0", "offset": "29753"}, {"partition": "1", "offset": "30595"}]
}

Moving the offsets to the earliest available offset:

POST localhost:8080/boundary?position=start
POST BODY:
{
  "consumerGroupId" : "consumer1",
  "topic": "test",
  "kafkaBroker" : "localhost:9092"
}

Moving the offsets to the latest available offset:

POST localhost:8080/boundary?position=end
POST BODY:
{
  "consumerGroupId" : "consumer1",
  "topic": "test",
  "kafkaBroker" : "localhost:9092"
}

Moving the offsets to the earliest available offset for a given timestamp

POST localhost:8080/time-based-offset
POST BODY:
{
  "consumerGroupId" : "consumer1",
  "topic": "test",
  "kafkaBroker" : "localhost:9092",
  "timeStampInMillis": "23123123123"
}

Docker

The project contains a docker-compose.yml file, which configures containers for Kafka and ZooKeeper. See the following Github project for details on configuration: https://github.com/wurstmeister/kafka-docker

kafka-offset-manager's Issues

Update to newer version of Kafka Clients

0.10.1.0 is pretty old already

Add a UI?

I think it'd be a good way to enhance usability. You could have it let you input cluster info and give you back topic info maybe?

Seems like some more brainstorming could be done.

Add tests

You don't have anything other than the default test created by Spring Initializr

Convert to Kotlin

It's the wave of the future mang.

Docker Compose example

Might be nice to be able to setup a test environment easily.

Create Test harness to seed data in kafka and test functionality

Basically, we should be able to make changes to the code and then run a command to make sure that it didnt break: 1. the contract 2: the offset changing functionality

add ssl config

the kernel is consumer.commitSync(offsets),maybe ssl config can be add by the way.