team-triage/triage

Triage is an open-source consumer proxy that solves head-of-line blocking for Kafka consumers

Triage

Triage is an open source Kafka consumer proxy service that solves head-of-line blocking while preventing data-loss through increased parallelism, commit tracking, and a dead-letter store.

Triage operates as an AWS Fargate service deployed to a users AWS account. It sits between your Kafka cluster and consumer applications and can be interacted with via a thin client library.

To read more about Triage, visit: https://team-triage.github.io/

How to Use Triage

Users can deploy Triage using the triage-cli command line tool, which is available via npm. It can be installed using the following command:

npm install -g triage-cli

For detailed instructions, see the README at: https://github.com/Team-Triage/triage-cli

To interact with Triage, consumer applications should use the triage-client-go library, found at https://github.com/Team-Triage/triage-client-go.

triage's Issues

Finalize infrastructure setup + diagram

Implement a Mac compiled Producer for testing

Currently only have a linux version

Implement HTTP authentication for consumer manager

gRPC Server Basics

Client requests a message through their gRPC endpoint.
We give them that message.

Client sends another request with the status code and an offset.
We send the offset and status to the Filter component.

We should be able to have multiple clients interact with the server.

Implement Wait Groups

Clean code

Remove comments etc.

Implement testing

Basic unit tests

Implement Reaper (component sends "nacked" raw messages to a DLQ

Confirm that CDK deploy works with multiple consumer instances.

A 3-partition topic on the Confluent Kafka cluster should have been connected to Triage on Fargate. Then, 3 consumer instances should be opened downstream to confirm that all 3 Triage tasks are working (1 for each consumer instance).

Delivery mechanism for messages

Message delivery from Sarama to Go channel

Fix OS exit signal problem in main.go

Deployment Research - look into initial setup and commands

Figure out what it takes to fully deploy Triage as a container using AWS Fargate.
Try to think about automation.

Securing gRPC connection

Create Reaper to handle sending messages to DLQ

Commit mechanism

Calculate commits to manually commit back to Kafka

Implement functionality for dispatch to kill senderRoutines

Complete individual repository 'tags'

Explore Interceptors to see if server can update the response before it's returned.

Add support for client library to return 'nack' reason

Our grpc client should expect a 'reason' field for failed messages, and supply that to the Acknowledgment struct for passing to Reaper and the eventually the DLQ

Fix concurrent map writes

Add support for restarting kafka consumer on max.poll.interval.ms exceeded error

We should probably implement some sort of notification here as well.

Switch to `serveMux` handler

Using http.handleFunc() creates a global variable that can create security concerns. Switch to serveMux() to strengthen security.

Update group.id to use topic name

group.id for kafka consumer should be dynamic (interpolated)

Message storage

Write messages to Redis container from Sarama client

Connection logic 2 - Invoke SendMessages goroutines (workers) for each connected consumer instance

Implement Consumer Manager

Connection logic 1 - connect downstream consumer instances with network addresses

Integrate DynamoDB with Reaper

Create filter logic to handle acknowledgments from dispatch

Explore 'go-get' functionality / how do we publish our modules online for developer use?

Code Spike - gRPC Dispatch and functionality

1. Create/persist "routes" to multiple consumer instances.
2. Get responses back and send them somewhere (eventually based on ack and nack)
3. If there is NO response, can I keep it somewhere and then send it?

Add support for environmental variables

Need env vars for:

Kafka Authentication
Topic Name
Dynamo Credentials
Authentication Token for triage-client-go

Add README details and add description to Triage Repo

Securing connection to the Kafka cluster

How do we authenticate with the Cluster?
What are the most common auth methods?
How do we integrate said methods with Triage?

What's the lightest/simplest HTTP server for Go?

To establish the initial connection from the consumer instance to Triage.

Understanding gRPC liveness - what happens if one end of the connection dies?