aduis/golang-kafka-templates

Production ready GoLang client templates for working with Apache Kafka

Golang Kafka Microservice Templates

A group of GoLang example templates for writing Apache Kafka production ready applications. The templates were created to be used as a starting point for microservices that utilize Kafka for Event Sourcing as an eventstore. An eventstore is the backbone of an event-sourced mircroservice architecture. Utilizing best practices in GoLang for producers and consumers is therefore critical to a successful project.

Each template does not have any domain specific logic. In some cases wordcount is implemented just to show were to place logic and how to do it in a thread safe fashion. The goal is that new developers can leverage the boiler plate code examples to add additional functionality to an application.

Example Templates

• consumer-wordcount is a minimal implementation of a Kafka consumer that provides:

  • Graceful shutdown to avoid data loss and unexpected behavior
  • Configuration via environment variables
  • Syslog logging of information and issues
  • Implements a word count logic for a basic accumulator of messages

• consumer-cluster-wordcount is a minimal implementation of a

  • Kafka Consumer groups allowing for fault tolerance of consumer clients and load-balancing.

• producer-wordcount-api is a minimal implementation of a Kafka producer for sending http post vars into a a Kafka topic.

  • Graceful shutdown to avoid data loss and unexpected behavior
  • Configuration via environment variables
  • Syslog logging of information and issues

Usage

It assumed that you have an understanding of Apache Kafka and have a version of Kafka running that you can connect to for development. The Devoxx video Introduction to Apache Kafka by James Ward is a very concise way to get up to speed with terminology quickly.

The Kafka quickstart tutorial can assist you in running a local copy. All examples require Kafka v0.9+ to utilize Kafka based client tracking rather than zookeeper.

On macOS with Homebrew installed the following starts a local copy of Kafka

localhost:~ me$ Brew install Kafka
localhost:~ me$ zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties

The code requires Sarama an MIT-licensed Go client library for Apache Kafka. We utilize this library because it is a pure go implementaion of the wire transfer protocol of Kafka with no C dependencies. It also takes advantage of consumer groups which ensures that for replicated consumers duplicated execution of published messages doesn't occur.

localhost:~ me$ go get github.com/Shopify/sarama

Clustered consumer examples utilize Sarama-Cluster a cluster Sarama extension enabling users to consume topics across balanced nodes. It's important to read the Client-side Assignment Proposal to understand use cases for a clustered consumer.

localhost:~ me$ go get github.com/bsm/sarama-cluster

Please do not fork the repository unless you are submitting a pull request to update the template. For your own usage please mirror the repo.

Design Assumptions

Service configuration should be done with functional options.

• Functional options for friendly APIs a dotGo presentation by Dave Cheney. Watch the presentation
• Self-referential functions and the design of options by Rob Pike

These templates follow The Twelve Factors a popular and widely used methodology for building web applications.

All logs are sent to syslog

localhost:~ me$ tail -f /var/log/system.log | grep "serviceName"

Thank you

This work is built on top of the incredible blog posts from Oscar Oranagwa

• Building Scalable Applications Using Event Sourcing and CQRS
• Implementing CQRS using Kafka and Sarama Library in Golang