docker-kafka-mirror-maker

An HDP 2.5.3 based Docker image for Kafka Mirror Maker. This allows configuration and deployment of Kafka Mirror-Maker in Dockerized environments like Kubernetes/Openshift or AWS ECS.

Build

This image is available from Docker hub however, if you would like to build it yourself here are the steps:

git clone https://github.com/srotya/docker-kafka-mirror-maker.git
cd docker-kafka-mirror-maker
docker build -t mirror-maker:latest .

Note: Docker is expected to be installed where you run the build

Environment Variables

Variable Name	Description	Defaults
DESTINATION	bootstrap.servers for the Destination Kafka Cluster	localhost:6667
SOURCE	bootstrap.servers for the Source Kafka Cluster	localhost:6667
WHITELIST	Topics to mirror	*
SECURITY	If kerberos is enabled, valid options: PLAINTEXT or SASL_PLAINTEXT	PLAINTEXT
GROUPID	Consumer group id for Kafka consumer	_mirror_maker
PRINCIPAL	Kerberos security principal name	kafka/[email protected]
KEYTAB_FILENAME	File name for Kerberos Keytab	mirror.keytab

Volumes

Image requires that the keytab file and krb5.conf be mounted if Kerberos is used when deploying this container.

Keytab files are expected in /etc/security/keytabs location

Krb5.conf file is expected in /etc/krb5.conf location

Non-Kerberized Usage

docker run -it -e DESTINATION=xxx.xxx.com:9092 -e SOURCE=xxx.xxx.com:9092 -e WHITELIST=<TOPIC NAME> mirror-maker:latest

Kerberized Usage

docker run -it -e DESTINATION=xxx.xxx.com:6667 -e SECURITY=SASL_PLAINTEXT -e SOURCE=xxx.xxx.com:6667 -e WHITELIST=<TOPIC NAME> -e KEYTAB_FILENAME=kafka.service.keytab -e PRINCIPAL=kafka/[email protected] -v /etc/security/keytabs/:/etc/security/keytabs/ -v /etc/krb5.conf:/etc/krb5.conf mirror-maker:latest

Note: the image automatically applies Kerberos runtime configuration for Mirror Maker if SASL_PLAINTEXT security is configured

Kubernetes Usage

To deploy this image in Kubernetes here's the template.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: mirror-maker
spec:
  replicas: 3
  template:
    metadata:
      labels:
        name: mirror-maker
    spec:
      containers:
      - name: mirror-maker
        image: ambuds/mirror-maker
        imagePullPolicy: Always
        env:
        - name: "WHITELIST"
          value: "*"
        - name: "DESTINATION"
          value: "localhost:9092"
        - name: "SOURCE"
          value: "localhost:9092"

License

Apache 2.0

srotya / docker-kafka-mirror-maker Goto Github PK