cloudfoundry-community/kafka-boshrelease

BOSH release for kafka

This BOSH release and deployment manifest deploy a cluster of kafka.

Usage

This repository includes base manifests and operator files. They can be used for initial deployments and subsequently used for updating your deployments:

export BOSH_ENVIRONMENT=<bosh-alias>
export BOSH_DEPLOYMENT=kafka
git clone https://github.com/cloudfoundry-community/kafka-boshrelease.git
bosh deploy kafka-boshrelease/manifests/kafka.yml

bosh run-errand sanity-test

If your BOSH does not have Credhub/Config Server, then remember --vars-store to allow generation of passwords and certificates.

Topics

You can pre-define some simple topics using an operator script ./manifests/operators/simple-topics.sh. Th

bosh deploy kafka-boshrelease/manifests/kafka.yml \
  -o <(kafka-boshrelease/manifests/operators/simple-topics.sh test1 test2)

Enable SASL/SCRAM and TLS

You can enable SASL/SCRAM using ./manifests/operators/add-jaas.yml. SASL_PLAINTEXT and SASL_TLS are supported as a security protocol.

bosh deploy kafka-boshrelease/manifests/kafka.yml \
  -o kafka-boshrelease/manifests/operators/enable-jaas.yml

You can find admin's password by credhub get -n /(director name)/kafka/jaas-admin-password.

If you want to use SASL_TLS, use ./manifests/operators/add-tls.yml as well.

bosh deploy kafka-boshrelease/manifests/kafka.yml \
  -o kafka-boshrelease/manifests/operators/enable-jaas.yml \
  -o kafka-boshrelease/manifests/operators/enable-tls.yml \
  -v kafka-external-host=${your-kafka-hostname-or-static-ip} \

kafka-boshrelease/manifests/operators/enable-tls.yml is supposed to be used for single kafka instance group. To scale out the kafka cluster, change advertised.listener property and kafka-tls variable.

You can use Let's Encrypt as follows:

bosh deploy kafka-boshrelease/manifests/kafka.yml \
  -o kafka-boshrelease/manifests/operators/enable-jaas.yml \
  -o kafka-boshrelease/manifests/operators/enable-tls.yml \
  --var-file kafka-tls.certificate=/etc/letsencrypt/live/your-kafka.example.com/fullchain.pem \
  --var-file kafka-tls.private_key=/etc/letsencrypt/live/your-kafka.example.com/privkey.pem \
  --var-file kafka-ca.certificate=<(curl https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt) \
  -v kafka-external-host=your-kafka.example.com

Kafka Manager

The Yahoo Kafka Manager UI is installed on each Kafka node. You can access it via port 8080. To access via http://localhost:8080, open a tunnel:

bosh ssh kafka-manager/0 -- -L 8080:127.0.0.1:8080

Kafka Manager requires basic auth credentials. The default username is admin, and the password is the ((kafka-manager-password)) value from either Credhub/Config Server, or your --vars-store creds.yml file.

Update

When new versions of kafka-boshrelease are released the manifests/kafka.yml file will be updated. This means you can easily git pull and bosh deploy to upgrade.

export BOSH_ENVIRONMENT=<bosh-alias>
export BOSH_DEPLOYMENT=kafka
cd kafka-boshrelease
git pull
cd -
bosh deploy kafka-boshrelease/manifests/kafka.yml

Development

To iterate on this BOSH release, use the create.yml manifest when you deploy:

bosh deploy manifests/kafka.yml -o manifests/operators/create.yml

kafka-boshrelease's Issues

Support SASL SCRAM for authentication

As a managed kafka service, it would be necessary to give a client username and password.
How does supporting SASL SCRAM for authentication sounds?
CloudKarafka supports that.
https://www.cloudkarafka.com/docs-kafkacat.html
https://docs.confluent.io/current/kafka/authentication_sasl_scram.html

kafka listing topics sometimes fails

Hi Team,
I am using SSL kafka service with my cloud foundry. I migrated from bosh v1 to bosh v2. I use this kafka deployment with kafka service broker in CF.
After the migration one of my CF application started having issue in listing the topics.
It is a random issue sometimes it is able to list the topics and sometimes it is not able to list the topics.
There are no error logs recorded in kafka in server, controller or state-change logs.
I tried recreation of topic and also repushed the app. But after few days the issue returned.
The app which has issue in listing the topic is a consumer. There is one more app which also lists the topics present in kafka , but this does not have any issue in listing the topics. [This app is a producer].
We have 3 kafka deployments for each environment as dev test and prod and we are having this same issue in all three environments.
Can you please help me know why sometimes the app is not able to list the topics and which component is causing this issue?

Balancing Replicas Across AZs

see http://kafka.apache.org/documentation/#basic_ops_racks

[discuss] There is a difference between the official kafka and the default value

I think difference between the official kafka default value.
i.e

properties	bosh-release value	official kafka value
delete.topic.enable	false	true
auto.create.topics.enable	false	true

https://kafka.apache.org/documentation/

(I overlooked auto.create.topics.enable official kafka default value...)

Should I fix it?
Or do you want to keep it this way?