harryleesan/kafka-logstash-docker

Docker containers for Kafka, Zookeeper and Logstash using docker-compose

docker-compose for Kafka, Zookeeper and Logstash

This set up is merely a demonstration for pushing logs to Logstash which then pushes the logs to a Kafka topic.

This set up is based on kafka-docker using the single-broker configuration.

Pre-requisites

• Docker needs to be installed.
• docker-compose needs to be installed.

Usage

Ensure that the following ports are not in use:

• 2181
• 9092
• 10000

Run:

docker-compose up -d

That's it! You are all set!

To see it in action:

• Open up a terminal session and run:

  docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh --zookeeper
  zookeeper:2181 --topic logstash_logs

• Open up a second terminal session and run:

  curl -X POST -d @test.json http://localhost:10000 --header "Content-Type:application/json"

  Ensure that curl is installed.

• You should be able to see a concise HTTP request in the form of an json object coming through in the first terminal session.

Now you can stream logs to localhost:10000 and see it in Kafka.

Configuration

The configurations for Logstash is in the logstash_pipeline and logstash_settings folders.

You can modify the settings before running docker-compose up -d.

In this demonstration, a Http input plugin and a Kafka output plugin using the json codec are used.

kafka-logstash-docker's Issues

Loss of connection?

Hi, I would need to have logstash producing and consuming from kafka. For this I have two instances of logstash, first with input from beats and output to kafka and the second with input from kafka and output to my app. Thing is when I add the network from logstash #1 to your compose:

this line: "docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh --zookeeper zookeeper:2181 --topic logstash_logs" returns nothing. After undo this modification I also don't have any feedback from zookeeper when curling, only way to make this communication work again is by download your code again. Do you have any idea of what's wrong with my setup?

Many Thanks