Loss of connection?

[harryleesan]

Loss of connection?

[harryleesan]

Hi @jmcarvalho, I have created an empty external network logstash_default and attached it to the docker-compose as you have mentioned. I then execute the curl command (curl -X POST -d @test.json http://localhost:10000 --header "Content-Type:application/json") and I was able to view the results in Kafka. There seems to be no problem on my side.

Please can you tell me more about the setup for the logstash_default network (what's in it and how did you set up the 2 logstashes).

[jmcarvalho]

Logstash_default was created by a compose with ELK stack. I configured this logstash output to Kafka. In order to connect the containers I did the modification in your compose but now I am not able to view messages in Kafka. Could it be port conflict between elk logstash and yours?

[harryleesan]

It is possible there could be some conflicts. Would you mind showing me the contents of your ELK stack docker-compose?

If I understand correctly, you would like the Kafka (in my compose) to ingest the messages coming from logstash (in your ELK compose)?

[jmcarvalho]

Thanks for the prompt reply.

[harryleesan]

To connect different docker-compose networks, I tried the below and it works. I have removed the contents of your compose file and replaced it with the logstash service in my compose file since I don't have your configurations.

I have also created one single network to unify them. You should run Kafka (my compose file) first before running Logstash (your compose file). The containers are linked through the external_links definition. The Logstash version that I am running is also newer: 5.6.3 (The container exits when I tried to curl it when using version 5.1.1).

My docker-compose.yml (with logstash removed):

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper:latest
    ports:
      - "2181:2181"
    container_name: zookeeper
  kafka:
    image: wurstmeister/kafka:0.11.0.1
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: docker.for.mac.localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "logstash_logs:1:1"
    links:
      - zookeeper
    depends_on:
      - zookeeper
    container_name: kafka
  # logstash:
  #   image: docker.elastic.co/logstash/logstash:5.6.3
  #   ports:
  #     - "10000:10000"
  #   links:
  #     - kafka
  #   volumes:
  #     - ./logstash_settings/:/usr/share/logstash/config/
  #     - ./logstash_pipeline/:/usr/share/logstash/pipeline/
  #   container_name: logstash
networks:
  default:
    external:
      name: logstash_kafka_default

Your docker-compose.yml (replaced with the logstash from my docker-compose.yml):

version: "2"
services:
  elasticsearch:
    image: elasticsearch:5.1.1
    ports:
      - "9200:9200"
    container_name: elasticsearch
    # volumes:
    #   - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
  kibana:
    image: kibana:5.1.1
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    container_name: kibana
  # logstash:
  #   image: logstash:5.1.1
  #   command: logstash -f /etc/logstash/conf.d/logstash.conf --config.reload.automatic
  #   ports:
  #     - 5044:5044
  #   depends_on:
  #     - elasticsearch
  #   container_name: logstash
  logstash:
    image: docker.elastic.co/logstash/logstash:5.6.3
    ports:
      - "10000:10000"
    depends_on:
      - elasticsearch
    external_links:
      - kafka
    volumes:
      - ./logstash_settings/:/usr/share/logstash/config/
      - ./logstash_pipeline/:/usr/share/logstash/pipeline/
    container_name: logstash
networks:
  default:
    external:
      name: logstash_kafka_default

[jmcarvalho]

Thing is, I have tried just running yours to find the problem and curl doesn't work after I make the network modification even if I replace it to the initial one so I think it may not be directly related with logstash but with containers communication. Only thing I wanted from this point was testing the communication which I am not able to do as zookeeper doesn't retrieve nothing .
After modifying network even by creating a producer I cant see nothing in zookeeper consumer:

So it's definitely not caused by mine or yours logstash, the problem is related with containers I just can't find what it is.

[jmcarvalho]

Don't know yet why zookeeper stops working when I modify the network. However I finally managed to get the entire flow working. Thanks for the help anyway. In addition as your familiar with this could you please let me know if it's possible to ignore the identifier of the kafka message when filtering it with logstash? After adding kafka into my arch I was filtering messages just by doing this:

However after adding kafka my message looks like this:

How can I bypass the first two fields and use only the json included in the message?

Thanks for the help

[harryleesan]

The http input plugin by default tracks the entire http request from the client. If you don't do any filtering and just pass the output straight to Kafka, the entire request will be passed through as is.

I think if you use the below (haven't test it myself) from Kafka Output Plugin, it should filter out everything besides the message portion.

    output {
      kafka {
        codec => plain {
           format => "%{message}"
        }
        topic_id => "mytopic"
      }
    }

[jmcarvalho]

Many thanks @harryleesan I finally have all the things working together!

[jmcarvalho]

Hi @harryleesan! Are you able to produce outside the container with that configuration?
I mean, using docker.for.mac.localhost as the KAFKA_ADVERTISED_HOST_NAME doesn't allow me to produce outside the container using localhost as "bootstrap.servers" configuration.

[harryleesan]

@jmcarvalho Can you please clarify what you mean by 'produce outside the container'?

When you use docker-compose, it actually creates a separate network overlay to your other docker containers. To connect this network overlay so that other docker containers may communicate with the containers in docker-compose you will have to specify an external network that the other containers are using. e.g add the following to the end of your docker-compose.yml:

networks:
  default:
    external:
      name: logstash_kafka_default

[jmcarvalho]

I can estabilish connection through containers by setting the same network. My problem is when I want to produce from my host machine to Kafka inside the container