Giter Site home page Giter Site logo

docker-elk's Introduction

Elasticsearch. Logstash. Kibana.

Creating an ELK stack could not be easier.

Important:

  • The master branch (image: willdurand/elk:latest or willdurand/elk:kibana-4.1.2) currently contains an image with Kibana 4.1.2, Elasticsearch 1.x, and Logstash 1.x;
  • Branch kibana-4.4.2 (image: willdurand/elk:kibana-4.4.2) provides Kibana 4.4.2, Elasticsearch 2.2, and Logstash 2.2;
  • Branch kibana-4.5.4 (image: willdurand/elk:kibana-4.5.4) provides Kibana 4.5.4, Elasticsearch 2.3, and Logstash 2.3;
  • Branch kibana-5.x (image: willdurand/elk:kibana-5.x) provides Kibana 5.x, Elasticsearch 5.x, and Logstash 5.x.

Quick Start

$ docker run -p 8080:80 \
    -v /path/to/your/logstash/config:/etc/logstash \
    willdurand/elk

Then, browse: http://localhost:8080 (replace localhost with your public IP address).

Your logstash configuration directory MUST contain at least one logstash configuration file. If several files are found in the configuration directory, logstash will use all of them, concatenated in lexicographical order, as the configuration.

Compose Configuration

elk:
    image: willdurand/elk
    ports:
        - "8080:80"
    volumes:
        - /path/to/your/logstash/config:/etc/logstash

Data

Elasticsearch data are located in the /data folder. It is probably a good idea to mount a volume in order to preserve data integrity. You can create a data only container:

$ docker run -d -v /data --name dataelk busybox

Then, use it:

$ docker run -p 8080:80 \
    -v /path/to/your/logstash/config:/etc/logstash \
    --volumes-from dataelk \
    willdurand/elk

If you want to rely on the logstash agent for processing files, you have to mount volumes as well, but you should rather only send logs to this container.

Compose Configuration

elk:
    image: willdurand/elk
    ports:
        - "8080:80"
    volumes:
        - /path/to/your/logstash/config:/etc/logstash
    volumes_from:
        - dataelk

dataelk:
    image: busybox
    volumes:
        - /data

Real Life Use Case

You can use this image to run an ELK stack that receives logs from your production servers, using Logstash Forwarder:

elk:
    image: willdurand/elk
    ports:
        - "80:80"
        - "XX.XX.XX.XX:5043:5043"
    volumes:
        - /path/to/your/ssl/files:/etc/ssl
        - /path/to/your/logstash/config:/etc/logstash
    volumes_from:
        - dataelk

dataelk:
    image: busybox
    volumes:
        - /data

Note that the 5043 port is binded to a private IP address in this case, which is recommended. Kibana is publicly available though.

Your logstash configuration SHOULD contain the following input definition:

input {
  lumberjack {
    port => 5043
    ssl_certificate => "/etc/ssl/logstash-forwarder.crt"
    ssl_key => "/etc/ssl/logstash-forwarder.key"
  }
}

Extend It

One of the Docker best practices is to avoid mapping a host folder to a container volume. Instead of specifying a volume, it is recommended to use this image as base image and configure your own image.

docker-elk's People

Contributors

defshift avatar k-phoen avatar krichprollsch avatar mhor avatar ovski4 avatar seekayel avatar willdurand avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

docker-elk's Issues

ElasticSearch not running on 9200 - tag 4.5.4

Given I run 'docker-compose up -d'
When I open kibana on localhost:8080
Then I see error 'Unable to connect to Elasticsearch at http://localhost:9200 .'

I have tried to curl localhost:9200 after docker-compose exec elk bash. But it not responding.
Anything I am missing? Going to see if I can find the yml config for elastic

Sample docker compose

elk:
    image: willdurand/elk:kibana-4.5.4
    ports:
        - "8080:80"
    volumes:
        - ./config:/etc/logstash
   volumes_from:
       - dataelk

dataelk:
    image: busybox
    volumes:
        - /data

Include /opt/logstash/bin in default PATH

This is a nice-to-have that the official library/logstash image has but your elk image does not. If I "docker run" this image with /bin/bash command for some interactive experimentation, the default root shell's PATH does not include /opt/logstash/bin so just typing logstash results in command not found error. In the library/logstash this is set up for you nicely.

"Real use case" description pretty confusing: when providing your own logstash.conf, do you have to redefine output section?

Your logstash configuration SHOULD contain the following input definition:

Does the provided config version complement the default one? Or do you have to include the output section in my custom logstash.conf as well:

output {
  elasticsearch {
    host => elasticsearch
  }
}

More generally, it would have been helpful if the README.md provided complete instructions to one complete example of this container being used.

Logstash does not find the conf file

Hi.

First, thanks for this github repo. It's really cool to be able to get started quickly with all that stuff without making some complicated networks and multiple containers with docker.

I have some troubles tho. I pulled 5.X docker images, and logstash refuses to start due to the config file he can't find (whereas the conf file is here in /etc/logstash) :

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /etc/logstash/log4j2.properties. Using default config which logs to console
ERROR: No configuration file was specified. Perhaps you forgot to provide the '-f yourlogstash.conf' flag?
usage:
  bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
  bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
  bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash --help
2017-05-07 00:32:19,592 INFO exited: logstash (exit status 1; not expected)

My command line :

docker run --name elk -d -p 4242:80 -v E:/docker/logstash:/etc/logstash willdurand/elk:kibana-5.x

The config file :

input {
  lumberjack {
    port => 5043
  }
}
output {
  elasticsearch {
    host => "127.0.0.1"
    cluster => "logstash"
    embedded => false
  }
}

I can see the config file by making a ls /etc/logstash into the container. Even a chmod 777 on the file did not change anything. Still no way to make logstash working. It's like he can't see the file while I'm seeing it on the console (attached to the container as root).

Am I missing something ? Sorry for the inconvenience if this is the case, but I really can't find why anywhere.

Thanks.

filebeat not starting when using an external ELK

The configuration script leaves a space between the ip host and port in /etc/filebeat/filebeat.yml

output:
logstash:
enabled: true
hosts:
- XX.XX.XX.XX: 5044

#Correct without space
output:
logstash:
enabled: true
hosts:
- XX.XX.XX.XX:5044

line 5: cannot unmarshal !!map into string. Exiting.
Loading config file error: YAML config parsing failed on /etc/filebeat/filebeat.yml: yaml: unmarshal errors:
line 5: cannot unmarshal !!map into string. Exiting.

Regards

Logstash configuration file

What is the minimum configuration I need to use this image?
I'm trying to send logs to logstash by using a custom winston transport (https://github.com/jaakkos/winston-logstash).

In the README I read "Your logstash configuration directory MUST contain at least one logstash configuration file" but by configuration file you mean the one in which I specify input, output, filter?

log stash continually restarting

When running the elk container, I see a continuous stream of the following:

elk_1 | 2015-05-20 19:45:05,570 INFO spawned: 'logstash' with pid 140
elk_1 | 2015-05-20 19:45:06,572 INFO success: logstash entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
elk_1 | 2015-05-20 19:45:17,404 INFO exited: logstash (exit status 1; not expected)
elk_1 | 2015-05-20 19:45:18,409 INFO spawned: 'logstash' with pid 172

Its seems that continuously restarts. I am not quite sure of the cause, and the logs are not tremendously informative so far.

large startup time

When I start the container I notice the log messages of the processes being started in the elk container. However they seem to take a while to actually start processing. E.g. logstash misses many of the initial log messsages produced by other containers. The Kibana site is also available only after some time. Is there some initialization processing going on before data can be processed?

Cannot initialize Kibana

I stuck at the first step of configuring Kibana, which is Configure an index pattern.

I checked the two boxes Index contains time-based events and Use event times to create index names. I used the default index name or pattern [logstash-]YYYY.MM.DD. But Kibana says Unable to fetch mapping. Do you have indices matching the pattern?.

I'm not able to continue.

elasticsearch unexpectedly terminated by SIGKILL

When I try to run a container based off this image, I get the following:

vagrant@dockerhello:~$ docker run -p 8200:80 -p 5000:5000 -p 9200:9200 elk
2015-09-16 17:52:12,185 CRIT Supervisor running as root (no user in config file)
2015-09-16 17:52:12,185 WARN Included extra file "/etc/supervisor/conf.d/kibana.conf" during parsing
2015-09-16 17:52:12,185 WARN Included extra file "/etc/supervisor/conf.d/logstash.conf" during parsing
2015-09-16 17:52:12,186 WARN Included extra file "/etc/supervisor/conf.d/elasticsearch.conf" during parsing
2015-09-16 17:52:12,199 INFO RPC interface 'supervisor' initialized
2015-09-16 17:52:12,200 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2015-09-16 17:52:12,200 INFO supervisord started with pid 1
2015-09-16 17:52:13,204 INFO spawned: 'elasticsearch' with pid 9
2015-09-16 17:52:13,205 INFO spawned: 'logstash' with pid 10
2015-09-16 17:52:13,209 INFO spawned: 'kibana' with pid 11
2015-09-16 17:52:14,367 INFO success: elasticsearch entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-09-16 17:52:14,368 INFO success: logstash entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-09-16 17:52:14,369 INFO success: kibana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-09-16 17:52:29,389 INFO exited: elasticsearch (terminated by SIGKILL; not expected)
2015-09-16 17:52:29,425 INFO spawned: 'elasticsearch' with pid 63
2015-09-16 17:52:29,848 INFO exited: elasticsearch (exit status 1; not expected)
2015-09-16 17:52:30,855 INFO spawned: 'elasticsearch' with pid 72
2015-09-16 17:52:30,983 INFO exited: elasticsearch (exit status 1; not expected)
2015-09-16 17:52:33,873 INFO spawned: 'elasticsearch' with pid 81
2015-09-16 17:52:33,974 INFO exited: elasticsearch (exit status 1; not expected)
2015-09-16 17:52:37,879 INFO spawned: 'elasticsearch' with pid 92
2015-09-16 17:52:37,983 INFO exited: elasticsearch (exit status 1; not expected)
2015-09-16 17:52:38,984 INFO gave up: elasticsearch entered FATAL state, too many start retries too quickly
# I killed the process here because elasticsearch had died.
^C2015-09-16 17:52:43,958 WARN received SIGINT indicating exit request
2015-09-16 17:52:43,963 INFO waiting for logstash, kibana to die
2015-09-16 17:52:43,967 INFO stopped: kibana (exit status 143)
2015-09-16 17:52:44,012 INFO stopped: logstash (exit status 0)

Note that the "elk" image is an image I made with this Dockerfile:

FROM willdurand/elk
COPY logstash-logspout.conf /etc/logstash/logstash-logspout.conf

Any idea why I might be seeing this?

gpgkeys: HTTP fetch error 7: couldn't connect: Connection timed out

Hi,

I've got the following error when running 'docker build .' on Step 5.

Step 5 : RUN apt-key adv --keyserver pool.sks-keyservers.net --recv-keys 46095ACC8548582C1A2699A9D27D666CD88E42B4 && if ! grep "elasticsearch" /etc/apt/sources.list; then echo "deb http://packages.elasticsearch.org/elasticsearch/1.4/debian stable main" >> /etc/apt/sources.list;fi && if ! grep "logstash" /etc/apt/sources.list; then echo "deb http://packages.elasticsearch.org/logstash/1.4/debian stable main" >> /etc/apt/sources.list;fi && apt-get update
---> Running in 5a6cff0bfe61
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.eNDWVDUFrg --no-auto-check-trustdb --trust-model always --primary-keyring /etc/apt/trusted.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-security-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-squeeze-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-squeeze-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-wheezy-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-wheezy-stable.gpg --keyserver pool.sks-keyservers.net --recv-keys 46095ACC8548582C1A2699A9D27D666CD88E42B4
gpg: requesting key D88E42B4 from hkp server pool.sks-keyservers.net
?: pool.sks-keyservers.net: Host not found
gpgkeys: HTTP fetch error 7: couldn't connect: Connection timed out
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
The command '/bin/sh -c apt-key adv --keyserver pool.sks-keyservers.net --recv-keys 46095ACC8548582C1A2699A9D27D666CD88E42B4 && if ! grep "elasticsearch" /etc/apt/sources.list; then echo "deb http://packages.elasticsearch.org/elasticsearch/1.4/debian stable main" >> /etc/apt/sources.list;fi && if ! grep "logstash" /etc/apt/sources.list; then echo "deb http://packages.elasticsearch.org/logstash/1.4/debian stable main" >> /etc/apt/sources.list;fi && apt-get update' returned a non-zero code: 2

And then the build stop... What should I do?

Your help will grandly appreciated.

Regards,

Rocky

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.