This repo is a demonstration of a 3 node ELK stack:
- server: runs Elasticsearch, Logstash, and Kibana. Logstash pulls from Redis on the broker vm and from /var/log/messages
- broker: runs Logstash and Redis. No filtering is done here. Inputs are setup for Logstash Forwarder (lumberjack), syslog via udp, NXLog(tcp json_lines), and /var/log/messages
- client: runs Logstash Forwarder and ships logs to the broker vm
If you have the Vagrant plugin "Vagrant Cachier" installed this project will utilize it to save you a fair amount of downloading. If you don't have the plugin installed already just run the following:
$ vagrant plugin install vagrant-cachier
As a byproduct of the type of setup this is, more memory is required than with most Vagrant setups... server uses 2G and the others each use 1G for a total of 4G.
To get started all you need to do is run
$ vagrant up
Once all machines are up you can go to the following URL's:
- Elasticsearch: http://localhost:9200
- kopf plugin to see info about this Elasticsearch setup: http://localhost:9200/_plugin/kopf/
- Kibana: http://localhost:5601
You can easily generate a bunch of log entries by running yum -y upgrade
on the client.
Both the broker and the server also log to stdout. By running
$ tail -f /var/log/logstash/logstash.stdout
you can see logs in Ruby format like this one:
{
"message" => "Jul 20 22:31:06 localhost yum[3791]: Updated: dracut-kernel-004-356.el6_6.3.noarch",
"@version" => "1",
"@timestamp" => "2015-07-21T02:31:13.196Z",
"type" => "syslog",
"tags" => [
[0] "lumberjack",
[1] "redis"
],
"file" => "/var/log/messages",
"host" => "client.localdomain",
"offset" => "87573"
}