xeraa/vagrant-elastic-stack

Giving the Elastic Stack a try in Vagrant

Elastic Stack in a Box

This repository will install the Elastic Stack (Elasticsearch, Logstash, Kibana, and Beats) and optionally start a trial of commercial features. You can either start from scratch and configure everything with Vagrant and Ansible or you can download the final OVA image.

Features

• Filebeat system, auditd, logstash, mongodb, nginx, osquery, and redis modules
• Filebeat collecting Kibana JSON logs from /var/log/kibana/kibana.log
• Auditbeat file_integrity module on /home/vagrant/ directory and auditd module
• Heartbeat pinging nginx every 10s
• Metricbeat system, docker, elasticsearch, kibana, logstash, mongodb, nginx and redis modules
• Packetbeat sending its data via Redis + Logstash, monitoring flows, ICMP, DNS, HTTP (nginx and Kibana), Redis, and MongoDB (generate traffic with $ mongo /elastic-stack/mongodb.js)
• The pattern for nginx is already prepared in /opt/logstash/patterns/ and you can collect /var/log/nginx/access.log with Filebeat and add a filter in Logstash with the pattern as an exercise

Vagrant and Ansible

Do a simple vagrant up by using Vagrant's Ansible provisioner. All you need is a working Vagrant installation (2.2.4+ but the latest version is always recommended), a provider (tested with the latest VirtualBox version), and 3GB of RAM.

With the Ansible playbooks in the /elastic-stack/ folder you can configure the whole system step by step. Just run them in the given order inside the Vagrant box:

> vagrant ssh
$ cd /elastic-stack/
$ ansible-playbook 1_configure-elasticsearch.yml
$ ansible-playbook 2_configure-kibana.yml
$ ansible-playbook 3_configure-logstash.yml
$ ansible-playbook 4_configure-auditbeat.yml
$ ansible-playbook 4_configure-filebeat.yml
$ ansible-playbook 4_configure-heartbeat.yml
$ ansible-playbook 4_configure-metricbeat.yml
$ ansible-playbook 4_configure-packetbeat.yml
$ ansible-playbook 5_configure-dashboards.yml

Or if you are in a hurry, run all playbooks with $ /elastic-stack/all.sh at once.

OVA Image

If Vagrant and Ansible sound too complicated, there is also the final result: An OVA image, which you can import directly into VirtualBox:

• Download the image from https://s3.eu-central-1.amazonaws.com/xeraa/public/elastic-stack.ova.
• Load the OVA file into VirtualBox and make sure you have 3GB of RAM available for it: File -> Import Appliance... -> Select the file and start it
• Connect to the instance with the credentials vagrant and vagrant in the VirtualBox window.
• Or use SSH with the same credentials:
  • Windows: Use http://www.putty.org and connect to [email protected] on port 2222.
  • Mac and Linux: $ ssh [email protected] -p 2222 -o PreferredAuthentications=password

Kibana

Access Kibana at https://127.0.0.1:5601.

Test Data

You can use /opt/injector.jar to generate test data in the person index. To generate 100,000 documents in batches of 1,000 run the following command:

$ java -jar /opt/injector.jar 100000 1000

Logstash Demo

You can play around with a Logstash example by calling $ sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /elastic-stack/raffle/raffle.conf (it can take some time) and you will find the result in the raffle index.

vagrant-elastic-stack's Issues

Kibana doesn't work

Hello,
I wanted to try the OVA Image, but I can't access kibana.

• I download the OVA
• Put in VirtualBox
• Start it
• And i wrote:
  curl https://127.0.0.1:5601

I got:

Running on Windows

I had a few issues running this on Windows so here is what I had to do if it helps anyone:

There is a bug in vagrant 1.8.3 for ansible on Windows which causes vagrant to prepend a drive letter to the playbook location. I had to install:

vagrant plugin install vagrant-guest_ansible

and change the Vagrantfile to:

provisioner = Vagrant::Util::Platform.windows?:guest_ansible : "ansible_local"

config.vm.provision provisioner do |ansible|
ansible.playbook = "0_install.yml"

end

Once the machine was provisioned I had some errors installing ansible:

Collecting cffi>=1.4.1 (from cryptography>=1.1->paramiko->ansible)
Downloading cffi-1.6.0.tar.gz (397kB)
Requirement already satisfied (use --upgrade to upgrade): pycparser in /usr/local/lib/python2.7/dist-packages (from cffi>=1.4.1->cryptography>=1.1->paramiko->ansible)
Installing collected packages: cffi, cryptography, paramiko, MarkupSafe, jinja2, ansible
Running setup.py install for cffi: started
Running setup.py install for cffi: finished with status 'error'
Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-BmlYsM/cffi/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-n00gG6-record/inst
gle-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-2.7/cffi
copying cffi/cparser.py -> build/lib.linux-x86_64-2.7/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-2.7/cffi
copying cffi/lock.py -> build/lib.linux-x86_64-2.7/cffi
copying cffi/api.py -> build/lib.linux-x86_64-2.7/cffi
copying cffi/gc_weakref.py -> build/lib.linux-x86_64-2.7/cffi
copying cffi/commontypes.py -> build/lib.linux-x86_64-2.7/cffi
copying cffi/ffiplatform.py -> build/lib.linux-x86_64-2.7/cffi
copying cffi/verifier.py -> build/lib.linux-x86_64-2.7/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-2.7/cffi
copying cffi/vengine_gen.py -> build/lib.linux-x86_64-2.7/cffi
copying cffi/recompiler.py -> build/lib.linux-x86_64-2.7/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-2.7/cffi
copying cffi/init.py -> build/lib.linux-x86_64-2.7/cffi
copying cffi/model.py -> build/lib.linux-x86_64-2.7/cffi
copying cffi/_cffi_include.h -> build/lib.linux-x86_64-2.7/cffi
copying cffi/parse_c_type.h -> build/lib.linux-x86_64-2.7/cffi
copying cffi/_embedding.h -> build/lib.linux-x86_64-2.7/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/c
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DUSE__THREAD -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
#include <ffi.h>
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

so had to run:

sudo apt-get update sudo apt-get install libffi-dev g++ libssl-dev

then re run
vagrant provision

Running on Windows

I had an issue running on Windows (Vagrant 1.8.5, Virtualbox 5.1.4)

λ vagrant up
Bringing machine 'ubuntu' up with 'virtualbox' provider...
==> ubuntu: Checking if box 'ubuntu/trusty32' is up to date...
==> ubuntu: Clearing any previously set forwarded ports...
"rsync" could not be found on your PATH. Make sure that rsync
is properly installed on your system and available on the PATH.

Change in Vagrantfile from
ubuntu.vm.synced_folder "elastic-stack/", "/elastic-stack/", type: "rsync"
to
ubuntu.vm.synced_folder "elastic-stack/", "/elastic-stack/", type: "virtualbox"

solved the problem

Access Error While Downloading File: Access Denied

Hi, I can't download this link
https://s3.eu-central-1.amazonaws.com/xeraa/public/elastic-stack.ova.

gives me this error:

This XML file does not appear to have any style information associated with it. The document tree is shown below.

AccessDenied
Access Denied
SFD6M13BCXJVVPXV
YA8C0/AEebk7KqI8FlZIrj/CBJprGfmX3HPDd3CIN3JVXDKDdCZFpA/rpF/9bz9YxmWlH5M4UmA=

vagrant up won't complete

Hello,

I got an error while doing vagrant up and it's pretty mysterious for me.

TASK [Add osquery's server repository] *****************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: apt.cache.FetchFailedException: E:The repository 'https://osquery-packages.s3.amazonaws.com/bionic bionic Release' does not have a Release file.
fatal: [ubuntu]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n File "/tmp/ansible_Gqglog/ansible_module_apt_repository.py", line 551, in \n main()\n File "/tmp/ansible_Gqglog/ansible_module_apt_repository.py", line 543, in main\n cache.update()\n File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 543, in update\n raise FetchFailedException(e)\napt.cache.FetchFailedException: E:The repository 'https://osquery-packages.s3.amazonaws.com/bionic bionic Release' does not have a Release file.\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}
to retry, use: --limit @/elastic-stack/0_install.retry

PLAY RECAP *********************************************************************
ubuntu : ok=20 changed=13 unreachable=0 failed=1

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

Then I tried again, but I was suggested to run vagrant provision this time. I did so, only to have the same error in a more brief way,

TASK [Install the JRE] *********************************************************
fatal: [ubuntu]: FAILED! => {"changed": false, "msg": "Failed to update apt cache: E:The repository 'https://osquery-packages.s3.amazonaws.com/bionic bionic Release' does not have a Release file."}
to retry, use: --limit @/elastic-stack/0_install.retry

PLAY RECAP *********************************************************************
ubuntu : ok=8 changed=1 unreachable=0 failed=1

Sorry for posting before doing any research, I wanted to check if it has a quick fix since the workshop is tomorrow.

Thanks!

Kibana prompt to enter password

In Mac, after success full installation. Kibana login page is popping up. As per the document it should prompt to select the default Index. On configuring user name and password in kibana.yml file the system throws elastic licensing error. Please add the default user name and password in documents.

Able to login with elastic/changeme in Kibana but the rest of components (logstash, beats) are unable to establish the connection.

Here is a error message
root@vagrant-ubuntu-trusty-32:/var/log/elasticsearch# curl http://localhost:9200/
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"missing authentication token for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}root@vagrant-ubuntu-trusty-32:/var/log/elasticsearch# curl http://localhost:9200/
{

Port 9200 and 9300 are not running tcp 4
vagrant@vagrant-ubuntu-trusty-32:~/.ssh$ netstat -lt |egrep '9200|9300|5601'
tcp 0 0 :5601 : LISTEN
tcp6 0 0 localhost:9300 [::]: LISTEN
tcp6 0 0 ip6-localhost:9300 [::]:* LISTEN
tcp6 0 0 ip6-localhost:5601 [::]:* LISTEN
tcp6 0 0 localhost:9200 [::]:* LISTEN
tcp6 0 0 ip6-localhost:9200 [::]:* LISTEN

Collect all log files of the Elastic Stack

Filebeat modules:

• Logstash
• Elasticsearch, probably added in 6.4
• Kibana, probably added in 6.4
• Beats

Can't run all.sh

Hello,

I had an error at the end of running vagrant up (forgot to copy the issue) but had a successful message as well.

Now, when I try run /elastic-stack/all.sh, I have this issue.

vagrant@elastic-stack:~$ /elastic-stack/all.sh
-bash: /elastic-stack/all.sh: /bin/bash^M: bad interpreter: No such file or directory

Do you have a quick solution ?

Thank you,

Cyril

Question

This is a great example of building an elastic stack in a VM. I just have one question regarding the x-pack implementation. How do I log in to kibana with this up? I looked in the files and it looks like a variable was assigned {{kibana_user_password}}. But I can't really seem to get it done. Granted I'm a newbie at elk and I was planning on using this to ingest some old logs for a project to visualize the data with kibana. SOS.

Make the Docker settings platform dependent

Do not include the Docker settings in Filebeat and Metricbeat on 32bit platforms

Tried the recepie on a fresh 16.04 LTS and got this issue; all prerequisites were installed via apt-get

No usable default provider could be found for your system.

Vagrant relies on interactions with 3rd party systems, known as
"providers", to provide Vagrant with resources to run development
environments. Examples are VirtualBox, VMware, Hyper-V.

The easiest solution to this message is to install VirtualBox, which
is available for free on all major platforms.

If you believe you already have a provider available, make sure it
is properly installed and configured. You can see more details about
why a particular provider isn't working by forcing usage with
vagrant up --provider=PROVIDER, which should give you a more specific
error message for that particular provider.

Add Packer support

It would be useful to make it easy to deploy this to Amazon. I'll open a pull request for this.