kpolley/relk

RELK -- The Research Elastic Stack (Kafka, Beats, Zookeeper, Logstash, ElasticSearch, Kibana, Spark, & Jupyter -- All in Docker)

RELK

The Research Elastic Stack (ELK)

Goal

To make it just as easy to analyze data as it is to collect it.

Features

• Kafka: A distributed event streaming platform capable of handling trillions of events a day
• FileBeat: A lightweight single-purpose data shipper from Elastic
• Elasticsearch: A highly scalable search and analytics engine
• Logstash: A dynamic data collection pipeline with an extensible plugin ecosystem.
• Kibana: An analytics and visualization platform designed to work with Elasticsearch.
• ES-Hadoop: A library that allows Hadoop jobs (& therefore Spark) to interact with Elasticsearch.
• Spark: A fast and general-purpose cluster computing system. It provides high-level APIs in Scala, Python and R.
• GraphFrames: A package for Apache Spark which provides DataFrame-based Graphs.
• Jupyter Notebook: A web application that allows you to create interactive notebooks.

Preparation

The only major modifications needed are:

1. Remove and replace the elasticsearch index templates

   • Located in RELK/elasticsearch/output_templates

2. Remove and replace the logstash conf files

   • Located in RELK/logstash/pipelines

3. Either add the files to analyze into RELK/filebeat/input_files or configure Kafka/FileBeat to ingest files for your use-case.

Running

1. Easy! docker-compose up

If you'd like to have the containers running in the background: docker-compose up -d

Other Notes

• Kafka listens on port 9092
• Kibana uses 5601 (Access it via localhost:5601)
• Jupyter uses 8888 (Access it via localhost:8888)
  • By default, Jupyter notebooks password is 'research'. This can be changed in the docker-compose file

TODO

• Password protect ES/Kibana

Worthy mentions

• The inspiration: HELK -- The Hunting Elastic Stack
• Jupyer/Docker Stacks. An excellent repository with a ton of plug-and-play notebooks. It is incredible how easy it is to set up.
• Docker @ Elastic: Plug-and-play docker containers for Beats, Logstash, Elasticsearch, and Kibana.

relk's Issues

use case question / data ingestion

Hi,

Hope you are all well !

I was looking after a big data stack and I found your repository. After careful reading, some questions popped in my brain :-) about how I can re-use your great repository to my specific project.

So, I am developing an NLP analyzer on news articles, and I have more than 3000 rss/xml feeds. For managing the aggregation, I use this tool https://github.com/ncarlier/feedpushr and it has a kafka plugin to forward articles.

How can I plug it to your stack ? as I am quite new in handling such stacks.
Is there any example of data ingestion based on your stack ?

Thanks in advance for any insights or inputs on these questions.

Cheers,
X