berthayes/cp-zeek

Analyze Zeek IDS data with ksqlDB running on Confluent Platform via Docker on your laptop. Or spin up an arbitrary number of AWS hosts, each running Confluent Platform and ksqlDB for use in an instructor-led workshop.

Streaming Zeek Events with Apache Kafka and ksqlDB

Custom Start

Custom Start - Download a 1GB PCAP (~ 60 minutes) file from S3 or supply your own

Quickstart

• Running on localhost

• Running on an external host

• Starting the Connectors

• Running on a bunch of EC2 hosts in AWS

ksqlDB Walkthrough

ksqlDB Walkthrough - A guided walk through using ksqlDB to analyze Zeek and Syslog data.

Running on localhost

git clone https://github.com/berthayes/cp-zeek
cd cp-zeek
docker-compose up -d

Wait about 5 minutes or so for everything to start up, then point your web browser to http://localhost:9021

To start the Syslog and Spooldir connectors, see: Starting the Connectors

Running on an external host

To run this environment on a system that is not your laptop/workstation, edit the docker-compose.yml file.

Look for this line:

CONTROL_CENTER_KSQL_KSQLDB1_ADVERTISED_URL: "http://localhost:8088"

And change it to something like this:

CONTROL_CENTER_KSQL_KSQLDB1_ADVERTISED_URL: "http://yourhost.yourdomain.com:8088"

Then start up docker as above with:

docker-compose up -d

Wait about 5 minutes or so for everything to start up, then point your web browser to http://yourhost.yourdomain.com:9021

To start the Syslog and Spooldir connectors, see: Starting the Connectors

Starting the Connectors

If you also want to analyze streaming Syslog data, run:

curl localhost:8083/connectors -X POST -H "Content-Type: application/json" -d @syslog_connect.json

To start the Spooldir connector that consumes a sample URLHaus watchlist, run:

curl localhost:8083/connectors -X POST -H "Content-Type: application/json" -d @urlhaus_spooldir.json

To start the Spooldir connector that consumes a watchlist of ad servers, run:

curl localhost:8083/connectors -X POST -H "Content-Type: application/json" -d @adhosts_spooldir.json

Custom Start

This repository is a customization based on the Quick Start for Apache Kafka Using Confluent Platform (Docker) documentation available here: https://docs.confluent.io/current/quickstart/ce-docker-quickstart.html It focuses on analyzing Zeek and Syslog events with ksqlDB running on the Confluent Platform for Apache Kafka.

Customize Your Environment

This docker-compose.yml leverages an additional Docker image: bertisondocker/zeek-tcpreplay-kafka for generating Zeek data to stream to Apache Kafka in real time. When the image runs, tcpreplay automatically starts reading ./pcaps/zeek_streamer.pcap

To run this with the included packet capture simply run: docker-compose up -d

Analyze Your Own PCAP

To analyze your own packet capture, Copy your pcap file to ./cp-zeek/pcaps/zeek_streamer.pcap The zeek-streamer Docker image begins reading the zeek_streamer.pcap file automatically at startup.

A super-fun pcap for analysis is available here: s3://bhayes-pcaps/garage-2020-10-18.pcap

The get_pcap_from_s3.py script in the workshop_scripts directory will download this PCAP file and rename it to zeek_streamer.pcap for you.

Edit yak_shaving.conf to read:

download_pcap = True

Optionally, edit the S3 bucket and object name to point to your own PCAP file. Then run:

python3 get_pcap_from_s3.py

This packet capture is around 1GB in size and was originally captured over an hour or so. The local network is 192.168.1.0/24 and there are maybe a dozen or so hosts that are active. Some hosts are more active than others, and some hosts’ activities are more interesting than others.

Have fun!