gourmetproject/gourmet

An exquisite network traffic analysis framework

Gourmet

An exquisite network traffic analysis framework
Fast, simple, and customizable

Overview

Features

• Libpcap support
• AF_PACKET support
• Zero copy packet processing (fast!)
• Automatic TCP stream reassembly
• Berkeley Packet Filter support (currently only for libpcap)
• Easily extendable through Go Plugins (see Analyzers section below)

Upcoming Features

• BPF support for AF_PACKET
• Binary release w/ command-line configuration

Usage

Gourmet is not yet finished. But if you would like to give it a test ride, you can do the following:

git clone https://github.com/gourmetproject/gourmet
cd gourmet
cp example_config/get_started_config.yml my_config.yml
docker-compose up --build

Make sure you change the interface argument in my_config.yml to the network interface on your host machine that you want capture traffic on (to know about config.yml see the Configration section below). Gourmet will log all captured traffic to gourmet.log.

Once your container is running, you can just open gourmet.log file to see what gourmet is capturing.

Basic configuration

You can specify configuration file explicitly by adding option -c <path/to/config.yml>. You can see a bunch of example you can get started with in the example_configs folder. Full documentation for the configuration file can be found in the official documentation.

Design

Written in Go

Gourmet is designed from the ground up in Go, the number one language developers want to learn in 2019. It utilizes Google's gopacket library to quickly decode and analyze large amounts of network traffic. Go makes it fast, easy to maintain, and not C/C++.

Highly Concurrent

One of Go's shining features is goroutines. Goroutines are simply functions that run concurrently with other functions. They are much more lightweight, flexible, and easy to work with than standard threads. Goroutines communicate with each other using channels. Channels make it extremely simple to synchronize multithreaded Go programs.

These two language paradigms dramatically improve the speed, memory efficiency, and simplicity of concurrently processing thousands, or even millions, of packets per second.

Easily Customized through Go plugins

Go 1.8, released in February 2017, introduced a new plugin build mode. This build mode allows Go programs (and C programs, through cgo) to export symbols that are loaded and resolved by other Go programs at runtime. The Gourmet Project uses plugins as a way to load custom analyzers passed to the Gourmet sensor at runtime through a YAML configuration file defined by the user. More information how developers can create their own analyzers as Go plugins can be found below.

Analyzers

The Gourmet Project consists of the core Gourmet network sensor and a multitude of common protocol analyzers implemented as Go plugins. We provide a simple interface for other third-party developers to create and share their own analyzers as Go plugins.

In order to create your own analyzer, you must implement the Analyzer interface. This interface is fully documented in the Gourmet documentation. A simple example can be found in the simple_analyzer repository.

In order to implement the interface, you must create a new struct that has a Filter and Analyze function.

Filter

The Filter function takes a *gourmet.Connection object pointer as a parameter, determines whether the analyzer should analyze the connection, and returns true or false. The logic contained within the Filter function should be as simple as possible to filter out irrelevant packets or TCP streams. For example, if you want to write an Analyzer that only looks at DNS traffic, then your filter function should return true if the source or destination port is 53, and false otherwise.

Analyze

The Analyze function takes a gourmet Connection object as a parameter, conducts whatever logic necessary to analyze that connection, and returns an implementation of the Result interface. A Result object can be any data structure you like, such as a string, map, array, or struct. The Result interface only requires you implement the Key function, which returns a string. This string is used as the key value when we add the Result object to the JSON log for the Connection.

Analyzer List

• HTTP Analyzer - Logs information about HTTP traffic
• DNS Analyzer - Logs information about DNS traffic
• Simple Analyzer - Logs the number of bytes in the connection payload
• Bedtime Analyzer - If a specificed domain (such as Netflix) was accessed between certain hours of the day, a Slack bot sends you a message
  • Good example of analyzers depending on other analyzers and using the init() function to maintain state.

Gourmet vs. Zeek (aka Bro)

It is no secret that Zeek is the top choice for network security monitoring. One of the goals of this project is to provide an alternative to Zeek. The table below illustrates some key differences between the two projects.

Feature	Gourmet	Zeek
Log format	Single JSON file; each connection is a root-level JSON object	Multiple CSV files; connection data across files is linked through connection UIDs
Language	Pure Go	Zeek scripting language as a wrapper around C/C++
Customization	Go Plugins	Zeek scripts
Production-ready	Not yet, work in progress	Yes
Open Source	Yes	Yes
Multithreaded	Yes	No (see Zeek Cluster)

Contact Us

Support Us

gourmet's Issues

Minimal mode and verbose mode

If a user only wants to log Connection objects that have at least one analyzer, and ignore everything else, they should be able to.

For example, if someone wants to write an analyzer that filters for DNS traffic that contains domains ending in .io, they should be able to configure Gourmet to only log Connections that meet this filter, and ignore/drop everything else.

To implement this:

1. Create a new config.yml option called capture_mode with three options: minimal, normal, and payloads.
2. If minimal is set, only log connections that have met one or filters for the loaded analyzers
3. If normal is set, log all connections objects and any analyzer results (default, current mode)
4. If verbose is set, log all connection objects, a base64 dump of the connection payload, and any analyzer results

GitHub CI/CD for x86 and x64 builds

Create CI/CD pipeline using GitHub Actions to validate pull requests

Fix Dockerfile and docker-compose.yml and add documentation to README

Testing, Testing, Testing

Right now, as of writing this issue, Gourmet has no unit testing or code coverage.

For a brief overview of Go testing, here are some resources

This issue will stay open as long as there is less than 90% code coverage. If you are looking to help on this effort for Hacktoberfest, try to improve coverage by ~5-10% (or more!) before submitting a pull request.

Memory slowly grows over time

Because we are lazily reading in the entire JSON log file each time we want to add a new entry, the amount of memory we use also grows with the file.

Instead, Logger.Log(c Connection) should append a new JSON record to the file without reading the entire file in. ioutil.ReadFile(l.fileName) is bad.

Steps to fix:

1. Remove ioutil.ReadFile(l.fileName)
2. For each call to Logger.Log, append the JSON as bytes to the file
3. Make sure to close the file, use mutexes correctly, etc.

Create your own Analyzer

Got a cool analyzer idea? Submit a pull request updating the README.md with a link to your analyzer repo!

Migrate from standard plugin package to Hashicorp go-plugin

Go's built-in plugin package, part of the standard library as of Go 1.8, is very finicky with dependency versions across plugins. If a plugin is built using Go Modules, and two plugins use the same package with different versions, then the build process gets angry. This will naturally create a lot of headaches.

Hashicorp's go-plugin package uses an RPC model instead of shared object, which should (hopefully) fix this problem. Migrating is going to be a pain in the ass, but its better to do it now then later.

Create TLS Analyzer

In order to implement more sophisticated analyzers like JA3, self-signed cert detection, etc., we first need a basic TLS analyzer.

gopacket has a TLS Layer already defined, but it does not currently decode a lot of the data we care about, such as Handshake messages or ChangeCipherSpec messages. Ultimately, you should ignore it, as the maintainers are not planning to use the layers package for TCP-specific protocols anyways, and it shouldn't have been merged in the first place.

In order to implement:

1. Look at the DecodeFromBytes source code at https://github.com/google/gopacket/blob/master/layers/tls.go#L130 to get an idea for an idiomatic way in Go to take raw bytes and decode them into structs.
2. Take the full application payload of a reassembled TCP stream, which will be the payload field into a Connection object, and, if the TCP payload is TLS, decode it into structs that store more information about handshakes, cipher specs, etc.
3. Make sure we only store unencrypted data in these structs. Encrypted data is not useful.
4. See the DNS analyzer and how the Analyze function just uses layers.DNS from gopacket because gopacket already has a robust DNS decoder. You will need to write your own TLS decoder since gopacket's layers.TLS doesn't decode enough for us.