devopsbox/gollum

A n:m message multiplexer written in Go

Gollum

Gollum is a n:m multiplexer that gathers messages from different sources and broadcasts them to a set of destinations.

There are a few basic terms used throughout Gollum:

• "Consumers" read data from other services
• "Producers" write data to other services
• "Streams" route data between consumers and producers
• A "message" is a set of data passed between consumers and producers
• "Formatters" can transform the content of messages
• "Filters" can block/pass messages based on their content

Writing a custom plugin does not require you to change any additional code besides your new plugin file.

Consumers (reading data)

• Console read from stdin.
• File read from a file (like tail).
• Http read http requests.
• Kafka read from a Kafka topic.
• Kinesis read from a Kinesis stream.
• Profiler Generate profiling messages.
• Proxy use in combination with a proxy producer to enable two-way communication.
• Socket read from a socket (gollum specfic protocol).
• Syslogd read from a socket (syslogd protocol).

Producers (writing data)

• Console write to stdin or stdout.
• ElasticSearch write to elasticsearch via http/bulk.
• File write to a file. Supports log rotation and compression.
• HTTPRequest HTTP request forwarder.
• InfluxDB send data to an InfluxDB server.
• Kafka write to a Kafka topic.
• Kinesis write data to a Kinesis stream.
• Null like /dev/null.
• Proxy two-way communication proxy for simple protocols.
• Scribe send messages to a Facebook scribe server.
• Socket send messages to a socket (gollum specfic protocol).
• Spooling write messages to disk and retry them later.
• Websocket send messages to a websocket.

Streams (multiplexing)

• Broadcast send to all producers in a stream.
• Random send to a random producer in a stream.
• RoundRobin switch the producer after each send in a round robin fashion.
• Route convert streams to one or multiple others

Formatters (modifying data)

• Base64Encode encode messages to base64.
• Base64Decode decode messages from base64.
• CollectdToInflux08 convert CollectD 0.8 data to InfluxDB compatible values.
• CollectdToInflux09 convert CollectD 0.9 data to InfluxDB compatible values.
• CollectdToInflux10 convert CollectD 0.10 data to InfluxDB compatible values.
• Envelope add a prefix and/or postfix string to a message.
• Forward write the message without modifying it.
• Hostname prepend the current machine's hostname to a message.
• Identifier hash the message to generate a (mostly) unique id.
• JSON write the message as a JSON object. Messages can be parsed to generate fields.
• ProcessJSON Modify fields of a JSON object.
• Runlength prepend the length of the message.
• Sequence prepend the sequence number of the message.
• SplitToJSON tokenize a message and put the values into JSON fields.
• StreamName prepend the name of a stream to the payload.
• StreamRevert route a message to the previous stream (e.g. after it has been routed).
• StreamRoute route a message to another stream by reading a prefix.
• Timestamp prepend a timestamp to the message.

Filters (filtering data)

• All lets all message pass.
• JSON blocks or lets json messages pass based on their content.
• None blocks all messages.
• RegExp blocks or lets messages pass based on a regular expression.
• Stream blocks or lets messages pass based on their stream name.

Installation

From source

Installation from source requires the installation of the Go toolchain.
Gollum has Godeps support but this is considered optional.

$ go get .
$ go build
$ gollum --help

You can use the supplied make file to trigger cross platform builds.
Make will produce ready to deploy .tar.gz files with the corresponding platform builds.
This does require a cross platform golang build. For details see the "build" section below.

Usage

To test gollum you can make a local profiler run with a predefined configuration:

$ gollum -c config/profile.conf -ps -ll 3

By default this test profiles the theoretic maximum throughput of 256 Byte messages.
You can enable different producers to test the write performance of these producers, too.

Configuration

Configuration files are written in the YAML format and have to be loaded via command line switch. Each plugin has a different set of configuration options which are currently described in the plugin itself, i.e. you can find examples in the GoDocs.

Commandline

-c or --config [file]

Use a given configuration file.

-h or --help

Print this help message.

-ll or --loglevel [0-3]

Set the loglevel [0-3]. Higher levels produce more messages as in 0=Errors, 1=Warnings, 2=Notes, 3=Debug.

-m or --metrics [port]

Port to use for metric queries. Set 0 to disable.

-n or --numcpu [number]

Number of CPUs to use. Set 0 for all CPUs.

-p or --pidfile [file]

Write the process id into a given file.

-pc or --profilecpu [file]

Write CPU profiler results to a given file.

-pm or --profilemem [file]

Write heap profile results to a given file.

-ps or --profilespeed

Write msg/sec measurements to log.

-tc or --testconfig [file]

Test a given configuration file and exit.

-v or --version

Print version information and quit.

Building

Mac OS X

The easiest way to install go is by using homebrew:
brew install go

If you want to do cross platform builds you need to specify an additional option (Go 1.5 does not require this anymore):
brew install go --with-cc-all

Linux

Download Go from the golang website and unzip it to e.g. /usr/local/go.
You have to set the GOROOT environment variable to the folder you chose:
export GOROOT=/usr/local/go

Prerequisites

If you do not already have a GOPATH set up you need to create one.
The location is free of choice, we prefer to put it into each users home folder:

mkdir -p ~/go
export GOPATH=$(HOME)/go

You can download gollum via go get github.com/trivago/gollum or clone it directly into your GOPATH.
If you choose this way you need to download your dependencies directly from that folder

mkdir -p $(GOPATH)/src/github.com/trivago
cd $(GOPATH)/src/github.com/trivago
git clone https://github.com/trivago/gollum.git
cd gollum
go get -u .

Build

Building gollum is as easy as go build.
If you want to do cross platform builds use make all or specifiy one of the following platforms instead of "all":

• current build for current OS (default)
• freebsd build for FreeBSD
• linux build for Linux x64
• mac build for MacOS X
• pi build for Linux ARM
• win build for Windows

There are also supplementary targets for make:

• clean clean all artifacts created by the build process
• test run unittests
• restore install godep and restore all dependencies
• aws build for Linux x64 and generate an Elastic Beanstalk package

If you want to use native plugins (contrib/native) you will have to enable the corresponding imports in the file contrib/loader.go. Doing so will disable the possibility to do cross-platform builds for most users.

Dockerfile

The repository contains a Dockerfile which enables you to build and run gollum inside a Docker container.

$ docker build -t trivago/gollum .
$ docker run -it --rm trivago/gollum -c config/profile.conf -ps -ll 3

To use your own configuration you could run:

$ docker run -it --rm -v /path/to/config.conf:/etc/gollum/gollum.conf:ro trivago/gollum -c /etc/gollum/gollum.conf

Solving dependency problems

If you got any errors during build regarding external dependencies (i.e. the error message points to another repository than github.com/trivago) you can restore the last dependency snapshot using godep. Install godep via go get github.com/tools/godep and restore the dependency via godep restore when inside the gollum base folder.

License

This project is released under the terms of the Apache 2.0 license.