Benthos is a low latency stream buffer service for piping large volumes of data from one source to another, able to protect against back pressure caused by data surges or service outages further down the pipeline.
Benthos supports a range of input and output protocols out of the box, allowing you to drop benthos in between many existing pipeline services with minimal changes to your architecture.
A range of internal buffer strategies are available, allowing you to choose a balance between latency, protection against back pressure and file based persistence.
Benthos has inputs, an optional buffer, and outputs, which are all set in a single config file.
+-------------------------+ +-------------------------+
| Input Stream | | Output Stream |
| ( ZMQ, HTTP Post, etc ) |--+ +->| ( ZMQ, HTTP Post, etc ) |
+-------------------------+ | | +-------------------------+
v |
+--------------------------------------------+
| Buffer |
| ( Memory-Mapped Files, Memory, None, etc ) |
+--------------------------------------------+
For a full list of available inputs, outputs or buffers you can run:
# Print inputs
benthos --print-inputs | less
# Print buffers
benthos --print-buffers | less
# Print outputs
benthos --print-outputs | less
Currently supported input/output targets:
- ZMQ4 (PUSH, PULL, SUB, PUB)
- Nanomsg/Scalability Protocols (PUSH, PULL, SUB, PUB)
- RabbitMQ (AMQP)
- HTTP 1.1 POST
- STDIN/STDOUT
- File
- Kafka 0.9
You can also have multiple outputs or inputs by configuring a routing strategy (fan in, fan out, round robin, etc).
The main goal of Benthos is to be stable, low-latency and high throughput.
Benthos comes with two benchmarking tools using ZMQ4 benthos_zmq_producer
and
benthos_zmq_consumer
. The producer pushes an endless stream of fixed size byte
messages with a configured time interval between each message. The consumer will
read an endless stream of messages and print a running average of latency,
throughput (per second) and total messages received.
These tools are useful for getting ballpark figures, but it would be good to get some more meaningful third party benchmarks to publish.
Heres a table of results from an 8-core (2.4ghz) machine using the ZMQ4 input, mmap_file persisted buffer, ZMQ4 output configuration, and with messages of size 5000 bytes:
Stream Interval | Avg. Latency (us) | Msg. Rate (msgs/s) | Byte Rate (MB/s) |
---|---|---|---|
100ms | 247 | 9.66 | 0.05 |
10ms | 518 | 75.71 | 0.36 |
1ms | 561 | 606.35 | 2.90 |
100us | 734 | 4454.05 | 21.32 |
10us | 2665 | 7178.02 | 34.36 |
1us | 174099 | 20098.26 | 96.20 |
go get github.com/jeffail/benthos/cmd/...
Benthos supports ZMQ4 for both data input and output. To add this you need to install libzmq4 and use the compile time flag when building benthos:
go install -tags "ZMQ4" ./cmd/...
benthos -c ./config.yaml
Create a fully populated default configuration file:
benthos --print-yaml > config.yaml
benthos --print-json > config.json
The configuration file should contain a section for an input, output, and a buffer. For example, if we wanted to output to a ZMQ4 push socket our output section in a YAML config might look like this:
output:
type: zmq4
zmq4:
addresses:
- tcp://*:1234
socket_type: PUSH
For further information check out the ./docs
directory.
Versions of go above 1.6 should automatically go get
all vendored libraries.
Otherwise, while cloning use --recursive
:
git clone https://github.com/jeffail/benthos --recursive
Or, if the repo is already cloned, get the latest libraries with:
git submodule update --init
To add new libraries simply run:
PACKAGE=github.com/jeffail/util
git submodule add https://$PACKAGE vendor/$PACKAGE"
It might be handy to set yourself a function for this in your .bashrc
:
function go-add-vendor {
git submodule add https://$1 vendor/$1
}
I've added a Dockerfile
for easily building benthos with ZMQ4 support. If you
are new to docker and want to spin up a container:
docker build -t benthos .
docker run \
-v /path/to/your/config.yaml:/config/benthos.yaml \
-p 5555:5555 -p 5556:5556 \ # Map to expose any ports used in your config
benthos