jason-cooke/bifrost

Safely archive data from Apache Kafka to S3 with no Hadoop dependencies :)

.

bifrost

Archive Kafka data safely to either Amazon's S3 or Microsoft's Azure Blob Storage.

bifrost is a daemon that connects to Kafka, consumes all topics and persists them to the cloud for long term storage and analysis. It uses the baldr file format to store Kafka messages. baldr-files are gzipped before they are written to disk.

There are other services for persisting messages from Kafka to S3, such as secor from pinterest. The main difference between bifrost and these is, that whereas they rely on the Hadoop sequence files for storage, bifrost uses the baldr format. This means that consumers of the persisted messages do not need to rely on often very large implementations of libraries for reading Hadoop sequence files. The baldr file format follows a minimal design that allows for easy and quick streaming with a very small code-footprint. It does not allow for arbitrary indexing.

Usage

bifrost can be run directly from a checkout of the project by using leiningen. The app requires some basic configuration, namely ZooKeeper configuration to connect to Kafka and cloud storage credentials to store baldr-files on the chosen cloud storage. The project contains an example configuration in etc/config.edn.example.

 $ lein run -- --config ./etc/config.edn

To run the app in production, we recomment building an uberjar and run that on the app server.

$ lein uberjar
$ java -jar target/*-standalone.jar --config /opt/uswitch/bifrost/etc/config.edn

The Java temp-dir is used for storing baldr-files locally before uploading them. Files are removed upon succesful upload and program exit. To change the temp-directory, override java.io.tmpdir.

Logging is done through logback. To configure logback, please set logback.configurationFile. The logback configuration is only respected for uberjars. sl4j is used in development.

Here's a complete example of configuring and running an uberjar in production.

$ java -Djava.io.tmpdir=/mnt/bifrost-tmp \
       -Dlogback.configurationFile=/opt/uswitch/bifrost/etc/logback.xml \
       -server \
       -jar /opt/uswitch/bifrost/lib/*-standalone.jar \
       --config /opt/uswitch/bifrost/etc/config.edn

License

Copyright © 2014 uSwitch

Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.