mahmoudimus/tank

A very high performance distributed log service

Tank is a very high performance distributed log, inspired in part by Kafka, and other similar services and technologies.

You should begin by reading about the core concepts and the client API (A new Java Tank Client is now available).

It depends on our Switch library, so a lean/stripped-down Switch is included in the repo. Please see building instructions. You may also want to run Tank using its Docker image.

This is our first major open source release as a company, and we plan to accelerate our OSS release efforts in the future.

It will eventually support, among other features:

• clusters via leader/followers arrangement using etcd, similar in semantics to Kafka (but no single controller, and simpler configuration and operation)
• higher level clients, based on Kafka's current client design (depending on the needs of our developers, but PRs will be welcome)
• hooks into other Phaistos infrastructure
• a Kafka/DataFlow like streams topologies abstraction/framework
• encryption (wire transfers and bundle serialization)
• improved client and extended API
• HTTP/1 and HTTP/2 REST APIs

Features include:

• Very high performance
• Very tight encoding of messages(into bundles)
• Simple design (Simple is Beautiful; Convenience is Key)
• compactions based on message keys and in the future based on programmable logic
• A powerful CLI tool for managing, querying, setting messages, mirroring across brokers, etc.

You should probably use Kafka (the Confluent folk are particularly great), or Google Pub/Sub, or any other open source broker/queue instead of Tank - they are all perfectly fine, some more than others, if support for cluster-aware setups is crucial to you(this feature is in the works).

Tank's goal is highest performance and simplicity. If you need very high performance, operation simplicity and no reliance on other services (when running Tank in stand-alone mode), consider Tank.

Please see the wiki for more information.

We chose the name Tank because its a storage chamber, suitable for liquids and gas - which we think is analogous to a storage container for data that flows, from and to other containers and other systems via 'pipes' (connections).