allegro/hermes

Fast and reliable message broker built on top of Kafka.

Hermes

Hermes is an asynchronous message broker built on top of Kafka. We provide reliable, fault tolerant REST interface for message publishing and adaptive push mechanisms for message sending.

See our 10-minute getting started guide with Docker: Getting started

Visit our page: hermes.allegro.tech

See our full documentation: http://hermes-pubsub.readthedocs.org/en/latest/

If you have any question or idea regarding the project, please feel free to reach us out using GitHub discussions.

License

hermes is published under Apache License 2.0.

hermes's Issues

BrokerListener might not be called if Undertow throws exception

Retransmission dry-run

There should be a dry-run option in retransmission which just returns calculated offsets for each partition

support for custom Jersey package scanning

As Hermes user i want to be able to specify custom packages that Jersey will scan on startup (i.e. to attach custom filters).

Broker seems to be down

Going through the tutorial, interacting with groups/topics/subscriptions worked fine for me. At the last step of publishing though (the first step at port :8888), I'm met with an HTTP 500 error: {"message":"Broker seems to be down, cause: Failed to update metadata after 500 ms.","code":"INTERNAL_ERROR"}. However, trying to access :8888 by browser, it yields me a http://localhost:8888/status/health with content UP. Maybe this relates to Kafka settings like topic.metadata.refresh.interval.ms, which apparently had a default value of 600 * 1000 (i.e. 1200 times as much). I hadn't changed anything, so not sure why it's 500, though on localhost I'd expect this 500 ms should suffice... What might be a good way to debug this?

Automatic conversion from JSON to AVRO for topics with AVRO content type

As a hermes user I want my json messages to be automatically converted to avro when publishing to topic with content type avro.

Wrap message sending in a "transaction"

We should set a timeout for message sending and not rely on futures created by MessageSender

Output rate limiter - additional treshold

We need to add additional treshold in consumer rate limiter, so it can increase speed even if there is some percent of failures - it's too easy to starve consumer now.

Add Elasticsearch tracker

image allegro/hermes-consumers not found

Following the guide:

./run.sh
Starting docker_kafka_1...
Creating docker_consumers_1...
Pulling image allegro/hermes-consumers:latest...
Pulling repository allegro/hermes-consumers
[...]
Error: image allegro/hermes-consumers:latest not found

I also had to remove the 'distZip' parameter from build.sh to prevent another error (on there being no such task)...

As the error stated though, I can't find hermes-related images in allegro's Docker repository.

I noticed there were already local Dockerfiles that corresponded to the ones it can't find online, so I tried building images using those:

$ docker build Dockerfile-frontend
FATA[0000] The Dockerfile (Dockerfile) must be within the build context (Dockerfile-frontend) 

I must concede I'm relatively new to Docker, but I'm not sure if I'm missing an easy way to address these issues... I'd appreciate any help! :)

Running listeners from kafka-network-thread

Listeners should be run from undertow worker pool.

NPE on publishing when contentType not found on topic

Sending message fails with Broker seems to be down, cause: null

Hermes reports with short hostname

Problem with consumer restart after retransmission

There is a problem with consumer being spawned too many times after restart for partition that has been retransmitted recently. This may be connected with retransmission marker not being cleaned up from ZK sometimes.

shared timeout executor

Timeout executor should use shared thread pool. Also pool should be monitored.

Extract MongoLogRepository to external module

TODO:

• Extract MongoLogRepository to external module
• add posibility to register multiple implementations of LogRepository

AC:
should be possible to use multiple implementations of LogRepository simultaneously

Separated broker latency metric for ack-all and ack-leader metrics

Hermes client support for HTTP/2

Hermes client should support sending events via HTTP/2 protocol.
https://http2.github.io/http2-spec/

Handle invalid host name in subscription endpoint

We can get NPE when using invalid host name in subscription endpoint.

Peace of code which throws NPE:

UriUtils.extractHost(URI.create("http://underscore_are_invalid_chars_in_host/path"))

Publishing Hermes in Maven Central

We need to publish Hermes in Maven Central (and snapshots in OSS Nexus).

frontend graceful startup

Hermes frontend should enable health check service in last startup hook.

Hello

commit	file reference	pull request
8a08c22	hermes-tracker-elasticsearch/src/main/java/pl/allegro/tech/hermes/tracker/elasticsearch/DocumentBuilder.java#L6	#90

Wrong Message-Id in sent messages

There is bug in Jetty Sender that results in generating new MessageId instead of sending the old one.

Message size related metrics for frontend

As a hermes maintainer I want to see what is average and max message size per topic.

Batching and metrics in Elasticsearch tracker

single message read with avro

Rest operation for single message read should also read messages in avro format. Avro messages should be converted to json for that operation.

Producer status codes wrong meter name

subscription last undelivered fails

Getting /undelivered/last on subscription always returns 404

Opening up Management API

We need Hermes management API to be accessible from any location (like i.e. github API).

Hermes client event send retry on connection closed exception

Currently hermes client retries to send events on connection timeout / 5xx server response.
It should retry on connection closed exception as well.

Hermes client with jersey message sender may leak connections

When no getBody is called on JerseyHermesResponse the connection stays open.

As Jersey documentation says:

The underlying connections are opened for each request and closed after the response is received and entity is processed (entity is read)

https://jersey.java.net/documentation/latest/client.html#d0e5011

Hermes support for publishing events using HTTP/2

As a hermes user I want to be able to publish events using http2 protocol.

logging message publishing time in consumer trace

We should log message publishing timestamp in consumer tracker, so it would be easy to track down messages that have been delayed.

[refactor] remove any password occurencess left

There are some superuser passwd leftovers, remove them from OS version.

Thread pools queues monitoring

When using thread pool it is always difficult to set correct number of threads.

To make it easier we want to measure a time each task is waiting for execution in thread pool, and store it together with other metrics in graphite.

First pool we want to measure this way is jetty pool in consumers module.

Additional metrics for producer with ack=ALL

Hermes uses same metric to meter latency for producer with ack=-1 and ack=1. Due to that we have one chart with latency for 99 precentile which show us always value 65ms. We could visualize latency more precisely by separating latency metric for both producers.

AC:

• additional metric for producer latency with ack=-1
• two charts with latency. First for ack = 1, second for ack=-1

HermesManagement will not start on new clean zk instance

The problem is that root zookeeper node is created by frontend, so you need to run frontend first before you run managment (only when running new clean installation of hermes)

Avro support

AC:

• On every topic I can choose json or avro serialization
• Hermes frontend accepts data in both avro and json
• Hermes subscriber transforms all data to json
• avro schemas are stored for all topics that use avro

Consumer offsets stored in Kafka not in Zookeeper

statsd support

I would like to report stats to statsd to get overview of system behavior.

empty subscriptions

When I do some GET operation on non existing subscription then it will be created. Later I can't delete it via DELETE operation.

When creating hermes Consumer fails, kafka Consumer is still active

ConsumersSupervisor should close kafka Consumer when creating hermes Consumer fails.

Broken image link in docs

Here http://hermes-pubsub.readthedocs.org/en/latest/contents/overview/01_architecture.html#hermes-consumers is problem with rendering image named "/_static/consumer_state_machine.png". Image is available here: http://hermes-pubsub.readthedocs.org/en/latest/_static/consumer_state_machine.png

Daily indices in Elasticsearch tracker repositories

Retransmission should accept human readable date time

Now it accepts timestamp in millis which is error-prone

Listing all topics and subscriptions that are currently traced

As a hermes user I want to be able to check which topics and which subscriptions are currently being traced.

Cassandra implementation for LogRepository

Travis

What about Travis or other CI integration?

New offset commiting mechanism with lag raporting

Current offset commiting mechanism in extreme cases (quick output rate decrease) might lead to situations, when we commit events that might have not been sent. If sender is be okay - this is fine, but in case of any sender bugs, we will acknowledge unsent event.

This change might lead to situation, when lag is growing, as we don't commit messages. There should be some graphite metric (and log message) that we can observe and react.

Delete of subscription doesn't fully work

Hermes doesn't remove metrics node under subscription node. Therefore subscription node is not removed and we have empty subscription. Concerns 0.7.6 version of hermes.

Publish docker images

Docker images should be publicly available without manual building