Add sample HTTP/2 commands to consume messages

[polarstreams]

Add sample HTTP/2 commands to consume messages

[jorgebay]

In the latest release, we included REST API that supports full "produce and consume" flow.

The getting started on Docker and getting started on K8s guides now include these commands. I've added REST API docs as well to describe the parameters and different responses.

@rmarting what do you think? should we close this ticket?

[rmarting]

Great news!!! I could not test it, however, we could close this issue as it is basically resolved. If I found something later, I will open a new issue.

Thanks!!! A great improvement!!

[jorgebay]

Good one. Currently the consumer API, although the message themselves are HTTP/2, relies on long lived connections to track state.

We could either simplify the semantics, to favour more stateless clients, or at least provide sample HTTP/2 requests.

[rmarting]

TBH I am not sure if stateless clients are the best approach in streaming architectures where low latency is very important. I tried to use in enterprise use cases the HTTP clients of Kafka, and I only found the producer API very useful for apps where the Kafka protocol is not available, or when we want to have small clients. However, all the consumers always used the Kafka clients to consume as soon as possible the streams (centralizing in the Consumer API the subscription, poll, and commit lifecycles). To reimplement that using single HTTP/2 verbs could not fit and satisfy this kind of architecture.

I am not sure if Barco is looking to achieve this kind of architecture. But IMHO it is something to keep in mind.

Maybe I am wrong, but this is only a comment.

[jorgebay]

Good point, the main goal when designing Barco APIs was to allow producing using simple HTTP and HTTP/2 requests (HTTP/2 for max performance thanks to request pipelining, framing, ... and HTTP/1 to maximize compatibility), to avoid having to deploy a bridge or something like that.

For consuming, we should expect the brokers to be consumer-aware and balance across them and, as you say, stateless communication is not well suited for it.