Healthchecks aware of State Stores

[Fryuni]

Healthchecks aware of State Stores

[fhussonnois]

Hi @Fryuni, currently Azkarra exposes the /health endpoint that returns some information about the state of all KafkaStreams instances. In addition, it provides a basic HealthIndicator for Kafka Streams (StreamsHealthIndicator).

Basically, the service is considered to be:

• UP if the KafkaStreams's state is RUNNING (/health returned with HTTP code 200).
• DOWN if the KafkaStreams's state ERROR (/health returned with HTTP code 503).
• Otherwhise, the status for the service is set to UNKNOWN (/health returned with HTTP code 200).

Internally, Azkarra encapsualtes the KafkaStreams instance into the KafkaStreamsContainer class which has a state() method aware of state store recovery. Since Azkarra 0.9, the container will return the the following states during state-store recovering process: STATE_RESTORE_START, STATE_RESTORE_IN_PROGRESS, STATE_RESTORE_COMPLETED.

So, to solve your issue I think you have two solutions :

1. You can implement and register an additional HealthIndicator that will set the status DOWN whenever the Kafka Streams state is not RUNNING.

2. You can implement and register a custom JAX-RS resource to expose your own REST endpoints for liveness and readyness. See Azkarra REST Extensions. When implementinb an extension you can get an access to the AzkarraStreamsService to retrieve all KafkaStreamsContainer instance.

Hope this help!

[marcospassos]

I just checked it out, and it seems easy to implement a custom health check. Thank you!

One more question: how to register a custom HealthIndicator?

[fhussonnois]

You can just annotate your class with @Component so that the class will be registered to the AzkarraContext. Then, the ApiHealthRoutes will look for all registered components that implement the HealthIndicator interface to build an aggregate HealthIndicator.

[marcospassos]

Is there any way to register programmatically (I mean, using the builder)?

[fhussonnois]

Yes, you can just use the AzkarraContext#registerComponent method. Here is an example: https://github.com/streamthoughts/azkarra-streams/blob/master/azkarra-examples/src/main/java/io/streamthoughts/examples/azkarra/noannotation/StreamsApplication.java

Why don't you use the component scan mechanism ?

[marcospassos]

Oh, I see. We're just not a big fan of annotations :)

I'll try to implement the health check and bring some feedback.

Thanks!