Export partition status

[jirwin]

Export partition status

[kanga333]

@ercliou
Hello.
I also want this metrics.
You seem to have made some changes after forking, but are you planning to send a patch upstream?

[ercliou-zz]

I ended up implementing by sending all metrics at every scrap. When the status is not the matched one, it sends 0. This increases 1:5 with number of partitions (could be a problem if you have a lot of them).
e.g.

kafka_burrow_partition_state{cluster="MY_CLUSTER",group="MY_GROUP",partition="13",topic="MY_TOPIC",state:"OK"} 1
kafka_burrow_partition_state{cluster="MY_CLUSTER",group="MY_GROUP",partition="13",topic="MY_TOPIC",state:"STOP"} 0
kafka_burrow_partition_state{cluster="MY_CLUSTER",group="MY_GROUP",partition="13",topic="MY_TOPIC",state:"REWIND"} 0
kafka_burrow_partition_state{cluster="MY_CLUSTER",group="MY_GROUP",partition="13",topic="MY_TOPIC",state:"STALL"} 0
kafka_burrow_partition_state{cluster="MY_CLUSTER",group="MY_GROUP",partition="13",topic="MY_TOPIC",state:"WARN"} 0

This is so each one of them stay as one independent time series. The reason of this is that I could query the lag + status at Grafana by partition.
Query:

kafka_burrow_partition_lag{group="MY_GROUP",topic="MY_TOPIC"}
* on (topic, partition, group) group_left(status) 
(kafka_burrow_partition_status{group="MY_GROUP",topic="MY_TOPIC"} == 1)

I could send a patch if @jirwin agrees with this :)

[jirwin]

I'm +1 to this. Partition count isn't generally unbound. Maybe it could be enabled by a command line flag, so people can use their own judgement as to whether the surge in new time series is acceptable to them. Maybe --per-partition-stats or something?

[shibug]

How about we define a numeric scheme for the value of this time series? This will save us from 1:5 time series bloat. Our system has 2525 partitions for 52 topics. I am definitely worried about the bloat.

NOTFOUND = 1
OK = 2
WARN = 3
ERR = 4
STOP = 5
STALL = 6

kafka_burrow_partition_state{cluster="MY_CLUSTER",group="MY_GROUP",partition="13",topic="MY_TOPIC"} 2

[ercliou-zz]

Hi @shibug , I explained a lil bit about the reasoning behind in the above PR (centered mostly around Grafana).

We have 15k partitions and haven't encountered performance problems (yet).
I can't look into command line flag right now, if someone would like to look into this, I appreciate it.

[jirwin]

Fixed by #19.