ES_exporter doesn't export cluster statistics

[prometheus-community]

ES_exporter doesn't export cluster statistics

[tinder-ubhatnagar]

Also, can we cut out a new release 0.3.1 or something so that we can have this change!

[tamsky]

I agree that cluster status missing is huge.
Personally I'm hoping to collect /_cluster/health from all nodes.

The /_cluster/health endpoint does support a local=true param, which is what we want from a monitoring and "additional load" perspective.

• _cluster/health api-spec
• _cluster/health implementation

Regarding concerns over additional load, looking at the Response code, most of it looks O(1), with one loop of O(# of indices) which looks at each index's state. If the exporter can have a configurable scrape frequency, that should provide operators with what they need.

Monitoring _cluster/health metrics from all nodes will provide useful signals when the cluster is unhealthy, has a bad or lame node, or other odd split-brain/partition situations. In your "I'm not 100% what would be returned" situation, scraping all nodes should provide a good answer.

[tamsky]

Also, I'd like to suggest the cluster states {green,yellow,red} be exported as three possible integer values assigned to a single metric — Basically avoiding metrics with labels that appear and disappear.

The enum values defined in the es source should be used.

example <metric_name>{} 's :

# green:
elasticsearch_cluster_health_status{cluster_name="mycluster"} 0

later:

# yellow:
elasticsearch_cluster_health_status{cluster_name="mycluster"} 1

[tamsky]

I wrote a _cluster stats collector using bash/jq:

https://gist.github.com/tamsky/5cdf8f36aea09dd797bc

example stat output for a RED state cluster

# HELP elasticsearch_cluster_status value from _cluster/health/
# TYPE elasticsearch_cluster_status gauge
elasticsearch_cluster_status 2 1454454625000

[ewr]

Would you want to pull cluster stats from a subset, or just one at a time? I agree that there's important stuff in there, but I want to think through how you would use it.

I'm not sure exactly what internal ES traffic gets created for the cluster to pull those stats together, or what results you get if you try to run on a node that is currently somehow in a bad state.

[dan-cleinmark]

Any node in the cluster should return the same stats for the cluster level stats. Because of that, I would prefer to pull the cluster stats from a single instance, or at most maybe a handful of the master nodes in a large cluster.

There's always a possibility that a node in the cluster could disconnect from the cluster in which case I'm not 100% what would be returned - I believe that ES should fail-safe into a 'red' cluster state and not display any cluster stats.

As far as additional load for aggregating these metrics, /stats does take a bit of time (~5 seconds on a 7 node cluster) and /health is ~200ms.

[tinder-ubhatnagar]

Is this resolved?

[robothor]

I'll echo the previous comment. Could someone please make a new release of this?