emiller42/splunk-statsd-backend

A backend plugin for Statsd to output metrics to the Splunk HTTP Event Collector (HEC)

splunk-statsd-backend's Issues

Unsuccessful sending JSON events to Splunk

Hi @emiller42, we recently picked up this backend and have been experimenting sending events from Apache Airflow --> StatsD --> splunk-statsd-backend --> Splunk.
It looks like the JSON events we are sending look like

{"rate":0.2,"count":2,"metricType":"counter","metricName":"airflow.scheduler_heartbeat"}

... whereas the statsd-splunk-backend documentation states that the JSON should look like

{ "time": <timestamp>, "source": "my_source", "sourcetype": "my_sourcetype", "index": "my_index", "event": {...event payload...} }

Questions

1. I suppose my first question, is this project still alive? Please don't take this the wrong way, I just noticed that there hasn't been a commit for quite some time.
2. If it is alive, can you comment on the discrepancy in the JSON event above?

If this backend needs to be augmented to accommodate new versions of Splunk, I am happy to do that with a PR. I just want to know if this project is alive though! Thank you

Add functionality for globalPrefix/globalSuffix similar to graphite backend

This might be worth replicating.

Explicitly state support model in documentation

This project is effectively in maintenance mode, but that is not clearly stated in documentation. Because of this, users (both new and existing) may not have accurate expectations of how the library will be maintained in the future.

Support Indexer Acknowledgement

The Splunk HEC in version 6.4.0+ supports indexer acknowledgement on HEC data. This allows clients sending data to validate that data was not only received (successful POST) but also successfully processed, and then re-send data that failed to index.

details

This would potentially add complexity and overhead that may not be worthwhile. Further investigation would be needed.

Releasing 0.2.0

Hi @emiller42 do you intend to push 0.2.0 to npm?
Thank you

Break up namespace by delimiter

The delimited namespace is really just a carryover from how metrics are sent to graphite. There isn't any real reason we need to maintain the format.

It may be useful to instead break up the namespace into an array of values, which Splunk will see as a multivalued field. So something like:

production.webserver.loadbalancerA.host.responsetime would turn into something like:

key: [ 'production', 'webserver', 'loadbalancerA', 'host', 'responsetime']

Then in splunk instead of having to search for wildcards which are order dependent ( metricName="production.*.responsetime") you could treat them like tags: (key="production" AND key="responsetime")

This might be much more friendly to things like Data Models within Splunk.