default kafka.yaml doesnt seem to work

[datadog]

default kafka.yaml doesnt seem to work

[miketheman]

It appears that this may be a logical bug vs a code bug - as the kafka integration should automatically load and store the "standard" jmx metrics we collect, like jvm.gc.cms.count and jvm.heap_memory - can you confirm these are not appearing in your Metrics Summary?

[odise]

It seems that this is a weird yaml parsing issue. The kafka.yaml is causing the following exception on dd-agent startup:

ERROR | dd.collector | jmxfetch(jmxfetch.py:145) | kafka check is not a valid jmx configuration: Each configuration must have an 'include' section. See http://docs.datadoghq.com/integrations/java/ for more information

Changing the template a bit (adding two more spaces before the domain: keyword) is fixing the problem:

instances:
  - host: localhost
    port: 9999

init_config:
  is_jmx: true

  # Metrics collected by this check. You should not have to modify this.
  conf:
    #
    # Aggregate cluster stats
    #
    - include:
        domain: '"kafka.server"'
        bean: '"kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesOutPerSec"'
        attribute:
          MeanRate:
            metric_type: counter
            alias: kafka.net.bytes_out

[solarce]

I ran into this as well. I created a PR with a fix, which I've tested in my dev environment as is working.

/etc/init.d/datadog-agent info
...
    kafka
    -----
      - instance #lookout-kafka-bburton-0 [OK] collected 22 metrics
      - Collected 22 metrics, 0 events & 0 service checks

@miketheman to also answer your last question, without this fix the agent does not start collecting any metrics via JMX. Once I applied this fix to my test host I began seeing the Kafka specific metrics (from JMX) appear on the node in the datadog UI.

It'd be great to see this get in and a 2.0.1 released

[donaldguy]

See also #163 ... I was still waiting for direction on what's wanted on the tests

[solarce]

FWIW I don't care which PR gets merged, let's just get a fix shipped plz :)

[flutist599]

adding the 2 spaces solved it!! spent 2 hours troubleshooting, now I found your help many thanks!!!

[sam-obeid]

This was very useful! Thanks for sharing.