paologallinaharbur/nri-kafka

New Relic Infrastructure Kafka Integration

New Relic integration for Kafka

The New Relic integration for Kafka captures critical performance metrics and inventory reported by Kafka clusters. Data on Brokers, Topics, Java Producers, and Java Consumers is collected.

Inventory data is obtained mainly from Zookeeper nodes and metrics are collected through JMX.

See our documentation web site for more details.

Requirements

JMX is required to be enabled on the following entities in order to be able to collect metrics:

• Brokers
• Java Producers
• Java Consumers

Information on configuring JMX can be found here.

Installation and usage

For installation and usage instructions, see our documentation web site.

Compatibility

• Supported OS: No limitations
• Kafka versions: 0.8+

Building

Golang is required to build the integration. We recommend Golang 1.11 or higher.

After cloning this repository, go to the directory of the Kafka integration and build it:

$ make

The command above executes the tests for the Kafka integration and builds an executable file called nri-kafka under the bin directory.

To start the integration, run nri-kafka:

$ ./bin/nri-kafka

If you want to know more about usage of ./bin/nri-kafka, pass the -help parameter:

$ ./bin/nri-kafka -help

External dependencies are managed through the govendor tool. Locking all external dependencies to a specific version (if possible) into the vendor directory is required.

Testing

To run the tests execute:

$ make test

Consumer/Producer container image

A container image for testing purposes to create a simple Java consumer/producer exposing JMX metrics, kafka-consumer-producer.

The consumer/producer will be launched with the client.id set to the name of the hostname for easier matching when setting the nri-kafka config file.

The container can be launched as a consumer with the command:

command: ["consumer","broker:9092","topicA","groupA"]

And as a producer:

command: ["producer","broker:9092","topicA"]

Refer to this example to see how to overwrite the JMX settings if you want to use it (note we set the hostname to not be auto-generated): docker-compose-example.

Support

Should you need assistance with New Relic products, you are in good hands with several support diagnostic tools and support channels.

New Relic offers NRDiag, a client-side diagnostic utility that automatically detects common problems with New Relic agents. If NRDiag detects a problem, it suggests troubleshooting steps. NRDiag can also automatically attach troubleshooting data to a New Relic Support ticket.

If the issue has been confirmed as a bug or is a Feature request, please file a Github issue.

Support Channels

• New Relic Documentation: Comprehensive guidance for using our platform
• New Relic Community: The best place to engage in troubleshooting questions
• New Relic Developer: Resources for building a custom observability applications
• New Relic University: A range of online training for New Relic users of every level
• New Relic Technical Support 24/7/365 ticketed support. Read more about our Technical Support Offerings.

Privacy

At New Relic we take your privacy and the security of your information seriously, and are committed to protecting your information. We must emphasize the importance of not sharing personal data in public forums, and ask all users to scrub logs and diagnostic information for sensitive information, whether personal, proprietary, or otherwise.

We define “Personal Data” as any information relating to an identified or identifiable individual, including, for example, your name, phone number, post code or zip code, Device ID, IP address, and email address.

For more information, review New Relic’s General Data Privacy Notice.

Contribute

We encourage your contributions to improve this project! Keep in mind that when you submit your pull request, you'll need to sign the CLA via the click-through using CLA-Assistant. You only have to sign the CLA one time per project.

If you have any questions, or to execute our corporate CLA (which is required if your contribution is on behalf of a company), drop us an email at [email protected].

A note about vulnerabilities

As noted in our security policy, New Relic is committed to the privacy and security of our customers and their data. We believe that providing coordinated disclosure by security researchers and engaging with the security community are important means to achieve our security goals.

If you believe you have found a security vulnerability in this project or any of New Relic's products or websites, we welcome and greatly appreciate you reporting it to New Relic through HackerOne.

If you would like to contribute to this project, review these guidelines.

To all contributors, we thank you! Without your contribution, this project would not be what it is today.

License

nri-kafka is licensed under the MIT License.

nri-kafka's Issues

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: Cannot find preset's package (github>newrelic/coreint-automation:renovate-base.json5#renovateGolang)

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• fix(deps): update github.com/newrelic/nrjmx/gojmx digest to 90d4470
• fix(deps): update dependency org.apache.kafka:kafka-clients to v0.11.0.3
• fix(deps): update dependency org.apache.kafka:kafka-streams to v0.11.0.3
• fix(deps): update module github.com/shopify/sarama to v1.40.0
• chore(deps): update actions/github-script action to v6
• fix(deps): update dependency org.apache.kafka:kafka-clients to v3
• fix(deps): update dependency org.apache.kafka:kafka-streams to v3
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

docker-compose

tests/integration/docker-compose.yml

dockerfile

Dockerfile

• golang 1.20
• maven 3-jdk-11

build/Dockerfile

• golang 1.20.6-bookworm

tests/integration/Dockerfile

• golang 1.20.6

tests/integration/consumer-producer/Dockerfile

• maven 3.9.3-eclipse-temurin-17

github-actions

.github/workflows/on_prerelease.yml

• actions/checkout v3
• docker/login-action v2
• actions/checkout v3
• actions/setup-go v4
• actions/checkout v3
• docker/login-action v2
• newrelic/integrations-pkg-test-action v1
• actions/checkout v3
• newrelic/integrations-pkg-test-action v1
• actions/checkout v3
• actions/setup-go v4
• docker/login-action v2
• newrelic/infrastructure-publish-action v1
• newrelic/integrations-pkg-test-action v1
• actions/checkout v3
• docker/setup-buildx-action v2
• docker/login-action v2

.github/workflows/on_release.yml

• docker/login-action v2
• newrelic/infrastructure-publish-action v1
• newrelic/integrations-pkg-test-action v1

.github/workflows/push_pr.yml

• actions/checkout v3
• actions/setup-go v4
• newrelic/newrelic-infra-checkers v1
• returntocorp/semgrep-action v1
• golangci/golangci-lint-action v3
• newrelic/release-toolkit v1
• actions/checkout v3
• docker/login-action v2
• actions/checkout v3
• actions/setup-go v4
• actions/checkout v3
• actions/setup-go v4
• actions/checkout v3
• docker/login-action v2

.github/workflows/repolinter.yml

• actions/github-script v2
• actions/checkout v3
• newrelic/repolinter-action v1

.github/workflows/trigger_prerelease.yml

• newrelic/coreint-automation v1

gomod

go.mod

• go 1.20
• github.com/Shopify/sarama v1.30.1
• github.com/newrelic/infra-integrations-sdk v3.7.3+incompatible
• github.com/newrelic/nrjmx/gojmx v0.0.0-20230328105658-176501e60193@176501e60193
• github.com/samuel/go-zookeeper v0.0.0-20201211165307-7117e9ea2414@7117e9ea2414
• github.com/stretchr/testify v1.8.4
• github.com/xdg-go/scram v1.1.2
• github.com/xeipuuv/gojsonschema v1.2.0
• golang.org/x/sync v0.3.0

maven

tests/integration/consumer-producer/pom.xml

• org.apache.kafka:kafka-clients 0.10.0.0
• org.apache.kafka:kafka-streams 0.10.0.0
• org.apache.flink:flink-connector-kafka-0.10_2.11 1.11.6
• org.apache.flink:flink-streaming-java_2.11 1.14.6
• org.apache.maven.plugins:maven-compiler-plugin 3.11.0

---

• Check this box to trigger a request for Renovate to run again on this repository