scholtzan/stackdriver-emitter

Apache Druid Stackdriver Emitter Extension

Stackdriver Emitter Plugin for Druid

Custom extension for sending metric events from Apache Druid to Google Stackdriver.

Getting it working in Druid

Run mvn package to create a .jar file that contains all dependencies.

The generated .jar file needs to be copied into druids extensions/ directory (for example into /app/druid/extensions/stackdriver-emitter/).

When running druid it should automatically detect and execute this extension.

Stackdriver Authentication in GKE

To authenticate to the Stackdriver API the GOOGLE_APPLICATION_CREDENTIALS environment variable needs to be set pointing to the .json file with the necessary credentials.

To set these credentials in GKE the following steps are necessary:

• Create service account keys which should download a .json with the credentials
• Create a Kubernetes secret resource for those service account credentials: kubectl create secret generic service-account-credentials --from-file /path/to/credentials.json
• Mount the credentials in the container that needs access
• Set the GOOGLE_APPLICATION_CREDENTIALS environment variable

See https://stackoverflow.com/questions/47021469/how-to-set-google-application-credentials-on-gke-running-through-kubernetes

Configuration

Configuration parameters for the plugin that can be set in the config file (eg. common.runtime.properties):

druid.emitter=stackdriver

# number of events sent together in one request to Stackdriver
druid.emitter.stackdriver.flushThreshold=5

# max. queue size for event queue
druid.emitter.stackdriver.maxQueueSize=1000

# frequency in ms of sending collected events to Stackdriver
druid.emitter.stackdriver.consumeDelay=60000

# Stackdriver project ID
druid.emitter.stackdriver.projectId=streaming-events-dev

# optionally, file with whitelist of druid events that should be processed
druid.emitter.stackdriver.metricMapPath=/app/druid/extensions/stackdriver-emitter/defaultMetrics.json

The extension must also be registered in the config file:

druid.extensions.loadList=[..., "stackdriver-emitter", ...]

Metrics in Stackdriver

The emitted metric events should appear in Stackdriver:

Resources

• Stackdriver Monitoring API
• GKE Credentials Configuration
• Druid Plugins

stackdriver-emitter's Issues

CredentialsProvider class

Instead of reading and unwrapping the Google credentials inside StackdriverSender it would probably be cleaner to put that logic into a separate GoogleCredentialsProvider class.