datadog/docker-compose-example

A working example of using Docker Compose with Datadog

Using Docker Compose with Datadog

Datadog offers native Docker container monitoring, either by running the Agent on the host or running in a sidecar container. Which is the best way to run it? It ultimately depends on the tooling you have in place to manage the Agent's configuration. If you want to go Docker all the way, you can run the Agent as a sidecar and control its configuration with custom Dockerfiles.

Let's see what it looks like.

Starting off from the Compose example

To build a meaningful setup, we start from the example that Docker put together to illustrate Compose. A simple python web application that connects to Redis to store the number of hits.

Here is the docker-compose.yml that powers the whole setup.

version: "3"
services:
  web:
    build: web
    command: python app.py
    ports:
     - "5000:5000"
    volumes:
     - ./web:/code # modified here to take into account the new app path
    links:
     - redis
    environment:
     - DATADOG_HOST=datadog # used by the web app to initialize the Datadog library
  redis:
    image: redis
  # agent section
  datadog:
    build: datadog
    links:
     - redis # ensures that redis is a host that the container can find
     - web # ensures that the web app can send metrics
    environment:
     - DD_API_KEY=__your_datadog_api_key_here__
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock
     - /proc/:/host/proc/:ro
     - /sys/fs/cgroup:/host/sys/fs/cgroup:ro

Configuring the Agent

Because the Agent needs to monitor redis it needs:

1. the proper redisdb.yaml in the container's /etc/datadog-agent/conf.d
2. to find the redis node.

The Agent's Dockerfile takes care of #1.

FROM datadog/agent:latest
ADD conf.d/redisdb.yaml /etc/datadog-agent/conf.d/redisdb.yaml

And the Compose yaml files creates the link to redis with:

  links:
    - redis

All in one

How to test this?

1. Install Docker Compose
2. Clone this repository
3. Update your DD_API_KEY in docker-compose.yml
4. Run all containers with docker-compose up
5. Verify in Datadog that your container picks up the docker and redis metrics

docker-compose-example's Issues

Instructions on how to connect to app running on different host would be helpful

I'm running the docker image for DataDog on a VM (using my laptop as host). I'd like to be able to connect to a node app running locally at http://localhost:3000 . It is not immediately clear to me how to make that happen. Any help would be greatly appreciated, and maybe add something which addresses a similar scenario to the README doc

DD_API_KEY environment variable ignored by the container

Steps to reproduce

1. Run Docker Compose with the following service:

  datadog:
    image: datadog/agent:latest
    links:
      - service1
      - service2
      - service3
    environment:
      - DD_APY_KEY=<disclosed>
      - DD_SITE=datadoghq.eu
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup:/host/sys/fs/cgroup:ro

What happens?

app-engine-datadog-1  | [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
app-engine-datadog-1  | [s6-init] ensuring user provided files have correct perms...exited 0.
app-engine-datadog-1  | [fix-attrs.d] applying ownership & permissions fixes...
app-engine-datadog-1  | [fix-attrs.d] done.
app-engine-datadog-1  | [cont-init.d] executing container initialization scripts...
app-engine-datadog-1  | [cont-init.d] 01-check-apikey.sh: executing... 
app-engine-datadog-1  | 
app-engine-datadog-1  | ==================================================================================
app-engine-datadog-1  | You must set an DD_API_KEY environment variable to run the Datadog Agent container
app-engine-datadog-1  | ==================================================================================
app-engine-datadog-1  | 
app-engine-datadog-1  | [cont-init.d] 01-check-apikey.sh: exited 1.
app-engine-datadog-1  | [cont-finish.d] executing container finish scripts...
app-engine-datadog-1  | [cont-finish.d] done.
app-engine-datadog-1  | [s6-finish] waiting for services.
app-engine-datadog-1  | [s6-finish] sending all processes the TERM signal.
app-engine-datadog-1  | [s6-finish] sending all processes the KILL signal and exiting.
app-engine-datadog-1 exited with code 1

What should happen?

Service should not fail over missing DD_API_KEY environment variable since it is defined.

Does the configuration work for dd.api send metrics call ?

I am using the same configuration as you have in the example. But when running api.send_metric, I get the following stack trace:

  File "/usr/local/lib/python2.7/dist-packages/datadog/api/api_client.py", line 95, in submit
    if obj_params.get('host', "") == "":
AttributeError: 'str' object has no attribute 'get'

I have set api_key, app_key in my options dictionary. Using datadog python client.

Wrong API_KEY env var name in example?

docker-compose-example/docker-compose.yml

Line 23 in 95d5297

- API_KEY=__your_datadog_api_key_here__

should this be DD_API_KEY instead of API_KEY? docker logs had this and it started working when I switched the env var name

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01-check-apikey.sh: executing...

==================================================================================
You must set an DD_API_KEY environment variable to run the Datadog Agent container
==================================================================================

[cont-init.d] 01-check-apikey.sh: exited 1.

Datadog docker compose file format version 2 error

Hi! I got this error when I have tried to change the format of docker compose file to the second version. Do you have an example of datadog docker-compose.yml for newer format?

➜  docker-compose-example docker exec dockercomposeexample_datadog_1 cat /var/log/datadog/dogstatsd.log
2016-03-20 19:39:59 UTC | INFO | dd.dogstatsd | utils.pidfile(pidfile.py:31) | Pid file is: /opt/datadog-agent/run/dogstatsd.pid
2016-03-20 19:39:59 UTC | INFO | dd.dogstatsd | daemon(daemon.py:157) | Starting
2016-03-20 19:39:59 UTC | INFO | dd.dogstatsd | daemon(daemon.py:171) | Pidfile: /opt/datadog-agent/run/dogstatsd.pid
2016-03-20 19:39:59 UTC | INFO | dd.dogstatsd | dogstatsd(dogstatsd.py:329) | Listening on host & port: ('', 8125)
2016-03-20 19:39:59 UTC | INFO | dd.dogstatsd | dogstatsd(dogstatsd.py:160) | Reporting to http://localhost:17123 every 10s
2016-03-20 19:40:09 UTC | INFO | dd.dogstatsd | dogstatsd(dogstatsd.py:205) | Flush #1: flushed 0 metrics, 0 events, and 0 service check runs
2016-03-20 19:40:24 UTC | ERROR | dd.dogstatsd | dogstatsd(dogstatsd.py:269) | Unable to post payload.
Traceback (most recent call last):
  File "/opt/datadog-agent/agent/dogstatsd.py", line 259, in submit_http
    r = requests.post(url, data=data, timeout=5, headers=headers)
  File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/api.py", line 107, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/adapters.py", line 449, in send
    raise ReadTimeout(e, request=request)
ReadTimeout: HTTPConnectionPool(host='localhost', port=17123): Read timed out. (read timeout=5)
2016-03-20 19:40:24 UTC | INFO | dd.dogstatsd | dogstatsd(dogstatsd.py:205) | Flush #2: flushed 1 metric, 0 events, and 0 service check runs

docker-compose.yml example

version: "2"
services:
  web:
    build: web
    command: python app.py
    ports:
     - "5000:5000"
    volumes:
     - ./web:/code
    links:
     - redis
  redis:
    image: redis
  datadog:
    build: datadog
    links:
     - redis
    environment:
     - API_KEY=__your_datadog_api_key_here__
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock
     - /proc/mounts:/host/proc/mounts:ro
     - /sys/fs/cgroup:/host/sys/fs/cgroup:ro

Update for more recent docker?

Any chance of updating this example to use a more recent docker version (with networks instead of links)?

Would be awesome to see how this would work with Docker Swarm as well!

Integration with docker-compose that has java application

Hi guys.
Do you know how to integration datadog-agent with docker-compose that has java application ?

Trace: No data received, I can not see any metrics tracked in dashboard as well

Any reason for this? Everything run just fine but I got no traces in APM.

trying to get this to run

updated DD_API_KEY=, and also added DD_SITE=us5.datadoghq.com into docker-compose
fired up the services with docker-compose up

can hit the python app just fine with localhost:5000, but nothing is showing up in DataDog portal under AMP

in the logs, the datadog agent complains about the redis check
datadog-dc-datadog-1 | 2023-12-18 22:48:05 UTC | CORE | ERROR | (pkg/collector/worker/check_logger.go:70 in Error) | check:redisdb

however I see reports of this being a known issue, and wouldn't affect pushing entries over to datadog