chanzuckerberg/prometheus-demo

Simulates monitoring HTTP microservice and short-lived job metrics

Prometheus Demo

This repo simulates monitoring HTTP microservices and short-lived batched jobs with Prometheus.

Grafana is deployed alongside Prometheus with sample dashboards for visualizing the data.

Setup

Before running any of the demonstrations below, you need to build the docker image for running the test kv_store and aggregator applications.

make image

To start the servers, use docker-compose.

docker-compose up

You will have to wait a few minutes before metrics start to appear on the dashboards described in the following sections.

Monitoring HTTP microservices

The docker-compose.yaml configuration sets up three instances of an HTTP key/value store microservice (similar to memcached).

• The code that implements of the key/value store with prometheus instrumentation in kv_store.py.
• The code that synthesizes traffic on these instances can be found in simulate_http_traffic.py.

The Grafana server is preconfigured with the dashboard for HTTP metrics shown below.

Monitoring short-lived jobs

The docker-compose.yaml configuration sets up a metrics aggregator that short-lived jobs can send their metrics to.

• The code that implements the metrics aggregator lives in aggregator.py.
• The code that synthesizes batch job metrics is in simulate_jobs.py

The Grafana server is preconfigured with the dashboard job metrics shown below.

Operating the servers

You will then have the following servers running.

• kv_store_server_0
• kv_store_server_1
• kv_store_server_2
• prometheus
• grafana
• aggregator

You can check the metrics collection status of the kv_store servers on the Prometheus targets page.

On the grafana server there is a sample HTTP dashboard showing off some standard http metrics.

Teardown

Shut it down when you're done!

docker-compose down

prometheus-demo's Issues

errors in "make image"

Hi,

just wanted to try this repo and executed make image, after a while I get this error:

...
Successfully installed pip
Cleaning up...
Removing intermediate container a75c30c9217d
 ---> 65aff6ff7aae
Step 6/11 : RUN pip3 install --upgrade -r requirements.txt
 ---> Running in c3f076ee4b89
Traceback (most recent call last):
  File "/usr/local/bin/pip3", line 7, in <module>
    from pip._internal.cli.main import main
  File "/usr/local/lib/python3.4/dist-packages/pip/__init__.py", line 1, in <module>
    from typing import List, Optional
ImportError: No module named 'typing'
The command '/bin/sh -c pip3 install --upgrade -r requirements.txt' returned a non-zero code: 1
make: *** [Makefile:8: image] Fehler 1