nomad-prometheus-playground

The goal of this repository is to create a playground of a custom cluster using the hashi-stack (nomad and consul) and getting metrics automatically from prometheus in order to visualize everything using grafana

Repository structure and building

The whole functionality has been splitted into different folders representing each entity:

hashi-base-docker

This is an ubuntu18.04 based image with both Consul and Nomad binaries already installed. Just build it:

cd hashi-base-docker && docker-compose build

cluster-client-docker

Using the previously built image, this image represent a client in the cluster that includes the Consul and Nomads services running at the same time. The variables $CONSUL_MASTER_ADDR and $NOMAD_MASTER_ADDR are configured in runtime. The telemetry stanza is configured to report nomad status to prometheus.

cd cluster-client-docker && docker-compose build

cluster-server-docker

Same concept for the server machines. In this case, our cluster will have a single server (bootstrap_expect = 1)

cd cluster-server-docker && docker-compose build

prometheus-scraper

We will use a golang application to create a custom metric and send it to prometheus. In this case, we will ping our client-X instances and push the RTT (time from ping command) to prometheus every 5 seconds. The docker image uses the official golang image and compiles the code and generate the binary.

cd prometheus-scraper && docker-compose build

prometheus-docker

Using a multistage build from the original prometheus docker image, a custom image is created in order to use a template for the config file in runtime.

cd prometheus-docker && docker-compose build

Running the example

After all the required images have been generated, just start everything from the root docker-compose.yaml file

docker-compose up

We use the default bridge network, and let the server be the first instantiated docker (172.17.0.2). From now on, the following services are available from your browser:

juaruipav / nomad-prometheus-playground Goto Github PK