akainocat/tns

Observability Demo App

TNS Observability Demo

A simple three-tier demo application, fully instrumented with Prometheus, Jaeger and Loki logging.

The "TNS" name comes from "The New Stack", where the original demo code was used for an article.

Prerequisites

There are a set of tools you will need to download and install.

Note: You can run this app without kubernetes. See the instructions here.

Docker

This demo assumes you have Docker installed. Follow instructions here for more details.

Note: The k3d kubernetes distribution used for this demo runs as a single node cluster inside docker. Make sure your docker daemon has at least 2.5 GB of total memory available for all pods in this deployment to be scheduled.

k3d

To run this demo, you need a Kubernetes cluster. While the demo should work against any Kubernetes cluster, these docs will assume a locally running k3d cluster. Download and install k3d from here.

kubectl

kubectl is used to interact with Kubernetes clusters. Follow the instructions here to install it.

tanka

Tanka uses the Jsonnet language to interact with Kubernetes, via the kubectl tool. Download and install it from here.

jsonnet-bundler

Jsonnet bundler downloads Jsonnet dependencies. Download and install it from here.). Rename the downloaded binary to jb and move it to the location where $PATH points. Also make sure the binary is executable:

$ chmod +x /usr/local/bin/jb

Instructions

If you wish to use a Kubernetes cluster other than a local k3d one, please adjust these instructions accordingly.

1. Clone TNS repository

$ git clone https://github.com/grafana/tns
$ cd tns

2. Install K3D Cluster

$ ./create-k3d-cluster
$ export KUBECONFIG=$(k3d kubeconfig write tns)

If you see any error like,

permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock

then try to add your user to docker group to run docker without sudo.

$ sudo usermod -aG docker <your-user>

you should logout and login again for the changes to take effect.

3. Install TNS applications This step will ask you to confirm yes four times.

$ ./install

4. Wait It will take some time to install the demo - there's a lot of downloading to do. It is not unreasonable for it to take over ten minutes for everything to download then start up.

This command will show you the status of the cluster:

$ kubectl get pods -A

All pods should be listed as either running or completed. If this is the case, your cluster should be ready for use.

Accessing the Demo

You should now be able to access the demo via http://localhost:8080/.

Demoable things

Metrics -> Logs -> Traces

• Go to the TNS dashboard
• Zoom in on a section with failed requests if you are so inclined
• Panel Drop Down -> Explore
• Choose Datasource Loki
• Choose a log line with a traceID -> Tempo

Metrics -> Traces -> Logs

• Go to Explore
• Choose Datasource Prometheus
• Run this query histogram_quantile(.99, sum(rate(tns_request_duration_seconds_bucket{}[1m])) by (le))
• Click an exemplar
• Click the log icon on a span line

LogQLV2

• Go to Explore
• Choose Datasource Loki
• Run this query {job="tns/app", level="info"} | logfmt | status>=500 and status <=599 and duration > 50ms
• Choose a log line with a traceID -> Tempo

Reviewing the Tanka Code

This installation will have created a tanka directory in your TNS checkout. This directory contains all of the Jsonnet resources used to install this demo. You will now have a tanka directory within your checkout that contains all of the Jsonnet resources that were needed to deploy this monitoring stack. To find out more about Tanka, visit https://tanka.dev.

Disabling/enabling your cluster

Should you wish to disable your cluster, use this command:

$ k3d cluster stop tns

To re-enable it, do this:

$ k3d cluster start tns

Removing the Cluster

Once you have finished with the cluster, this should remove it and leave you ready to recreate it on another occasion:

$ k3d cluster delete tns
$ rm -rf tanka

Troubleshooting

• Tempo 404's when trying to load traces
  • This is likely because the jaeger agent is not running correctly. Check that all pods were successfully scheduled

Development instructions

Updating Go code:

• After modifying source code of the TNS apps, do a make. This will compile the code and tag new images in your local Docker image registry.
• Tell k3d to pull the new images on a pod restart (and not use the image from it's local cache), for example: k3d image import -c tns grafana/tns-app && k3d image import -c tns grafana/tns-db && k3d image import -c tns grafana/tns-loadgen
• Kill the relevant pod(s), for example: kubectl delete pod app-69db48747b-s6qq6 --namespace=tns

Updating Grafana Dashboards and k8s infrastucture:

• You can update the manifests by running tanka: tk apply --force environments/<ENV>/main.jsonnet
• For updating Grafana, for example when changing the dashboards, you can run:tk apply --force environments/default/main.jsonnet