ho-cto/sre-monitoring-as-code

Monitoring-as-Code (MaC) is a jsonnet mixin implementation of SLIs/SLO/Error Budgets using the open-source monitoring and alerting eco-system of Prometheus and Grafana.

sre-monitoring-as-code

SRE Monitoring-as-Code (MaC) is a Jsonnet Mixin implementation of SLIs/SLO/Error Budgets using the open-source monitoring and alerting eco-system of Prometheus and Grafana. Our documentation is available to view online.

About the framework

Monitoring Mixins bundle up SLI configuration, Alerting, Grafana dashboards, and Runbooks into a single package. Engineers commit a monitoring definition file and this triggers the packaging of Prometheus Rules and Grafana Dashboards and injects them into the monitoring tools. This way, we can ease up engineers' burden of writing alerting rules, manually drawing up Grafana dashboards, and scribing runbooks.

• Monitoring Mixins¹ are a lightweight flexible configuration, which don’t mandate specific labels or expressions. You can configure and overwrite everything.
• Mixins use data templating language called Jsonnet, which is the only templating language which has fully supported libraries for Grafana and Prometheus.
• jsonnet-bundler is used for package management. Once you have a Monitoring Mixin package, you need to install it, keep track of versions and update them
• SRE MaC will be open-sourced and live on UKHomeOffice GitHub and can be integrated with any Platform which supports pulling containers from GitHub.
• SLI/SLO/Error Budget configurations match Google SRE² industry patterns.

Repository structure

Directory	Description
.githooks/	Contains the client-side pre-commit and pre-push git hooks which form part of our engineering workflow.
.github/	Contains the GitHub Action workflows and associated config.
docs/	Contains the technical documentation for Monitoring-as-Code using Tech Docs Template and Middleman.
example-apps/	Contains example apps to showcase how custom metrics can be shown within the MaC framework.
local/	Contains a docker-compose implementation of Prometheus, Thanos, Grafana and Alertmanager. The purpose of this project is to test Monitoring-as-Code locally with your application.
monitoring-as-code/	Contains the Jsonnet mixin implementation of SLIs/SLO/Error Budgets for Prometheus and Grafana.
security/	Contains the GitLeaks secret scan configuration.

Installation and usage information is provided in a Readme within each of the directories.

Resources

1. Prometheus - Monitoring Mixins
2. Google SRE - Implementing SLOs
3. Google SRE - Setting SLOs: a step-by-step guide
4. Liz Fong-Jones - Adopting SRE and Error Budgets
5. GDS - Run a Service Level Indicator workshop

Licence

Unless stated otherwise, the codebase is released under the MIT License. This covers both the codebase and any sample code in the documentation.

The documentation is © Crown copyright and available under the terms of the Open Government 3.0 licence.

sre-monitoring-as-code's Issues

Refactor run-mixin.sh

Make a variety of small changes to run-mixin:

• make output path flag required
• make changes to improve POSIX shell compatibility
• rename temporary-mixin directory to _input and make it temporary

Change process for MaC

The team need to build the change process for the MaC framework release notes/ release cycle etc and build it into the existing process'

Update root readme link to new url of published tech docs

Update root readme to use new github pages url which has now been fixed since we made repository public.
New url is https://ho-cto.github.io/sre-monitoring-as-code/

Add repo skeleton structure

Thinking of the following structure: -

|-- sre-monitoring-as-code (top level repository)
| |-- components (directory for grouping components)
| | |-- monaco-pg (directory for component a)
| | |-- monaco-dt (directory for component b)
| |-- local (directory for local environment)
| |-- documentation (directory for local environment)

Add monitoring-local base code

Fork bitbucket monitoring-as-code project to GitHub.

import - no match locally or in the Jsonnet library paths

couldn't open import "grafonnet/grafana.libsonnet": no match locally or in the Jsonnet library paths

local grafana = import 'grafonnet/grafana.libsonnet';

Import Grafana link from Github rather than locally using relative paths.

MVP docs part 1

Test comment

Updating deploy script to use docker run

Update Dockerfile and runscripts to include promtool checks

bug with extra bullet point in repo description

Extra bullet point on line 3 of root readme.

Remove sensitive commit history

Remove sensitive commit history for open-sourcing.

Add underscore to output directory

Create a branch for this issue function results in "not found"

Resolved. This was disabled for a limited time whilst the repo visibility was changed from private to public.

Links to internal confluence architecture information

In 'Interpret MaC outputs', search for "Most of the labels are outlined in Designing Service Alerting”. There are couple of sections that have placeholder NOTE text. This information is currently in internal documentation, it will need to be decided where this info is kept and how it's best presented in the external docs.

Create SQS detail dashboard

Add panels and templates to SQS metric libraries for detail dashboard

Add Readme content based on structure detailed

Add ReadMe follow the GitHub standards of: -

• Project's title
• Project's description
• How to install project
• How to use project
• Contribution
• License

Add monitoring use-cases to docs

Detail types of monitoring use-cases and what we would use sre-monaco for and what we would use Dynatrace for.

remove redundant .gitkeep files

gitkeep was being used to retain empty directories on GitHub. These directories have now been populated so these files need to be removed from local/ and monitoring-as-code/

add root readme to repo

docs config.rb file needs updating to use relative assets

Changing the visibility of this repository to public has resulted in the switch to the fixed url https://ho-cto.github.io/sre-monitoring-as-code/. The introduction of a docpath to our site has rendered the CSS stylesheet unuseable. A change is required the config.rb file to use relative assets.

eg https://github.com/edgecase/middleman-gh-pages#project-page-path-issues

container image build and push

Document running middleman in docker

Now that we can run the local docs through middleman in docker the readme should be updated to instruct how this works.

Purge local environment awaiting formal migration from bitbucket

Do not want to maintain two monitoring local environments on Bitbucket and GitHub hence we'll clear down and formally migrated monitoring-as-code and monitoring-local together at retain commit history.

rename components directory

Use single directory called monitoring-as-code instead of nesting under components

Add "Responding to Alerts" section

Add "Responding to Alerts" section which details how users of Monitoring-as-Code should conduct a standard troubleshooting workflow.

Add checks to run-mixin

Add any needed checks for flags to script

Published docs not rendering stylesheets

Changing the visibility of this repository to public has resulted in the switch to the fixed url https://ho-cto.github.io/sre-monitoring-as-code/. The introduction of a docpath to our site has rendered the CSS stylesheet unuseable. A change is required to the tech docs template config file to set the host and service_link as follows: -

host: https://ho-cto.github.io/sre-monitoring-as-code
service_link: /sre-monitoring-as-code

Explain how error budget is calculated with an example.

jsonnet lint

L2T-528 - jsonnet lint.

Builder Dockerfile refinement

Migrate monitoring-local base files to GitHub

Migrate monitoring-local files from Bitbucket to Github without any history.

Consolidate .gitignore files on Github

Move all rules into root directory .gitignore file.

Migrate monitoring-as-code base files to GitHub

Migrate monitoring-as-code files from Bitbucket to GitHub without any history.

Configure dependabot scanning

Configure dependabot scanning (https://github.com/HO-CTO/sre-monitoring-as-code/blob/main/.github/dependabot.yml) for all the required project tooling.

• /local (Docker Compose)
• /workflow (GitHub Actions)
• /monitoring-as-code (Docker / Jsonnet etc)

Add secret scanning to repository

Trial use of Gitleaks action.

Github actions to deploy static docs site to pages

To automate the process of publishing the static site docs to Github pages a GH actions pipeline should be created.

Update Dockerfile to include entrypoint which executes the deploy script

rename docker image to match repo name

Update run mixin shell script to meet Docker friendly tests

SLA's for SAS SRE

The team need to define some SLAs for SAS SRE support and backlog

MVP docs

Document MaC onboarding flow.

jsonnetfmt

L2T-529

SLI Set up

How do teams know what percentage of error to set their SLI's. There should be a level of guidance, as 99.99% doesn't mean that you are measuring the performance of your service realistically or cost effectively. Not all services need that level of accuracy, (unless they are a critical service), as it might cost you more in resource power to keep the service to that level or there may be know issues/ dependency that until they are resolved your service will not be performing to that level to consider.

Business strategies and priorities are continually changing, therefore you will also need to ensure that SLI's are continually reviewed when set up. Primarily collate data for a month and review, once you are confident that the SLI's are at the correct threshold, review every 6 months, continually aligning business priorities.

Update environment template

Update the environment template on Grafana dashboards to only target sli_value environment labels

failed deploy gh pages action

Revert config.rb to switch build to public directory.

Add alerting methodology detail to docs

Add detailed alert methodology and actions to docs.

Add skeleton structure to docs-as-code template

Add skeleton structure as follows -
Introduction and background
Featureset
SLIs/SLOs/Error Budget
How to monitor your service.
Alerting.
Runbooks.

Image in Overview > How MaC works

The image in Overview > How MaC works shows the workflow in more detail than we currently cover in the Overview chapter.

1. Highlight the overview steps of 1. Define SLIs 2. Implement MaC on a local environment 3. Distribute MaC on your environment in the Overview chapter.
2. Move the image from Overview to Get started > Distribute, as the detail in the diagram is about the steps to distribute MaC.

Add MIT License

Add MIT license file for open sourcing using GDS Way guidance https://gds-way.cloudapps.digital/manuals/licensing.html#guidelines-for-repositories-that-are-open-documentation.

SLI menu/metric libraries terminology

It would be good to use consistent terminology for the SLI menu/metric libraries:

In Overview > How MaC Works: change "MaC contains configuration libraries..." to "MaC contains metric libraries (Service-level indicator menu)..." with a link to Service-level indicator menu.