ca-scribner/alertmanager-k8s-operator

Alertmanager Operator (k8s)

This Charmed Operator handles instantiation, scaling, configuration, and Day 2 operations specific to Alertmanager.

This operator drives the Alertmanager application, and it can be composed with other operators to deliver a complex application or service, such as COS Lite.

Alertmanager receives alerts from supporting applications, such as Prometheus or Loki, then deduplicates, groups and routes them to the configured receiver(s).

Getting started

Basic deployment

Once you have a controller and model ready, you can deploy alertmanager using the Juju CLI:

juju deploy --channel=beta alertmanager-k8s

The available channels are listed at the top of the page and can also be retrieved with Charmcraft CLI:

$ charmcraft status alertmanager-k8s

Track    Base                  Channel    Version    Revision    Resources
latest   ubuntu 20.04 (amd64)  stable     -          -           -
                               candidate  -          -           -
                               beta       9          9           alertmanager-image (r1)
                               edge       9          9           alertmanager-image (r1)

Once the Charmed Operator is deployed, the status can be checked by running:

juju status --relations --storage --color

Configuration

In order to have alerts dispatched to your receiver(s) of choice, a configuration file must be provided to Alertmanager using the config_file option:

juju config alertmanager-k8s \
  config_file='@path/to/alertmanager.yml'

Note that if you use templates, you should use the templates_file config option instead of having a templates section in your yaml configuration file. (This is a slight deviation from the official alertmanager config spec.)

Use the templates_file option to push templates that are being used by the configuration file:

juju config alertmanager-k8s \
  config_file='@path/to/alertmanager.yml' \
  templates_file='@path/to/templates.tmpl'

All templates need to go into this single config option, instead of the 'templates' section of the main configuration file. The templates will be pushed to the workload container, and the configuration file will be updated accordingly.

Refer to the official templates documentation for more details.

To verify Alertmanager is using the expected configuration you can use the show-config action:

juju run-action alertmanager-k8s/0 show-config --wait

Dashboard and HTTP API

The Alertmanager dashboard and HTTP API can be accessed at the default port (9093) on the Alertmanager IP address, which is determinable with a juju status command.

To obtain the load-balanaced application IP,

juju status alertmanager-k8s --format=json \
  | jq -r '.applications."alertmanager-k8s".address'

Similarly, to obtain an individual unit's IP address:

juju status alertmanager-k8s --format=json \
  | jq -r '.applications."alertmanager-k8s".units."alertmanager-k8s/0".address'

Clustering

Forming a cluster

Alertmanager supports clustering and all you need to do to create/update a cluster is to rescale the application to the desired number of units using add-unit:

juju add-unit alertmanager-k8s

or using scale-application:

juju scale-application alertmanager-k8s 3

Internally, HA is achieved by providing each Alertmanager instance at least one IP address of another instance. The cluster would then auto-update with subsequent changes to the units present.

Verification

Pebble plan

Cluster information is passed to Alertmanager via --cluster.peer command line arguments. This can be verified by looking at the current pebble plan:

$ juju exec --unit alertmanager-k8s/0 -- \
  PEBBLE_SOCKET=/charm/containers/alertmanager/pebble.socket \
  pebble plan

services:
    alertmanager:
        summary: alertmanager service
        startup: enabled
        override: replace
        command: alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager --web.listen-address=:9093 --cluster.listen-address=0.0.0.0:9094 --cluster.peer=10.1.179.220:9094 --cluster.peer=10.1.179.221:9094

HTTP API

To manually verify a cluster is indeed formed, you can query the alertmanager HTTP API directly:

$ curl -s $ALERTMANAGER_IP:9093/api/v1/status \
  | jq '.data.clusterStatus.peers[].address'
"10.1.179.220:9094"
"10.1.179.221:9094"
"10.1.179.217:9094"

OCI Images

This charm is published on Charmhub with alertmanager images from ubuntu/prometheus-alertmanager, however, it should also work with the official quay.io/prometheus/alertmanager.

To try the charm with a different image you can use juju refresh. For example:

juju refresh alertmanager-k8s \
  --resource alertmanager-image=quay.io/prometheus/alertmanager

(Note: currently, refreshing to a different image only works when deploying from a local charm - lp/1954462.)

Resource revisions

Workload images are archived on charmhub by revision number.

Resource	Revision	Image
alertmanager-image	r1	0.21-20.04_beta

You can use charmcraft to see the mapping between charm revisions and resource revisions:

charmcraft status alertmanager-k8s

Official alertmanager documentation

For further details about Alertmanager configuration and usage, please refer to the official Alertmanager documentation.

Additional Information

• Logging, Monitoring, and Alerting (LMA) - a tutorial for running Prometheus, Grafana and Alertmanager with LXD.
• Alertmanager README
• PromCon 2018: Life of an Alert