jonhiggs/flamingzombies

A lightweight monitoring service

Flaming Zombies

A simple yet extendable, lightweight monitoring daemon.

• Made for the command line.
• Configured with flat files.
• Complete documentation in man pages.
• Few dependencies.
• Stateless¹.
• Easily extended, and customised.
• Liberal 2-clause BSD license.

1. State of course exists, but it isn't persisted between restarts.

---

THIS IS STILL A WORK IN PROGRESS... but it's ready to test.

---

Flaming Zombies ties together three components with three distinct responsibilities; tasks, notifiers and gates.

• tasks check whether a condition is true or false, like if a host responds to pings?
• notifiers raise alerts.
• gates control when a notifier may execute.

sequenceDiagram
    task->>+notifier: new measurement
    notifier->>gate: are you open?
    gate->>+notifier: no
    task->>+notifier: new measurement
    notifier->>gate: are you open?
    gate->>+notifier: yes
    notifier->>Human: notification sent

You're expected to have many tasks. Each task can have one or more notifiers. Each notifier can have one or more gates.

And that is the basis of Flaming Zombies.

Documentation

The complete documentation is available in the man pages. You can read them in your shell before they're installed using using the command:

curl https://raw.githubusercontent.com/jonhiggs/flamingzombies/main/man/man1/fz.1 | man /dev/stdin

Building

Before you can build fz and fzctl, you'll need to have Go installed on your system.

To build, run:

go build ./cmd/fzctl/fzctl.go
go build ./cmd/fz/fz.go

That will produce the binaries for your system. The plugins are at ./libexec and the man pages are at ./man. Adapting the OpenBSD installation instructions should get you a long way to installing it on most UNIX-like system. You may find an init script for your operating system at ./scripts. If you end up writing one, I would appreciate it if you could share it back.

Installation

Installation is intended to be very simple. Eventually, I'd like to provide installation packages, but until then a manual process will need to suffice.

OpenBSD

The below sequence of commands will install the daemon on OpenBSD:

## fz
wget https://github.com/jonhiggs/flamingzombies/releases/download/${VERSION}/fz_openbsd_${ARCH} \
    -O /usr/local/bin/fz

chown root:wheel /usr/local/bin/fz
chmod 755 /usr/local/bin/fz

## fzctl
wget https://github.com/jonhiggs/flamingzombies/releases/download/${VERSION}/fzctl_openbsd_${ARCH} \
    -O /usr/local/bin/fzctl

chown root:wheel /usr/local/bin/fzctl
chmod 755 /usr/local/bin/fzctl

## rc script
wget https://raw.githubusercontent.com/jonhiggs/flamingzombies/main/scripts/openbsd_rc \
    -O /etc/rc.d/flamingzombies

chown root:wheel /etc/rc.d/flamingzombies
chmod 755 /etc/rc.d/flamingzombies

## plugins
wget https://github.com/jonhiggs/flamingzombies/releases/download/${VERSION}/plugins.tar.gz \
    -O /tmp/plugins.tar.gz

tar -C /usr/local/libexec -zxvf /tmp/plugins.tar.gz
rm /tmp/plugins.tar.gz

## man pages
for m in man1/fz.1 man1/fzctl.1 man5/flamingzombies.toml.5 man7/fz-gates.7 man7/fz-notifiers.7 man7/fz-tasks.7; do
    wget https://raw.githubusercontent.com/jonhiggs/flamingzombies/main/man/$f \
        -O /usr/local/man/$f
done

## config
# create a configuration at /etc/flamingzombies.toml
# see the flamingzombies.toml(5) man page.

## enable the daemon
rcctl enable flamingzombies
rcctl set flamingzombies logger daemon.info

Alpine Linux

The below sequence of commands will install the daemon on Alpine Linux:

## fz
wget https://github.com/jonhiggs/flamingzombies/releases/download/${VERSION}/fz_linux_${ARCH} \
    -O /usr/local/bin/fz

chown root:root /usr/local/bin/fz
chmod 755 /usr/local/bin/fz

## fzctl
wget https://github.com/jonhiggs/flamingzombies/releases/download/${VERSION}/fzctl_linux_${ARCH} \
    -O /usr/local/bin/fzctl

chown root:root /usr/local/bin/fzctl
chmod 755 /usr/local/bin/fzctl

## rc script
wget https://raw.githubusercontent.com/jonhiggs/flamingzombies/main/scripts/openrc \
    -O /etc/init.d/flamingzombies

chown root:root /etc/init.d/flamingzombies
chmod 755 /etc/init.d/flamingzombies

## plugins
wget https://github.com/jonhiggs/flamingzombies/releases/download/${VERSION}/plugins.tar.gz \
    -O /tmp/plugins.tar.gz

mkdir -p /usr/local/libexec
tar -C /usr/local/libexec -zxvf /tmp/plugins.tar.gz
rm /tmp/plugins.tar.gz

## man pages
for i in $(seq 1 7); do
    mkdir -p "/usr/local/man/man$i"
done

for m in man1/fz.1 man1/fzctl.1 man5/flamingzombies.toml.5 man7/fz-gates.7 man7/fz-notifiers.7 man7/fz-tasks.7; do
    wget https://raw.githubusercontent.com/jonhiggs/flamingzombies/main/man/$f \
        -O /usr/local/man/$f
done

## config
# create a configuration at /etc/flamingzombies.toml
# see the flamingzombies.toml(5) man page.

## enable the daemon
rc-update add flamingzombies
service flamingzombies start

flamingzombies's Issues

Defer notifications

Add support to defer a notification allowing it some time to auto-resolve before notifying.

Send the failing gateset to the notifier environment

A flappy monitor could have any status. A hint that the the flap gate tripped is needed.

Make the tasks support the 124 exit code

As documented in fz-tasks(7).

Consider chaning the transport between fz and fzctl to HTTP.

A raw tcp port has problems like:

• you can't use your load balancers to route it.
• I suspect it's currently single threaded, and possibly breaking the fz_errors task.

support `fzctl help` command

fzctl --help is awkward when you're in a cmd subcmd mode of thinking.

Log stderr from task scripts

task to check that filesystem is mounted

include a bash_completion script for fzctl

Support configurable `from` address for the email notifier.

This should probably be configured by sending custom environment variables back to the notifier.

Run as an unprivileged user

Allow querying the notifier details with fzctl

This will probably require splitting out tasks and notifier actions with a new subcommand.

task to check that directory contains new files

From a directory of backup files, make sure that the newest one is at newer than n seconds.

If the directory doesn't exist, return 3.
If the glob doesn't match anything, return 1.
If none of the matching files are new enough, return 1.

Add a directory of example configurations

Demonstrate how it can be used with a set of common and less common examples.

email notifier content

Print more useful information in the email notifications.

`error_count` doesn't increase when error is encounterd

Detect flapping tasks

Add a way to notify when a task is in a flapping state.

Consider when to reset `LastResultOutput`

I've noticed situations where tasks are failed, but the notification presents a successful output.

Add more example configuration files.

Support listening on multiple network interfaces

add task to check swap usage

Document flamingzombies.toml

installation instructions for debian

fz task plugin

Add a task plugin to detect if an fz daemon has experienced any errors for any of its tasks.

Add grace to the timeouts

Adding grace time to the timeouts makes managing the timeout in the tasks easier; you don't have to deduct any time from ${TIMEOUT} to avoid losing to the fz's context deadline.

add task to check disk inodes free

add task to measure tls expiration

don't segfault if a gate's script doesn't exist.

time="2024-05-01T13:40:44+10:00" level=info msg="executing task" file=lib/fz/task.go task_hash=2103319012 task_name=flappy
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x52c115]

goroutine 6 [running]:
github.com/jonhiggs/flamingzombies/lib/fz.Gate.IsOpen({{0xc000014520, 0xc}, {0xc00001c1a0, 0x1a}, {0x0, 0x0, 0x0}}, 0xc0000c6000)
        /home/jon/src/flamingzombies/lib/fz/gate.go:67 +0xa95
github.com/jonhiggs/flamingzombies/lib/fz.ProcessNotifications.func1()
        /home/jon/src/flamingzombies/lib/fz/notification.go:26 +0x10c7
created by github.com/jonhiggs/flamingzombies/lib/fz.ProcessNotifications in goroutine 1
        /home/jon/src/flamingzombies/lib/fz/notification.go:20 +0x1a
exit status 2

Configurable `deadline_grace` and docs

Make the deadline_grace configurable, and ensure that it plus the timeout isn't > the frequency or retry_frequency.

make sure task names don't contain spaces

makes working with the output of fzctl harder for no need.

Validate that notifiers don't reference unknown gates

Order the lists in the man pages

They're unsorted everywhere.

add task to check load average

Tasks need to include data in the notifications

It's hard to tune the tasks if you don't know by how much they're failing by.

Repeat notifications

Add a gate to renotify periodically.

Its too hard to chase the source of errors when using the `fz_error` task.

Work out how to make debugging easier when the tests are distributed across hosts.

Fix the initial value for a `task.LastState` to "unknown"

Support task decriptions

Jamming everythign into the name is getting difficult.

add task to check the age of the file

fails if it gets too old.

Add a deadline to fzctl

it will allow for a more reliable task/fz_errors plugin.

installation instructions for alpine linux

OpenBSD installation instructions

add task to check Eaton UPS

Support a config directory (`/etc/flamingzombies.d/`)

All files in the config directory gets concatenated to to the end of the config file.

AND and OR the gates.

Make the notification gates parameter behave as documented.

Support passing env vars to tasks

I've yet to need this, but I expect that sooner or later I will.

add task to check disk space free