adamdecaf/deadcheck

WIP Self-hosted dead mans switch that integrates with PagerDuty

deadcheck

deadcheck is an Operator Presence Control ("OPC") system to alert when an action has not occurred within a predefined window. This is also called a Dead man's switch. deadcheck relies on setting up a schedule with a third-party service and suppressing notifications. That way a failure to check-in with deadcheck or a failure within deadcheck will cause a notification to be triggered. High quality services are chosen as integrations for deadcheck as failures to fire are a known short coming of deadcheck.

Install

Download the latest release for your architecture.

Configuration

checks:
  - id: "<string>"
    name: "<string>"
    description: "<string>"
    schedule:
      every: "<duration>"
    pagerduty:
      apiKey: "<string>"
      escalationPolicy: "<string>"

  - id: "<string>"
    name: "<string>"
    schedule:
      weekdays:
        timezone: "<string>"
        times:
          - start: "<string>"
            end: "<string>"

  - id: "<string>"
    name: "<string>"
    schedule:
      bankingDays:
        timezone: "<string>"
        times:
          - start: "<string>"
            end: "<string>"

Usage

PUT /v1/checks/{id}/check-in

Successful response, or failure. Optional: Extension time.Duration value

Integrations

• PagerDuty: A service is used and incident created but there is a maintenance window preventing notifications. Each successful trigger pushes the maintenance window out longer into the future.

Supported and tested platforms

• 64-bit Linux (Ubuntu, Debian), macOS, and Windows

License

Apache License 2.0 - See LICENSE for details.

deadcheck's Issues

feature: require N check-ins

Often we may be running N instances of a service and expect them all to checkin. If not all services checkin then we'd want to alert somehow.

Use-case: If there are N kafka consumers staging work (e.g. achgateway) and none of them have work to commit (e.g. upload) then we could notify folks to expect zero commited work. (e.g. if there are zero pending files then expect no file uploaded)