ifixit/alertinator

Simplistic team-based alerting through email, sms and phone calls.

alertinator's Issues

Alerts: prevent multiples of the same alert

We need a system so that a consistently failing condition does not trigger an alert every time it is ran.

If we are checking for error conditions and triggering alerts every 5 minutes, we do not necessarily want to receive the exact same alert every 5 minutes.

It needs to support:

• Customizable intervals
  • Notify me when the check fails, when it's been failing for at least 30min, 6hours, ... and when it stops.
• Be applied per (check-function, alert level) tuple
  • So as to ensure 5 repeated warnings from one function don't hide the 6th one which is critical
• Resetting counters for a given check after a successful run (Or X successful runs)
• Storing any state locally on the file-system in whatever format makes sense.

Imagined Interface as exposed to the rest of the system:

   // Returns [(bool)$shouldAlert, $firstAlertTimestamp];
   // $shouldAlert is true if an alert should be sent.
   // $firstAlertTimestamp is the unix timestamp of the first
   // occurrence of successive failures (or null if that info
   // is unavailable).
   // 
   // @param $isFailing -- bool indicating if the check is currently failing
   // @param $level -- string identifying alert level
   // @param $checkFunction -- string identifying check function
   protected function shouldAlert($isFailing, $level, $checkFunction);

The default implementation will be simply return [$isFailing, null].

If $shouldAlert is true, the caller of this function will format a nice message using the $firstAlertTimestamp; something like "$alertMessage -- sustained for 30 minutes".

This code should live in a PHP Trait (the default implementation can be in the main class) so this trait can be included in a subclass if the user wants to utilize it.

There should be some unit tests to ensure it operates as expected.

Use full set of syslog severity levels

Unwittingly, I designed Alertinator to use a subset of syslog's severity levels (defined in rfc5424). It seems reasonable to support the full set.

It does seem like perhaps a bit much, though - not only will users need to determine which alert level is appropriate, but it makes user config more complex, particularly since we don't have a good way of specifying "this level and up".

Change All-Clear to alert specific group

Problem

The all-clear notice gets called out to people. People don't like that.

Do This

Add an "all-clear" group to send messages to so users can determine where to send the all-clear, and then default to prior behavior in the absence of the all-clear group.

Expand doc coverage of triggering alerts

Right now you have to piece together the quick look and the API docs to figure out how to write an alert. We also don't at all cover manual triggering (throwing an AlertinatorException from inside other code).

Switch documentation from rst to md, improve examples and readability.

Is there a good reason to use a Python-y doc format for a PHP project on github? Markdown seems like the natural choice.

Alert titles: make more useful

"Threshold of 0 reached ..." Is pretty useless as far as an alert title goes.

Let's get the message in there earlier:
No request stats recorded since ...

There are a number of pieces of data:

• Alert message
• Level
• Number of successive failures

Not sure exactly how those should be arranged, but the current behavior is pretty wonky.

Allow international numbers

We hardcode USA, Let's make numbers internationalize-able