Introduce repeatable alert

[megaease]

Introduce repeatable alert

[haoel]

This is a good issue.

I thought this before I wrote the first line code of EaseProbe, edge trigger or level trigger really put me in a dilemma. And it never disappears from my mind. Thanks to bring this here.

for the level trigger, I have the following concerns

• Most of the time, an accident could cause many services unavailable, and they all keep sending alerts, your phone or other notifiers would be flooded totally.

• The silence duration would be magic work, nobody can set it right.

Maybe, we can use something like "exponential backoff". For example, only sending 5 alerts, the time between them can be deltaTime = 2*deltaTime (deltaTime can be initialized as probe interval)

[samanhappy]

I like the backoff strategy, it's commonly used for handling failed operations in Kubernetes, all we need is to choose a reasonable amount of time and a maximum number of retries.