Automatically Define and Enable alarms on a subdirectory

[influxdata]

Automatically Define and Enable alarms on a subdirectory

[nathanielc]

@apsey How would you provide the other information a task needs, like dbrp, type, or possible vars?

[apsey]

Something similar to: https://github.com/rossmcdonald/kapacitor/blob/4db1c8c4dfe6dd9d383c2e15feaa357decf38eff/test.yml

[apsey]

Or maybe kapacitor could have another way of defining alerts at start-up that does not depend on running external commands.

[jsternberg]

@nathanielc do you think that's reasonable to build this feature directly into kapacitor or should we make a simple entrypoint script in python to do this?

[nathanielc]

@jsternberg Building this into Kapacitor itself is not a good idea, the daemon could be running in many different environments that would make doing this reliably difficult.

Adding a script to this docker conatiner would be much better since the environment is known.

[apsey]

I've added this to my dockerfile wrapper:

#!/bin/bash
# entrypoint.sh
set -ex

TICK_METADATA="/etc/kapacitor/tick_metadata.json"
TICK_ALARMS_FOLDER="/etc/kapacitor/alarms/"

function define_alarm {
  `kapacitor define $1 -type $2 -tick $TICK_ALARMS_FOLDER$3 -dbrp $4`
}
export -f define_alarm

function enable_alarm {
  `kapacitor enable $1`
}
export -f enable_alarm

function configure_all_alarms {
  sleep 60
  alarms=`jq -r '.[]|.name + " " + .type + " " + .tick + " " + .dbrp' $TICK_METADATA`
   while read b; do
    define_alarm $b
    enable_alarm $b
  done <<< "$alarms"
}

if [ "${1:0:1}" = '-' ]; then
    set -- kapacitord "$@"
fi

exec "$@" &
pid="$!"

# configure alarms
configure_all_alarms

# wait indefinitely
while true; do
    tail -f /dev/null & wait ${!}
done

ENV KAPACITOR_VERSION 1.0.0~beta3
RUN wget -q https://dl.influxdata.com/kapacitor/releases/kapacitor_${KAPACITOR_VERSION}_amd64.deb.asc && \
    wget -q https://dl.influxdata.com/kapacitor/releases/kapacitor_${KAPACITOR_VERSION}_amd64.deb && \
    gpg --batch --verify kapacitor_${KAPACITOR_VERSION}_amd64.deb.asc kapacitor_${KAPACITOR_VERSION}_amd64.deb && \
    dpkg -i kapacitor_${KAPACITOR_VERSION}_amd64.deb && \
    rm -f kapacitor_${KAPACITOR_VERSION}_amd64.deb*

RUN apt-get update --fix-missing && apt-get install -yq --no-install-recommends --force-yes \
    sqlite3 \
    vim \
    jq \
    && apt-get clean && \
    rm -rf /var/lib/apt/lists/*

EXPOSE 9092

VOLUME /var/lib/kapacitor

COPY kapacitor.conf /etc/kapacitor/kapacitor.conf
COPY tick_metadata.json /etc/kapacitor
COPY alarms /etc/kapacitor/alarms
COPY entrypoint.sh /opt/local/kapacitor/entrypoint.sh
ENTRYPOINT ["/opt/local/kapacitor/entrypoint.sh"]
CMD ["kapacitord"]

Then I have a folder called alarms with files, such as alarms/kafka_in_messages.tick:

stream
    |from()
        .database('prod_kafka')
        .retentionPolicy('default')
        .measurement('in-message-count')
    // Trigger critical alert if the throughput drops below 100 points per 300s and checked every 300s.
    |deadman(100.0, 300s, lambda: hour("time") >= 4 AND hour("time") <= 23)
        .slack()
        .channel('#channel')