rvadim/elastalert-k8s-automation

Use power of Kubernetes to automation configuration of elastalert

elastalert-k8s-automation's Issues

Elastalert not handle unkonwn id for notifiers

Admin part:

es_host: <host>
es_port: 9200
run_every:
  seconds: 60
writeback_index: logstash-kube-spb-test
buffer_time:
  minutes: 45
alert_configs:
  slack:
    default: to_vadim
    configs:
      to_vadim:
        slack_webhook_url: <url>

User part:

    name: ingress-alerts
    type: frequency
    index: <index>
    use_strftime_index: true
    num_events: 1
    timeframe:
      minutes: 5
    realert:
      minutes: 0
    #include: ["@timestamp", "fingerprint", "host", "index_name", "message", "source"]
    filter:
    - query_string:
        query: 'kubernetes.namespace: ingress-nginx AND status: 500'
    alert:
      - slack
    slack_id: to_vedim <- typo

Error:

elastalert.util.EAException: Error loading file /config/rules/rule_0.yaml: Error initiating alert ['slack']: Missing required option(s): slack_webhook_url

Describe how to install and configure product in README (admin part)

Read configuration from configmaps in user namespaces

• Read configuration files from configmaps in all available namespaces
• Create error handling and test coverage for configs

General options for alerts in the admin config

Add the ability to set general options for alerts in the admin config

• Add an alert_configs field to the admin config
• Add the config_name field to user rules
• When reading user rules, replace config_name with the corresponding options from the admin config

Use argparse library instead of ENV var for all none options

Priorities:

• command line argument - highest priority
• ENV var - used if no command line argument specified
• config.yaml - lowest priority

Describe how to use product in README (user part)

Add support for all ElastAlert configuration and rule properties

For now we added support only for a few properties of ElastAlert just for testing our app.

In the future it is necessary to add support for all possible ElastAlert properties.

This task requires:

• adding validation checks for parsed admin and user configs.
  Note: some properties require existing of some other properties that are not required by default.
• adding properties into jinja templates (admin template and rule template).

Documentation;
Admin Options
Rule Types and Configuration Options
About Rule Types

Setup project and automation

DoD:

• Setup initial project with tox which run flake8 and pytests
• Fix versions in requirements.txt
• Setup main function which can read 'in-cluster' kubernetes configuration
• Prepare Dockerfile for application
• Prepare docker-compose.yaml file for docker image building and pushing
• Write simple docs about how to get running project in minikube

Read configuration from configmap and secret data from Secret(env)

• Admin configuration will be stored in configmap in yaml
• All credentials to relay of slack tokens should be read from env(Kubernetes Secret map)
• Yaml configuration from file should have error handling and test coverage

Refactoring and testing

Refactoring is needed in order to simplify main file and split functions on different modules.

• Take out functions for reading configuration files into classes:
  -- LocalConfigReader for reading configs in local environment. Initialized by local path to user configurations.
  -- RemoteConfigReader for reading configs in Kubernetes namespaces. Initialized by Kubernetes cluster configuration.
• Take out functions for generating configurations via jinja templates into special class Renderer
• Test created entities.

Image Pull Error

Hi rvadim.

Your Image rvadim/elastalert-k8s-automation:elastalert-latest was not found. Does it removed? Many apologies I'm too late to clone the image docker before. Could you help?

Watch for changes in config maps instead of read them

Research: Elastalert configuration and running

DoD:

• Elastalert minimal configuration described
• Elastalert configuration with one rule and slack integration described
• What configuration we should keep in 'admin' part, what configuration we should delegate to 'user' part
• We know answer on: Can we use jinja2 templates for building configuration, or there are some issues?
• We know answer on: Can we use yaml for configuration parts or we should use another format?
  At the end:
• Configuration example for 'admin' part
• Configuration example for 'user' part