benskelker/detection-rules

Rules for Elastic Security's detection engine

Detection Rules

Detection Rules is the home for rules used by Elastic Security. This repository is used for the development, maintenance, testing, validation, and release of rules for Elastic Security’s Detection Engine.

This repository was first announced on Elastic's blog post, Elastic Security opens public detection rules repo.

Table of Contents

• Overview of this repository
• Getting started
• How to contribute
• Licensing

Overview of this repository

Detection Rules contains more than just static rule files. This repository also contains code for unit testing in Python and integrating with the Detection Engine in Kibana.

folder	description
detection_rules/	Python module for rule parsing, validating and packaging
etc/	Miscellaneous files, such as ECS and Beats schemas
kibana/	Python library for handling the API calls to Kibana and the Detection Engine
kql/	Python library for parsing and validating Kibana Query Language
rta/	Red Team Automation code used to emulate attacker techniques, used for rule testing
rules/	Root directory where rules are stored
tests/	Python code for unit testing rules

Getting started

Although rules can be added by manually creating .toml files, we don't recommend it. This repository also consists of a python module that aids rule creation and unit testing. Assuming you have Python 3.7+, run the below command to install the dependencies:

$ pip install -r requirements.txt
Collecting jsl==0.2.4
  Downloading jsl-0.2.4.tar.gz (21 kB)
Collecting jsonschema==3.2.0
  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
     |████████████████████████████████| 56 kB 318 kB/s 
Collecting requests==2.22.0
  Downloading requests-2.22.0-py2.py3-none-any.whl (57 kB)
     |████████████████████████████████| 57 kB 1.2 MB/s 
Collecting Click==7.0
  Downloading Click-7.0-py2.py3-none-any.whl (81 kB)
     |████████████████████████████████| 81 kB 2.6 MB/s 
...

To confirm that everything was properly installed, run with the --help flag

$  python -m detection_rules --help

Usage: detection_rules [OPTIONS] COMMAND [ARGS]...

  Commands for detection-rules repository.

Options:
  -h, --help  Show this message and exit.

Commands:
  build-release         Assemble all the rules into Kibana-ready release files.
  create-rule           Create a new rule TOML file.
  es                    Helper commands for integrating with Elasticsearch.
  kibana-diff           Diff rules against their version represented in...
  load-from-file        Load rules from file(s).
  mass-update           Update multiple rules based on eql results.
  rule-search           Use EQL to search the rules.
  test                  Run unit tests over all of the rules.
  toml-lint             Cleanup files with some simple toml formatting.
  update-lock-versions  Update rule hashes in version.lock.json file...
  validate-all          Check if all rules validates against a schema.
  validate-rule         Check if a rule staged in rules dir validates...
  view-rule             View an internal rule or specified rule file.

The contribution guide describes how to use the create-rule and test commands to create and test a new rule when contributing to Detection Rules.

How to contribute

We welcome your contributions to Detection Rules! Before contributing, please familiarize yourself with this repository, its directory structure, and our philosophy about rule creation. When you're ready to contribute, read the contribution guide to learn how we turn detection ideas into production rules and validate with testing.

Licensing

Everything in this repository — rules, code, RTA, etc. — is licensed under the Elastic License. These rules are designed to be used in the context of the Detection Engine within the Elastic Security application. If you are using the Elastic Stack with at least the Basic subscription, then you can use these rules for free.

Occasionally, we may want to import rules from another repository that already have a license, such as MIT or Apache 2.0. This is welcome, as long as the license permits sublicensing under the Elastic License. We keep those license notices in NOTICES.txt and sublicense as the Elastic License with all other rules. We also require contributors to sign a Contributor License Agreement before contributing code to any Elastic repositories.

Questions? Problems? Suggestions?

• Want to know more about the Detection Engine? Check out the overview in Kibana.
• This repository includes new and updated rules that have not been released yet. To see the latest set of rules released with the stack, see the Prebuilt rule reference.
• If you’d like to report a false positive or other type of bug, please create a GitHub issue and check if there's an existing one first.
• Need help with Detection Rules? Post an issue or ask away in our Security Discuss Forum or the #detection-rules channel within Slack workspace.