hamelin/anomaly-detection-heterogeneous-telemetry

Anomaly detection on heterogeneous time-bound telemetry, using embeddings of graph topology dynamics

Anomaly detection on heterogeneous telemetry, using embeddings of graph topology dynamics

These notebooks implement and experiment with an anomaly detection method applicable to heterogeneous time series of key-value records (denoted here as telemetry). This detection method follows the template of a 2018 paper by Palladino and Thissen, who apply it to streams of low-value cybersecurity alerts from multiple appliances. It also takes into account many of improvements applied to the detector by Element AI.

To demonstrate, study and improve the performance of this methodology, we apply it here to the Los Alamos Cybersecurity dataset. This dataset is composed of four independant streams with distinct event data schemas, correlated in time. The events reported in these streams are composed of artifacts that also appear as part of other streams, making them intersectable.

Setup

1. Have a Python 3.6+ install
2. Create and activate a Python virtual environment, using virtualenv, pipenv, poetry or conda.
3. pip install -r requirements.txt
4. Launch the notebook server.

Notebook summary

Basics

Approximative replication of the methodology described by Palladino and Thissen: graphs composed of all possible categorical/textual artifacts, embeddings derived by ReFEx and RolX, role change analysis to detect anomalies.