g-research/siembol

An open-source, real-time Security Information & Event Management tool based on big data technologies, providing a scalable, advanced security analytics framework.

Siembol

Siembol provides a scalable, advanced security analytics framework based on open-source big data technologies. Siembol normalizes, enriches, and alerts on data from various sources, which allows security teams to respond to attacks before they become incidents.

Introduction

Siembol is an open-source, real-time security information and event management tool developed in-house at G-Research.

Siembol's use cases:

• SIEM Log Collection Using Open Source Technologies

Siembol can be used to centralize both security data collecting and the monitoring of logs from different sources.

• Detection of Leaks and Attacks on Infrastructure

Siembol can be used as a tool for detecting attacks or leaks by teams responsible for the system platform.

For more extensive introduction, visit: Introduction.

Installation

To install locally, visit: Quickstart Guide.

How to contribute

If you wish to contribute to Siembol, first read: Contribution Guide.

Code of Conduct

G-Research has adopted a Code of Conduct that is to be honored by everyone who participates in the Siembol community formally or informally. Please read the full text: Code of Conduct

All notable changes to this project are documented in this file: CHANGELOG

Siembol UI

To learn more about Siembol's UI, visit: Siembol UI.

There you will find guides on:

• Adding a new configuration
• Submitting configurations
• Importing a sigma rule
• Releasing configurations
• Testing configurations
• Testing release
• Adding links to the homepage
• Setting up OAUTH2 OIDC
• Modifying the layout
• Managing applications
• Use ui-bootstrap file
• Filter configs and save searches

Services

To explore Siembol's services, visit: Siembol services.

There you will find guides on:

• Setting up a service in the config editor rest
• Alerting service
• Parsing service
  • Setting up NetFlow v9 parsing
• Enrichment service
  • Setting up an enrichment table
• Response service
  • Writing a response plugin

Deployment

To deploy Siembol, refer to: Siembol deployment.

There you will find guides on:

• Setting up ZooKeeper nodes
• Setting up a GitHub webhook
• Tuning the performance of Storm topologies
• Setting up Kerberos for external dependencies
• Customizing Helm chart

siembol's Issues

Support loading multiple zookeeper nodes in alerting instance

Try to merge rules in order to save resources since the alerting rules can be maintained by different teams. It is needed to eb decided how to avoid conflict in rule names

Importing Sigma rules into siembol alerting

Implement importing Sigma rules https://github.com/SigmaHQ/sigma into siembol alerting instance.

• Implement a rest endpoint in config editor rest for converting a Sigma rule into siembol alerting rule
• Integrate rest endpoint for converting Sigma rules into Siembol UI

Integrate evaluation test cases during submitting a rule/config

A service setting can force users to submit only configs that passed all assertions of related test cases and show warning if no test case is available. This requirements can be configured in a service configuration.

Quickstart guide on AWS

Prepare AWS deployment including writing documentation and scripts.

Try to use Amazon Elastic Kubernetes Service (EKS) for Siembol and its dependencies in the deployment. Analyse to use Fully Managed Apache Kafka – Amazon MSK instead of kafka from siembol/storm charts. Provide requirements for extending Siembol to support MSK authentication.

Try to enable authentication and use some public identity provider compatible with config editor rest and oauth proxy. Consider including dexidp into siembol helm charts in order to integrate public identity providers with Siembol.

Siembol parsing application: routing based on source

Implement routing parsing apps that are mapping a source of the message with the parser

• topic based router uses topic name to select parser
• header router uses Kafka header to select parser

Try to upgrade storm 2.2.0

Modify the code to run on storm 2.2.0. Test and modify kafka clients with compatible version with storm.

Sign docker images that are built in CI/CD pipeline

We should sign docker images before publishing to docker hub

Proposed application.properties Defaults

I'm hacking my way through creating helm charts for config-editor-ui and config-editor-rest and I thought I should write down some thoughts for improvements to the configuration system. I thought I should put my thoughts in the form of a proposal so we can all agree on what we're aiming for before going into the code and trying to make changes there.

Decent Defaults, Everything Customizable

I think that Siembol should default to a basic set of defaults and the user should only have to define the most minimal amount of configuration to get started. All of the customizations should still exist, but only if a user wants to configure custom locations for different repositories and directories.

Just the basics

Ultimately, I would like this as a minimal set of configuration options a user needs to set:

config-editor.service-types=alert,correlationalert,bdp-correlationalert,parserconfig,parsingapp,enrichment
config-editor.config-store.git-user-name=<my_username>
config-editor.config-store.store-repository-name=<my_repo>

Git defaults

Some of the git defaults that we could remove from application.properties would be:

config-editor.config-store.github-url=<default to https://github.com>
config-editor.config-store.git-password=<default_password_from_secret> ## deault this to a `kubectl create secret` name it ${GIT_PSWD}

Auth defaults

Additionally, I think authentication should default to:

config-editor-auth.type=disabled

That way, potential users aren't forced to set up authentication just to kick the tires.

However, not setting config-editor-auth.type throws an error right now. This should probably be a separate issue to make it not error.

Path defaults

For paths on the file system and in the repository, I would propose a couple of things:

• default to one repository (for both *-repository-names)
• default to one directory for the locally saved repository (for both *-repository-paths)
  • I'd love to find a more representative name for this option than "repository-name". It made me think that it was something to do with the remote repository in github for a while. It took me a second to realize that it meant the local file path where the repository was going to be checked out. It's a minor thing, but worth noting.
• put that one default repository in one default "repository-path" that can contain multiple repositories if required. i.e.:
  • /<ephemeral_volume>/siembol-repositories/<repo_name>
• default the subdirectories in each service-specific subdirectory to these three: "rules", "release", "testcases".
  • tiny question: should "release" be "releases" so make the pluralization of these directories match?
• default to the name of each of the service types for the service-specific rules.
  • i.e., for the service type alert, the repository-path of /usr/local/siembol-repositories, and my personal repository of siembol-config-stackedsax, we assume that the directory structure is /usr/local/siembol-repositories/siembol-config-stackedsax/alert.

The end result should look like:

• default remote repo: github://<repo_name>/<service_name>/['rules','release','testcases']
  • i.e.: https://github.com/G-Research/siembol_config/alert/rules
• default local repo: /<ephemeral_volume>/siembol-repositories/<repo_name>/<service_name>/[’rules,‘release’,‘testcases’]
  • i.e. /usr/local/siembol-repositories/siembol-config-stackedsax/alert/rules

UI Config Filename

Lastly, I think the UI Config filename can be relative, not absolute, and default to:

config-editor.services.<service>.ui-config-file-name=ui-config/<service>-layout-config.json

Customizations

I still agree with all the customizations that are available to a user. Being able to specify all of these items per service seems like something that is good to anticipate:

• github url
• git username and password
• store-repository-name
• release-repository-name
• store-repository-path
• release-repository-path
• store-directory
• release-directory
• test-case-directory

Eventually, someone will want to do something like this, so having the ability to override the defaults seems good. It looks like the ability to do this is already available, it's more just a matter of making that a custom, expert-level choice.

My expectation here is that you could choose any of these customizations a la carte. For example, you could choose to have different repositories for store and release, but you could still use the default repository-path and sub-directories without having to specify them, too.

Log Messages

While I worked through application.properties, I felt that the log messages could be improved for:

• Incorrect directory structure
  • This does report an error that the directory could not be found, but it comes amidst a much larger and unnecessary stacktrace
• Missing rules.json
  • I don't recall the stacktrace being very informative as to what was missing here
• Incorrect format for rules.json
  • I only knew this was the issue because Jonathan had mentioned it to me
• Incorrect/missing testcase format
  • I'm just guessing that this is the error I'm currently on
  • Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [uk.co.gresearch.siembol.configeditor.testcase.TestCaseEvaluator]: Factory method 'testCaseEvaluator' threw exception; nested exception is java.lang.NullPointerException

I can move these improvements to a separate issue -- I just wanted to make sure these ideas were captured.

Generate Skeleton Config

While we can suggest that people fork a siembol-config directory with the default structure as a way to get started, there may be plenty of people who don't follow that approach and instead create a bare config directory.

As an alternative approach, could we automatically create a PR on the initial startup that creates a skeleton rules repository? For example, if the default or the custom-specified directories don't exist when the repository if checked out locally, Siembol could create those directories and create a PR that offers to commit those directories in. Further, Siembol could add some default demo rules.json files. All a user would have to do is merge the PR and the next deployment of Siembol would work.

I want to make all these rules and directory creations as simple as possible for a new user, so any more ideas along these lines are welcome.

Try to improve CSS styles in Siembol UI

Config editor UI contains lot of custom styling. Try to analyze to use global css, mixins or other approaches to improve maintenance of siembol ui.

how to deployop siembol

I'm new to siembol
I fllowed the quickstart and depolyop instructions on my macos, while get failed for a lot of errors.

then i search the docker image on hub.docker.com, i don't know which image shoud i install first and what's the relationship between them.

any ideas ,thanks a lot

Simplify parsing app parallelism settings

Currently parallelism of is defined in parsing app settings in these fields:

• input_parallelism
• parsing_parallelism
• output_parallelism
  On the other hand, number of workers is defined in admin configuration. Unifi these configuration and move "number_workers" into parsing app settings. Moreover, use input_parallelism, parsing_parallelism, output_parallelism per one worker in order to not need to compute it in for all workers.

Release with sources once repo is public

Release should include sources jars when this repo is public. Simply add this to the root pom.xml:

    ...
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-source-plugin</artifactId>
        <version>3.2.1</version>
        <executions>
          <execution>
            <id>attach-sources</id>
            <goals>
              <goal>jar-no-fork</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

Siembol UI: Support testing in parser app service

Define test specification, try to mock a parsers in test specification.

Integration tests for springboot applications

Add integration tests for config editor rest, storm topology manager and siembol response project.

Config editor rest: endpoint for restarting all storm topologies

Implement rest API to restart all storm topologies for services in which the user has admin role. Analyze possible integration in Siembol UI

Siembol UI: Display released versions of rules/configs and storm topologies

Display in Siembol UI:

• version of configs synced in Zookeeper
• version of used admin config and topologies that should be released

Siembol UI: Add string replace(all) functionality

Add functionality to replace strings in values of configurations.

Siembol UI string Replace(all) in configuration and test cases

Add functionality to replace one or multiple occurrences of a string in configuration or test case.

Siembol alerting: add description and enabled checkbox to matcher specification

Description can help to better understand complicated regular expression or condition in a matcher. Enabled checkbox should improve testing of the rule. Moreover, some matchers can be part of the rule even if they are not used.

Fix intermittent CI failures during Java unit tests

Our CI sometimes fails with errors encountered whilst running the Java unit tests.

Here is an example:

As a rule of thumb, CI systems are not very reliable for time-sensitive tests. We might want to revisit our various test timeout values?

Siembol UI: implement copy/paste and Undo/Redo

Implement Copy/Paste and Undo/Redo in siembol UI in

• config editor
• admin config editor
• test case editor

[Parser Config] Add optional prefix when using kv extractor

Request: add an optional field which allows all keys extracted using a kv extractor in a parsingconfig to have a prefix added.

Use case: extracting fields from an existing field (which is not the original string) it would be nice to prepend the "parent" field name to the extracted keys so they can all be linked together. Not always necessary/wanted so having it optional would be good. For large parent fields/ changing sub fields it saves having to add long rename transformations

Integrate evaluation test cases during deployment - releasing configs

A service setting can force users to deploy only configs that has at least one test case and passed all assertions. This requirements can be configured in a service configuration.

Siembol-UI: regexes on multiple lines to increase readability

Problem: Currently a regex is shown on one line which is hard to read/troubleshoot when the regex is very long.

Solution: allow regexes on multiple lines where a new line will be ignored

quickstart dependencies

I'm new to siembol.
I fllowed quickstart to install

./dependencies.sh

`error: failed to create configmap: Post "http://localhost:8080/api/v1/namespaces/siembol/configmaps?fieldManager=kubectl-create&fieldValidation=Strict": dial tcp [::1]:8080: connect: connection refused
"bitnami" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "jetstack" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
Error: could not find deployment/helm-k8s/storm: stat deployment/helm-k8s/storm: no such file or directory
Error: INSTALLATION FAILED: repo deployment not found
Error: INSTALLATION FAILED: Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp [::1]:8080: connect: connection refused

---

Checking status by running: 'kubectl get pods -n siembol'
The connection to the server localhost:8080 was refused - did you specify the right host or port?

---

`
Is there any other dependenices?

My host is macos 12.4

Support downloading enrichment tables from a web server

Currently we are supporting downloading enrichment tables from hdfs only. It can be interesting to remove this deployment dependency and allows to download tables from web server too.

Migrate to gresearch on Docker Hub

We should migrate our Docker images to gresearch (instead of gresearchdev currently).

Setup CD release to Maven

Use https://docs.github.com/en/free-pro-team@latest/actions/guides/publishing-java-packages-with-maven to release to Maven.

Will require setting up an account to be used by the workflow, a GPG key pair, a manually triggerable workflow.

Migrate to java 11 in java modules

Try to upgrade dependencies and modify the codebase in order to build and run siembol java apps on java 11

Enable support for CodeQL

CI/CD: latest release and snapshot tags in docker images

We would like to distinguish between lastest built docker images from SNAPSHOT and latest docker images from release versions that should be stable. We can not guarantee stability of snapshot versions.

Config editor ui unstable versions should contain suffix 'dev' and java snapshots have suffix 'SNAPSHOT'. Latest docker images for these version should include the tag 'latest'. Latest docker images for release versions should include the tag 'stable'.

Set-up github actions environments

We would like to ensure integrity of CI/CD pipeline during publishing artifacts into central maven and docker hub. Try to use GHAE https://docs.github.com/en/actions/reference/environments and prepare protection rules with

• deployment branches
• review workflow
• storing secrets

Alerting: Add "Contains" Matcher

Request: Add a new "contains" matcher that searches for a specific substring in a given value

Being able to query for a specific substring (e.g. "user" in "test_user_1") would be very useful as this is something that we use regex for regularly. We could also use this to search using variables that were extracted using regex matching groups.

Potentially also add this to IS_IN_SET, so we can have multiple of these in a single matcher.

This should also have the standard SIEM setup of: "starts with", "ends with" or "contains" so that we can define how it should match.

Siembol response: time condition evaluator

Provide functionally to evaluate time conditions from event timestamp and provided timezone. The evaluator should support hours and days and list of intervals (start, end) for defining time condition.

Siembol UI: Add validators for fields with regex and json path

Use "format" field in json schema for labeling form fields as regular expressions or json path. Try to research for angular form validators for java regex and json paths. Implement the functionality in Siembol UI and modify ui layout files.