aws-samples/siem-on-amazon-opensearch-service

A solution for collecting, correlating and visualizing multiple types of logs to help investigate security incidents.

SIEM on Amazon OpenSearch Service

View this page in Japanese (日本語) | Chinese (简体中文) | Chinese (繁體中文)

SIEM on Amazon OpenSearch Service is a solution for collecting multiple types of logs from multiple AWS accounts, correlating and visualizing the logs to help investigate security incidents. Deployment is easily done with the help of AWS CloudFormation or AWS Cloud Development Kit (AWS CDK), taking only about 30 minutes to complete. As soon as AWS services logs are put into a specified Amazon Simple Storage Service (Amazon S3) bucket, a purpose-built AWS Lambda function automatically loads those logs into SIEM on OpenSearch Service, enabling you to view visualized logs in the dashboard and correlate multiple logs to investigate security incidents.

Jump to | Configuring AWS Services(Log Sources) | Changing Configurations of SIEM on OpenSearch Service | Advanced Deployment | OpenSearch Serverless | Dashboard | AWS Control Tower Integration | Amazon Security Lake Integration | Supported Log Types | FAQ | Changelog |

Architecture

AWS Control Tower Integration

Amazon Security Lake Integration

Supported Log Types

SIEM on OpenSearch Service can load and correlate the following log types.

	AWS Service	Log
Security, Identity, & Compliance	AWS CloudHSM	HSM audit logs
Security, Identity, & Compliance	Amazon GuardDuty	GuardDuty findings
Security, Identity, & Compliance	Amazon Inspector	Inspector findings
Security, Identity, & Compliance	AWS Directory Service	Microsoft AD
Security, Identity, & Compliance	AWS WAF	AWS WAF Web ACL traffic information AWS WAF Classic Web ACL traffic information
Security, Identity, & Compliance	AWS Security Hub	Security Hub findings GuardDuty findings Amazon Macie findings Amazon Inspector findings AWS IAM Access Analyzer findings
Security, Identity, & Compliance	AWS Network Firewall	Flow logs Alert logs
Management & Governance	AWS CloudTrail	CloudTrail Log Event CloudTrail Insight Event
Management & Governance	AWS Config	Configuration History Configuration Snapshot Config Rules
Management & Governance	AWS Trusted Advisor	Trusted Advisor Check Result
Networking & Content Delivery	Amazon CloudFront	Standard access log Real-time log
Networking & Content Delivery	Amazon Route 53 Resolver	VPC DNS query log
Networking & Content Delivery	Amazon Virtual Private Cloud (Amazon VPC)	VPC Flow Logs (Version5) Text / Parquet Format
Networking & Content Delivery	AWS Transit Gateway	VPC Flow Logs (Version6) Text / Parquet Format
Networking & Content Delivery	Elastic Load Balancing	Application Load Balancer access logs Network Load Balancer access logs Classic Load Balancer access logs
Networking & Content Delivery	AWS Client VPN	connection log
Storage	Amazon FSx for Windows File Server	audit log
Storage	Amazon Simple Storage Service (Amazon S3)	access log
Database	Amazon Relational Database Service (Amazon RDS)	Amazon Aurora(MySQL) Amazon Aurora(PostgreSQL) Amazon RDS for MariaDB Amazon RDS for MySQL Amazon RDS for PostgreSQL
Database	Amazon ElastiCache	ElastiCache for Redis SLOWLOG
Analytics	Amazon OpenSearch Service	Audit logs
Analytics	Amazon Managed Streaming for Apache Kafka (Amazon MSK)	Broker log
Compute	Linux OS via CloudWatch Logs	/var/log/messages /var/log/secure
Compute	Windows Server 2012/2016/2019 via CloudWatch Logs	System event log Security event log
Containers	Amazon Elastic Container Service (Amazon ECS) via FireLens	Framework only
End User Computing	Amazon WorkSpaces	Event log Inventory
Open Source Software	Apache Web Server	access log(CLF, combined, combinedio with XFF) error log
Open Source Software	NGINX Web Server	access log(combined with XFF) error log

Experimental Support: We may change field type, normalization and something in the future.

Supported logs are normalized in accordance with the Elastic Common Schema. Click here to see the correspondence table of the original and normalized field names for the logs.

Contribution

Product/Service	Pull Request	Doc	Contributor
TrendMicro Deep Security	#27	README	@EijiSugiura
Okta audit log	#168	README	@yopiyama

Dashboard

See this

Getting Started

In this tutorial, you will create a publicly accessible SIEM on OpenSearch Service domain using a CloudFormation template. See Advanced Deployment if you need to deploy it within an Amazon VPC or need to customize it.

You can add country information as well as latitude/longitude location information to each IP address. To get location information, SIEM on OpenSearch Service downloads and uses GeoLite2 Free by MaxMind. If you want to add location information, get your free license from MaxMind.

Threat information can be enriched based on IP addresses and domain names (EXPERIMANTAL). Threat information sources include your own IoCs (Indicators of compromise) in TXT and STIX 2.x formats, Tor Project, Abuse.ch Feodo Tracker, AlienVault OTX. If there are many IoCs, the processing time of Lambda will increase, so please select IoCs carefully. If you want to use the IoC on AlienVault OTX, please get your API key at AlienVault OTX. See Threat Information Enrichment by IoC for more details.

Note: The CloudFormation template will deploy OpenSearch Service with a t3.medium.search instance. It's not the AWS Free Tier. Change it to an instance type that can deliver higher performance than t3 when using SIEM in the production environment as it requires higher processing power when aggregating many logs. Use the AWS Management Console to change the instance type, extend the volume, or use UltraWarm. This is because the CloudFormation template for SIEM on OpenSearch Service is designed for the initial deployment purpose only, and cannot be used for management purposes like changing/deleting nodes.

1. Quick Start

Choose a region where you want to deploy SIEM on OpenSearch Service from the following. If there is no region below, please check CloudFormation Template For All Regions.

Region	CloudFormation	Template URL
US East (N. Virginia) us-east-1		https://aes-siem-us-east-1.s3.amazonaws.com/siem-on-amazon-opensearch-service.template
US West (Oregon) us-west-2		https://aes-siem-us-west-2.s3.amazonaws.com/siem-on-amazon-opensearch-service.template
Asia Pacific (Tokyo) ap-northeast-1		https://aes-siem-ap-northeast-1.s3.amazonaws.com/siem-on-amazon-opensearch-service.template
Asia Pacific (Singapore) ap-southeast-1		https://aes-siem-ap-southeast-1.s3.amazonaws.com/siem-on-amazon-opensearch-service.template
Europe (Frankfurt) eu-central-1		https://aes-siem-eu-central-1.s3.amazonaws.com/siem-on-amazon-opensearch-service.template
Europe (London) eu-west-2		https://aes-siem-eu-west-2.s3.amazonaws.com/siem-on-amazon-opensearch-service.template

Or you can create your own template by the steps.

2. Configuring OpenSearch Dashboards

It will take about 30 mins for the deployment of SIEM on OpenSearch Service to complete. You can then continue to configure OpenSearch Dashboards.

1. Navigate to the AWS CloudFormation console, choose the stack that you've just created, and then choose "Outputs" from the tab menu at the top right. You can find your username, password, and URL for OpenSearch Dashboards. Log into OpenSearch Dashboards using the credentials.
2. When you login for the first time, [Select your tenant] is displayed. Select [Global]. You can use the prepared dashboard etc.
3. You can also select [Private] instead of [Global] in [Select your tenant] and customize configuration and dashboard etc. for each user. The following is the procedure for each user. If you select Global, you do not need to set it.
   1. To import OpenSearch Dashboards' configuration files such as dashboard, download saved_objects.zip. Then unzip the file.
   2. Navigate to the OpenSearch Dashboards console. Click on "Stack Management" in the left pane, then choose "Saved Objects" --> "Import" --> "Import". Choose dashboard.ndjson which is contained in the unzipped folder. Then log out and log in again so that the imported configurations take effect.

3. Loading logs into OpenSearch Service

All you need to do to load logs into SIEM on OpenSearch Service is PUT logs to the S3 Bucket named aes-siem-<YOUR_AWS_ACCOUNT>-log. Then the logs will be automatically loaded into SIEM on OpenSearch Service. See this for detailed instructions on how to output AWS services logs to the S3 bucket.

Workshop

We have published the workshop, SIEM on Amazon OpenSearch Service Workshop. In this workshop, we will build the SIEM, ingest AWS resource logs, learn OpenSearch Dashboards, investigate security incident, create dashboard, configure alerts and ingest logs of Apache HTTPD server.

Updating SIEM

If you want to update "SIEM on OpenSearch Service" to the latest version, upgrade the OpenSearch / Elasticsearch domain and then update it in the same way as you did for the initial setup (using CloudFormation or AWS CDK.) You can view the changelog of SIEM here.

Note: When you update SIEM, Global tenant settings, dashboards, etc. will be overwritten automatically. The configuration files and dashboards used before the update will be backed up to aes-siem-[AWS_Account]-snapshot/saved_objects/ in the S3 bucket, so restore them manually if you want to restore the original settings.

Note: S3 bucket policy, KMS key policy, IAM policy, etc. are automatically generated by CDK/CloudFormation. Manual modification is not recommended, but if you have modified it, it will be overwritten, so please back up each and update the difference after updating. Or when updating CDK/CloudFormation, keep the current bucket policy by setting the parameter LogBucketPolicyUpdate to keep

Upgrading the OpenSearch Service domain

Upgrade the domain to OpenSearch 1.0 - 2.11 or Elasticsearch version 7.10. Some Dashboards assume OpenSearch Service 1.3 or higher. The recommended version is OpenSearch Service 2.11 with "Enable compatibility mode":

1. Navigate to the OpenSearch Service console
2. Choose domain: [aes-siem]
3. Choose [Actions] icon, and choose [Upgrade domain] from the drop-down menu
4. For "Version to upgrade to", choose [OpenSearch 2.11] (Recommended), [OpenSearch 1.0 - 2.9], or [Elasticsearch 7.10]
5. Choose "Enable compatibility mode" (Recommended)
6. Then choose [Submit]

If you completed the initial setup using CloudFormation, move on to the next step. If you completed the initial setup using the AWS CDK, see
"Updating SIEM with the AWS CDK" section in Advanced Deployment.

Updating the CloudFormation stack

You can update the CloudFormation stack by specifying the CloudFormation template below:

https://aes-siem-<REGION>.s3.amazonaws.com/siem-on-amazon-opensearch-service.template

1. Navigate to the CloudFormation console
2. Choose stack [aes-siem]
3. Choose [Update] at the right top on the screen
4. In Update stack, choose the following:
   • Prepare template: [Replace current template]
   • Template source: [Amazon S3 URL]
   • Amazon S3 URL:
   • Choose [Next]
5. Leave all the other settings as default, and continue to click Next to complete.

Updating is now complete.

Changing Configurations

Changing the OpenSearch Service domain resources after deployment

If you want to make changes to the OpenSearch Service domain itself such as changing the access policy of OpenSearch Service, changing the instance type, changing the Availability Zone or adding a new one, or changing to UltraWarm, perform the change from the OpenSearch Service console of AWS Management Console.

Managing the index and customizing SIEM

SIEM on OpenSearch Service saves logs in the index and rotates it once a month. If you want to change this interval or load logs from non-AWS services, see this.

Near-real-time logs loading from non-SIEM-managed S3 buckets

If you have an S3 bucket in the same account and region as the SIEM, you can load logs into OpenSearch Service. Please refer Near-real-time loading from other S3 buckets for the setting method.

Loading stored logs through batch processing

You can execute es-loader, which is a python script, in the local environment to load past logs stored in the S3 bucket into SIEM on OpenSearch Service. See Loading past data stored in the S3 bucket for details.

Throttling of es-loader in an emergency

To avoid unnecessary invocation of es-loader, throttle es-loader under the following conditions

• If total free space for the OpenSearch Service cluster remains less than 200MB for 30 minutes and aes-siem-TotalFreeStorageSpaceRemainsLowAlarm is triggered.
  • The OpenSearch cluster is running out of storage space. More free space is needed for recovery. To learn more, see Lack of available storage space.

If you want to resume loading logs, set the reserved concurrency of the Lambda function aes-siem-es-loader back to 10 from the AWS Management Console or AWS CLI. You can also load messages from the dead-letter queue (aes-siem-dlq) by referring to Loading data from SQS Dead Letter Queue.

AWS resources created by the CloudFormation template

Below is the list of AWS resources created by the CloudFormation template. AWS Identity and Access Management (IAM) resources can be found from the AWS Management Console.

AWS Resource	Resource Name	Purpose
OpenSearch Service	aes-siem	SIEM itself
S3 bucket	aes-siem-[AWS_Account]-log	For collecting logs
S3 bucket	aes-siem-[AWS_Account]-snapshot	For capturing manual snapshots of OpenSearch Service
S3 bucket	aes-siem-[AWS_Account]-geo	For storing downloaded GeoIPs
Step Functions	aes-siem-ioc-state-machine	For downloading IoC and creating database
Lambda function	aes-siem-aws-api-caller	CDK/CloudFormation custom resource to make an AWS API call
Lambda function	aes-siem-ioc-plan	For creating map to download IoC
Lambda function	aes-siem-ioc-createdb	For downloading IoC
Lambda function	aes-siem-ioc-download	For creating IoC Database
Lambda function	aes-siem-geoip-downloader	For downloading GeoIPs
Lambda function	aes-siem-es-loader	For normalizing logs and loading them into OpenSearch Service
Lambda function	aes-siem-es-loader-stopper	For throttling es-loader in case of emergency
Lambda function	aes-siem-deploy-aes	For creating the OpenSearch Service domain
Lambda function	aes-siem-configure-aes	For configuring OpenSearch Service
Lambda function	aes-siem-index-metrics-exporter	For OpenSearch Service index metrics
Lambda function	aes-siem-BucketNotificationsHandler	For configuring invent notification for the S3 bucket that stores logs
Lambda function	aes-siem-add-pandas-layer	For adding aws_sdk_pandas as Lambda layer to es-loader
AWS Key Management Service (AWS KMS) KMS key & Alias	aes-siem-key	For encrypting logs
SSM Parameter Store	/siem/bucketpolicy/log/policy1-8	Temporarily used when keeping the s3 log Bucket Policy
Amazon SQS Queue	aes-siem-sqs-splitted-logs	A log is split into multiple parts if it has many lines to process. This is the queue to coordinate it
Amazon SQS Queue	aes-siem-dlq	A dead-letter queue used when loading logs into OpenSearch Service fails
CloudWatch alarms	aes-siem-TotalFreeStorageSpaceRemainsLowAlarm	Triggered when total free space for the OpenSearch Service cluster remains less than 200MB for 30 minutes
CloudWatch dashboards	SIEM	Dashboard of resource information used by SIEM on OpenSearch Service
EventBridge events	aes-siem-EventBridgeRuleStepFunctionsIoc	For executing aes-siem-ioc-state-machine regularly
EventBridge events	aes-siem-EventBridgeRuleLambdaGeoipDownloader	For executing aes-siem-geoip-downloader every 12 hours
EventBridge events	aes-siem-EventBridgeRuleLambdaMetricsExporter	For executing aes-siem-geoip-downloader every 1 hour
EventBridge events	aes-siem-EsLoaderStopperRule	For passing alarm events to es-loader-stopper
Amazon SNS Topic	aes-siem-alert	This is selected as the destination for alerting in OpenSearch Service
Amazon SNS Subscription	inputed email	This is the email address where alerts are sent

Cleanup

1. Navigate to the CloudFormation console and delete stack: aes-siem
2. Delete the following AWS resources manually:
   • OpenSearch Service domain: aes-siem
   • Amazon S3 bucket: aes-siem-[AWS_Account]-log
   • Amazon S3 bucket: aes-siem-[AWS_Account]-snapshot
   • Amazon S3 bucket: aes-siem-[AWS_Account]-geo
   • AWS KMS customer-managed key: aes-siem-key
     • Please delete this with care. After deleting this customer-managed key, you will no longer be able to read logs if they are encrypted using this key.
3. If you deployed SIEM on OpenSearch Service within an Amazon VPC, delete the following AWS resources as well:
   • Amazon VPC: aes-siem/VpcAesSiem (if you created a new VPC)
   • SecurityGroup: aes-siem-vpc-sg

Note If you want to redeploy SIEM on OpenSearch Service right after deleting it, you need to delete the key alias using the AWS CLI commands below. Otherwise, redeployment will fail as the KMS key alias still remains:

export AWS_DEFAULT_REGION=<AWS_REGION>
aws kms delete-alias --alias-name  "alias/aes-siem-key"

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.

This product uses GeoLite2 data created by MaxMind and licensed under CC BY-SA 4.0, available from https://www.maxmind.com.

This product uses Tor exit list created by The Tor Project, Inc and licensed under CC BY 3.0 US, available from https://www.torproject.org

siem-on-amazon-opensearch-service's Issues

Split SIEM/index python class and tune up querying geodb(es-loader)

es-loader

• indexとSIEMクラスの関数とコンパイル済み正規表現をutilsクラスを作って移動する
• 定数はindexの先頭に集約
• geodb関連のモジュールをgeodbクラスを作成して移動する
• pythonのメモリキャッシュを活用してgeodbのパフォーマンスを向上させる
• Kinesis Data Stream用のコードを削除
• dotを含んだkeyをネストされたdictを作成する関数と値を取得する関数の書き換え
• 日付処理でもメモリキャッシュをを有効化

enable S3 bucket key

https://docs.aws.amazon.com/AmazonS3/latest/user-guide/enable-bucket-key.html
https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-key.html

Change default Amazon ES's system index setting

Amazon ESのインデックスの初期値を変更する
.opendistro-ism-managed-index-history-*
.opendistro-alerting-alerts-*
初期値は5シャード、30日間保存、1レプリカ

CLB logs parsing fails

内容

CLB の一部ログで、正規表現のエラーが発生しているため、インデックスできないログが存在します。

エラー内容

	
[ERROR] Exception: Invalid regex pattern of clb in aws.ini or use.ini.
regex_pattern:
re.compile('(?P<timestamp>[^ ]+) (?P<elb>[^ ]+) (?P<client_ip>[0-9:\\.]+):(?P<client_port>[0-9]+) (-|(?P<backend_ip>[0-9:\\.]+):(?P<backend_port>[-0-9]+)) (?P<request_processing_time>[0-9\\.-]+) (?P<backend_proc)
rawdata:
XXXXXXXX rawdata (エラーの発生したログの例は下記を参照) XXXXXXXX
Traceback (most recent call last):
  File "/var/task/index.py", line 291, in lambda_handler
    for data in get_es_entry(logfile, logconfig, not_loading_list):
  File "/var/task/index.py", line 170, in get_es_entry
    logparser = siem.LogParser(
  File "/var/task/siem/__init__.py", line 641, in __init__
    self.__logdata_dict = self.logdata_to_dict()
  File "/var/task/siem/__init__.py", line 665, in logdata_to_dict
    raise Exception(

対象の CLB ログ（例、一部マスク済み）

e.g. 1

2020-11-16T14:01:48.546165Z xxxxxx-xxxxxxx-com xxx.yyy.zzz.xxx:41348 zzz.yy.xx.www:80 0.000023 0.003112 0.000025 400 400 0 2963 "SSTP_DUPLEX_POST https://aaa.bbb.ccc.ddd:443/sra_{XXXXXXX-YYYYY-ZZZZ-YYYY-XXXXXXXXX}/ HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2

e.g. 2

2020-11-17T02:45:38.950903Z xxxxxx-xxxxxxx-com xxx.yyy.zzz.xxx:60104 zzz.yy.xx.www:80 0.000028 0.271633 0.000022 200 200 0 304514 "GET https://api.xxxxxxxxx.com:443/some/path/search?range=all&limit=300&nextPageToken=xxxxxxxxxxxxxx00000000000TokenString00000000000xxxxxxxxxxxxxx HTTP/1.1" "" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2

• 使用バージョン：es-loader v2.1.0 beta
• 処理したログ：CLB ログ（aws.ini に変更は加えていないため、上記のバージョンの物を使用）
  • https://github.com/aws-samples/siem-on-amazon-elasticsearch/blob/7065aff7f1f833c043a4397cca99332e381bbda2/source/lambda/es_loader/aws.ini#L371

Release v2.1.0

Change custom resource directory of CFn template

バージョンアップしてもLambdaが保存されてるディレクトリが同じだとスタックの更新に失敗することがあるので、ディレクトリにバージョン番号を付与する

Unable to parse elb logs with space in http_query

Details

以下のような ELB ログを es-loader でパースしようとした際に regex pattern のエラーが発生します。

https 2021-01-28T04:46:15.195544Z app/fizz/elb xxx.xxx.xxx.xxx:34423 xxx.xxx.xxx.xxx:80 0.005 0.002 0.000 400 400 67 529 "GET https://xxx.xxx.xxx.xxx:443/auth/\x22 / HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:ap-northeast-1:{Accunt ID}:some/Strings "Root=1-IDstrings-someIDstring" "-" "arn:aws:acm:ap-northeast-1:{Accunt ID}:certificate/{ID Strings}" 0 2021-01-28T04:46:15.187000Z "forward" "-" "-" "xxx.xxx.xxx.xxx:80" "400" "Acceptable" "SpaceInUri"

classification_reason で SpaceInUri となっているように、以下の箇所でクエリ中にスペースが入っている点がエラーの原因になっているように思えます。
"GET https://xxx.xxx.xxx.xxx:443/auth/\x22 / HTTP/1.1"

Enviroments

• es-loader v2.1.1

load nano seconds to timestamp

標準のPythonではナノ秒の時間を処理できない。
アプリケーションではナノ秒のタイムスタンプを持つものがあり、@timestampとして扱う場合は読み取る必要がある。

実装方法
ナノ秒があったら、少数7桁目移行を切り捨てて、マイクロ秒に変換する。
user.ini に次の設定値を追加
timestamp_nano = True or False

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) の例外で停止

2つの場合で発生

• FireLensで取り込んだログに複数のファイルフォーマットが混入してる場合
• CloudWatch Logsで取り込んだログにascii以外の文字列を含んでいる場合

Unable to parse ALB and CLB logs containing offensive requests

内容

攻撃的なリクエストを含む ALB/CLB のログのエラーに失敗しています。
具体的には以下のようなログをパースする際に (?P<http_host>[^:]+):(?P<http_port>\d+) あたりが正常に認識できていないため、エラーが発生しています。

CLB のログ例

2020-12-23T00:00:00.000000Z clb {Attacker IP}:00000 {Backend IP}:10080 0.000041 0.010378 0.000037 400 400 0 2963 "GET https://[::ffff:a9fe:a9fe]:443/ HTTP/1.1" "Attacker UA" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2

ALB のログ例

https 2020-12-23T00:00:00.000000Z alb {Attacker IP}:00000 - -1 -1 -1 400 - 172 370 "GET http://[::ffff:a9fe:a9fe]:443/ HTTP/1.1" "Attcker UA" - - - "-" "-" "-" - 2020-12-22T23:59:59.000000Z "-" "-" "-" "-" "-" "-" "-"

Create AWS CloudFormation template for log sender side

create es-loader role for EC2

過去ログを取り込むためにEC2を使うが、そのためのロールを作成する

Watchdog and kill switch for es-loader

es-loaderの異常事態が発生したらSNS通知またはes-loaderを停止させる

通知と停止条件の例
・es-loaderの成功率が100%でない状態が続く
・同時起動数が設定したReserved concurrencyに張り付く
・Throttleの増える
・ESのクラスターの状態が赤になり、継続している
などなど

推奨のアラーム
https://docs.aws.amazon.com/ja_jp/elasticsearch-service/latest/developerguide/cloudwatch-alarms.html

Dashboard of AWS Security Hub

Enhance logging function of es-loader

Limit the maximum concurrency of es-loader to prevent unexpected scaling

目的

• es-loaderの設定やなんらかの不具合による異常なスケールを防止するため
• 受信側のAmaozn ESの過負荷を防止するため(429エラーの発生)

設定値は、CloudFormationまたはCDKのパラメーターとして設定し、es-loaderのLambdaのReserved concurrencyを設定する
ログ量が多い場合は手動でこの値を増やす

CDK failed with "Policy contains a statement with one or more invalid principals."

cdk.jsonにorganizationsで管理されていないアカウントIDだけを指定するとエラーとなる。

19:15:56 | UPDATE_FAILED        | AWS::KMS::Key                       | KmsAesSiemLog44B26597
Policy contains a statement with one or more invalid principals. (Service: Kms, Status Code: 400, Request ID: 4ae49104-e6ff-435c-ba6c-350e0aa0de35,
Extended Request ID: null)

Support AWS Config/AWS Config Rules

Import results of AWS Trusted Advisor

Support AWS Systems Manager Session Manger

Allow Athena to query logs in S3 bucket encrypted by CMK.

SIEMが作成するCMK(aes-siem-key)で暗合して保存したデータに対して、Athenaで検索ができない。キーポリシーで制限されていることが原因なので許可をするキーポリシーに変更する

fix node version as LTS for CDK

CDKが推奨しているnodeのバージョンは Active LTS なのでバージョンを固定する

Raised exception to no data object of s3 key

S3 keyが / で終わるファイルであったり、ConfigWritabilityCheckFile等の管理用と推測できるデータが0バイトのファイルに対して、例外を投げてDLQ行きになっている。取り込み対象にならないデータなので、warningだけにして例外処理ではなく処理をスキップさせる。

Import AWS support case

support CEF file format

AWSリソースのログとしてはCEFフォーマットはないが、Deep Security等で使用されているのでフレームワークとしてサポートする

support for timestamp_key including dot(.)

Detail

timestamp_key に . を含むフィールド名を指定してもエラーが発生し読み込みに失敗します。
timestamp フィールド指定は柔軟に実施できることを望みます。

[ERROR] KeyError: 'id.time'Traceback (most recent call last): File "/var/task/index.py", line 261, in lambda_handler for data in get_es_entry(logfile, logconfig, not_loading_list): File "/var/task/index.py", line 184, in get_es_entry logparser.add_basic_field() File "/var/task/siem/__init__.py", line 691, in add_basic_field self.__timestamp = self.get_timestamp() File "/var/task/siem/__init__.py", line 813, in get_timestamp timestr = self.__logdata_dict[timestamp_key].replace(

Enviroments

• es-loader v2.1.0
• AWS ES v7.9 (R20201117)

対応したいログのサンプル（マスク済み）

{"kind": "xxx", "id": {"time": "2021-01-04T08:08:43.053Z", "uniqueQualifier": "xxxx", "applicationName": "calendar", "customerId": "xxx"}, "etag": "xxxx, "actor": {"email": "[email protected]", "profileId": "xxxx"}, "ownerDomain": "example.com", "ipAddress": "1.1.1.1", "events": [{"type": "event_change", "name": "change_event_start_time", "parameters": [{"name": "event_id", "value": "xxx"}, {"name": "organizer_calendar_id", "value": "[email protected]"}, {"name": "calendar_id", "value": "[email protected]"}, {"name": "target_calendar_id", "value": "[email protected]"}, {"name": "event_title", "value": "xxx"}, {"name": "start_time", "intValue": "xxxx"}, {"name": "api_kind", "value": "web"}, {"name": "user_agent", "value": "Mozilla/5.0"}]}]}

Change lambda's default memory of es-loader

es-loaderのデフォルトのメモリーを増やす。
Lambdaの課金体系でが1msになったことにより、今までは100ms以内で完了させると不必要に料金が発生してしまったが、この変更により短時間で終わらせてもコストの最適化ができるようになったため

Functionality of filtering unnecessary logs

例)

• cloudtrail で分析に不要なAPIコール
• vpcflowlogsのheartbet通信

s3のファイルパスに含まれている場合は、s3_key_ignored に指定することで可能だが、この場合は抽出したフィールドに対してフィルタリングをする

NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation:

s3_keyにメタキャラクタが含まれていると、ファイルの取得でエラーになる。

[ERROR] NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
Traceback (most recent call last):
  File "/var/task/aws_lambda_powertools/metrics/metrics.py", line 144, in decorate
    response = lambda_handler(event, context)
  File "/var/task/aws_lambda_powertools/logging/logger.py", line 258, in decorate
    return lambda_handler(event, context)
  File "/var/task/index.py", line 232, in decorator
    return func(*args, **kwargs)
  File "/var/task/index.py", line 246, in lambda_handler
    logfile = extract_logfile_from_s3(record)
  File "/var/task/index.py", line 39, in extract_logfile_from_s3
    logfile = siem.LogS3(record, etl_config, s3_client)
  File "/var/task/siem/__init__.py", line 59, in __init__
    self.__rawdata = self.extract_rawdata_from_s3obj()
  File "/var/task/siem/__init__.py", line 82, in extract_rawdata_from_s3obj
    obj = self.s3_client.get_object(Bucket=self.s3bucket, Key=self.s3key)
  File "/var/runtime/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)

対策は s3_key を URL デコードをしてからオブジェクトのクエリーをする。
ただし、デコードはオブジェクトの取得時のみで、空白やその他の危険な文字列もあるため安全のためElasticsearchへの挿入時やその他の処理ではエンコードされた文字列を利用する

https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/notification-content-structure.html

s3 キーは、イベントに関与したバケットとオブジェクトに関する情報を提供します。オブジェクトのキー名の値は URL エンコードされます。たとえば、「red flower.jpg」は「red+flower.jpg」となります (Amazon S3 はコンテンツタイプとして「application/x-www-form-urlencoded」をレスポンスで返します)。

Implement QoS and shaping for unexpected volumes of logs

2時間に1回などの定期的なログの取り込みや突然の過剰なログが来ると、Lambdaは自動スケールするので問題はないが、その次のAmazon ESに過剰な負荷がかかる。
負荷を平準化するために、そういったログは後回しにたり流用を制限する機能を実装する

Support AWS Security Hub

Support AWS Network Firewall

Unable Connect do SQS if using a VPC

VPC 内で SQS の Private Link を作っても接続できない。
原因はboto3のバグ(boto3のissueの1900)で、sqs の client や resource を作成後に、queueのURLを現行形式のドメインで指定してSQSに接続を試みても旧形式のドメインに接続してしまいタイムアウトとなる。
対策は、boto3でclientやresourceの作成時にURLを指定する

参考
https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-sending-messages-from-vpc.html

Update document for v2.3.0

• 目次をつける
• 英語版を追加
• CWLとFirehoseについて補足する
• その他

VPC config validation raises KeyError when using VPC peering instead IGW

$ cdk -v deploy aes-siem --parameters AllowedSourceIpAddresses="10.0.0.0/8 192.168.0.1" --parameters GeoLite2LicenseKey=********

cdk.json validation for vpc configuration is starting...

vpc_type:                       import
checking vpc...
checking vpc id...:             vpc-123456789012xxxx
checking dns support...:        True
checking dns hostname...:       True
checking vpc is...              [PASS]

Traceback (most recent call last):
  File "app.py", line 10, in <module>
    MyAesSiemStack(app, "aes-siem",
  File "/home/ec2-user/siem-on-amazon-elasticsearch/source/cdk/.env/lib64/python3.8/site-packages/jsii/runtime.py", line 69, in call
    inst = super().call(args, *kwargs)
  File "/home/ec2-user/siem-on-amazon-elasticsearch/source/cdk/mysiem/aes_siem_stack.py", line 138, in _init
    validate_cdk_json(self)
  File "/home/ec2-user/siem-on-amazon-elasticsearch/source/cdk/mysiem/aes_siem_stack.py", line 78, in validate_cdk_json
    subnet_type = get_pub_or_priv_subnet(rt_client.routes_attribute)
  File "/home/ec2-user/siem-on-amazon-elasticsearch/source/cdk/mysiem/aes_siem_stack.py", line 69, in get_pub_or_priv_subnet
    if route['GatewayId'].startswith('igw-'):
KeyError: 'GatewayId'
Subprocess exited with error 1

Error: Subprocess exited with error 1
    at ChildProcess.<anonymous> (/home/ec2-user/.nvm/versions/node/v14.15.1/lib/node_modules/aws-cdk/lib/api/cxapp/exec.ts:122:23)
    at ChildProcess.emit (events.js:315:20)
    at ChildProcess.EventEmitter.emit (domain.js:486:12)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)

Suport AWS Tags

KibanaからAWS Tagでフィルターをできるようにする

Elastic common schema の labels にマッピングする
https://www.elastic.co/guide/en/ecs/current/ecs-base.html#field-labels

update Amazon ES version to 7.9.x

Support RDS/Aurora/MySQL

Support ECS container log via Firelens

• https://aws.amazon.com/jp/blogs/opensource/centralized-container-logging-fluent-bit/

Import AWS Billing and Cost Management

Import Cost & Usage Reports

Support RDS/Aurora/PostgreSQL

• https://aws.amazon.com/jp/blogs/database/analyze-postgresql-logs-with-amazon-elasticsearch-service/
• https://aws.amazon.com/jp/blogs/database/configuring-and-authoring-kibana-dashboards/

es-loader can't parse responseElements.credentials in CloudTrail Logs

Detail

"responseElements.credentials": "arn:aws:iam::000000000000:role/foo-bar" というような形式のログをインデックス使用とする際に以下のようなエラーが発生します。
既存のインデックス済みのログにより、responseElements.credentials が objects として認識されているため型の不一致が発生しています。

ERROR[8]: {'_index': 'log-aws-cloudtrail-2020-12-25', '_type': '_doc', '_id': 'xxxxx-id-xxxxxxxx', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': 'object mapping for [responseElements.credentials] tried to parse field [credentials] as object, but found a concrete value'}}

Enviroments

• es-loader v2.1.0
• AWS ES v7.9 (R20201117)

Example log

どういったタイミングで下記のログが吐き出されるは不明

エラーが発生したログのサンプル（マスク済み）

{
  "eventVersion": "1.08",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "XXXXXXXXXXXXXX:oooooooooo",
    "arn": "arn:aws:sts::000000000000:assumed-role/foo-bar/oooooooooo",
    "accountId": "000000000000",
    "accessKeyId": "XXXXXXXXXXXXXX",
    "sessionContext": {
      "sessionIssuer": {
        "type": "Role",
        "principalId": "XXXXXXXXXXXXXX",
        "arn": "arn:aws:iam::000000000000:role/foo-bar",
        "accountId": "000000000000",
        "userName": "foo-bar"
      },
      "webIdFederationData": {},
      "attributes": {
        "mfaAuthenticated": "false",
        "creationDate": "2020-12-21T02:30:28Z"
      }
    }
  },
  "eventTime": "2020-12-21T02:33:02Z",
  "eventSource": "apigateway.amazonaws.com",
  "eventName": "PutIntegration",
  "awsRegion": "ap-northeast-1",
  "sourceIPAddress": "xxx.xxx.xxx.xxx",
  "userAgent": "aws-sdk-go/1.32.12 (go1.13.7; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.28 (+https://www.terraform.io)",
  "requestParameters": {
    "resourceId": "aaaaaa",
    "httpMethod": "GET",
    "putIntegrationInput": {
      "passthroughBehavior": "WHEN_NO_TEMPLATES",
      "httpMethod": "GET",
      "connectionType": "INTERNET",
      "type": "AWS",
      "requestParameters": {
        "integration.request.path.object-key": "method.request.path.object-key"
      },
      "timeoutInMillis": 29000,
      "uri": "arn:aws:apigateway:ap-northeast-1:s3:path/xxxxxxxxxxxxxxxxxxx/{object-key}",
      "requestTemplates": {},
      "credentials": "arn:aws:iam::000000000000:role/foo-bar"
    },
    "restApiId": "aaaaaaaaaaaaaaaa",
    "template": false
  },
  "responseElements": {
    "type": "AWS",
    "timeoutInMillis": 29000,
    "httpMethod": "GET",
    "integrationDelete": {
      "restApiId": "aaaaaaaaaaaaaaaa",
      "httpMethod": "GET",
      "resourceId": "aaaaaa",
      "template": false
    },
    "requestParameters": {
      "integration.request.path.object-key": "method.request.path.object-key"
    },
    "cacheKeyParameters": [],
    "integrationresponsePut": {
      "httpMethod": "GET",
      "resourceId": "aaaaaa",
      "restApiId": "aaaaaaaaaaaaaaaa",
      "template": true
    },
    "requestTemplates": {},
    "passthroughBehavior": "WHEN_NO_TEMPLATES",
    "credentials": "arn:aws:iam::000000000000:role/foo-bar",
    "cacheNamespace": "aaaaaa",
    "integrationUpdate": {
      "restApiId": "aaaaaaaaaaaaaaaa",
      "resourceId": "aaaaaa",
      "httpMethod": "GET",
      "template": false
    },
    "uri": "arn:aws:apigateway:ap-northeast-1:s3:path/xxxxxxxxxxxxxxxxxxx/{object-key}",
    "self": {
      "restApiId": "aaaaaaaaaaaaaaaa",
      "httpMethod": "GET",
      "resourceId": "aaaaaa",
      "template": false
    }
  },
  "requestID": "request-id-strings-xxxxxxxxx",
  "eventID": "event-id-strings-xxxxxxxxx",
  "readOnly": false,
  "eventType": "AwsApiCall",
  "managementEvent": true,
  "eventCategory": "Management",
  "recipientAccountId": "000000000000"
}

To be

• responseElements.credentials が Strings の場合、"responseElements.credentials.iam": "value" という形に変換してインデックスする

Support multi-line text file format

ELB Dashboard

Change log count logic of GuardDuty Dashboard

GuardDutyのダッシュボードのカウント方法を行数ではなく、findings数に変更
GuardDutyが一連のfindingsと判断したものは継続してても1つとカウントする

Unable to parse CLB logs with space after http_version

概要

CLB のログで以下のように http_method や http_protocl などを含まないログを es-loader で処理しようとしたところ、パースエラーが発生しました。

2021-01-18T05:05:37.412050Z fizz-buzz-proxy xx.xx.xx.xx:oooo cc.cc.cc.ccc:vvvv 0.00042 0.000009 0.000014 - - 524 1199 "- - - " "-" - -

手元で確認した限りだと、エラーとなるログでは http_version の後ろに半角スペースが入っているため、以下の箇所で regex のエラーが発生している物と思われます。

(-|\w+/(?P<http_version>[0-9\.]*)))\"

エラーのログ

[ERROR] Exception: Invalid regex pattern of clb in aws.ini or use.ini.regex_pattern:re.compile('(?P<timestamp>[^ ]+) (?P<elb>[^ ]+) (?P<client_ip>[0-9:\\.]+):(?P<client_port>[0-9]+) (-|(?P<backend_ip>[0-9:\\.]+):(?P<backend_port>[-0-9]+)) (?P<request_processing_time>[0-9\\.-]+) (?P<backend_proc)rawdata: ----↑ のログ----- Traceback (most recent call last):  File "/var/task/index.py", line 322, in lambda_handler    for data in get_es_entry(logfile, logconfig, exclude_log_patterns):  File "/var/task/index.py", line 227, in get_es_entry    logparser = siem.LogParser(  File "/var/task/siem/__init__.py", line 718, in __init__    self.__logdata_dict = self.logdata_to_dict()  File "/var/task/siem/__init__.py", line 763, in logdata_to_dict    raise Exception(

環境

es-loader v2.1.1

Dashboard of AWS Config/AWS Config Rules

An error occurred when processing the file of size 0 byte with es-loader.

内容

es-loader でファイルサイズが 0 バイトのログを処理しようとすると以下のようなエラーが発生します。

[ERROR] UnboundLocalError: local variable 'log_count' referenced before assignment
Traceback (most recent call last):
  File "/var/task/index.py", line 291, in lambda_handler
    for data in get_es_entry(logfile, logconfig, not_loading_list):
  File "/var/task/index.py", line 168, in get_es_entry
    for logdata in logfile.logdata_list:
  File "/var/task/siem/__init__.py", line 498, in logdata_list
    self.split_logs_to_sqs(log_count, max_log_count)

こちらで使用する際に少々編集し行数変わっているため、行番号ずれていますがエラーが発生している箇所としては下記の部分の処理です。(logdata_list 関数内で JSON のログを処理する部分)
https://github.com/aws-samples/siem-on-amazon-elasticsearch/blob/7065aff7f1f833c043a4397cca99332e381bbda2/source/lambda/es_loader/siem/__init__.py#L514

• 使用バージョン：es-loader v2.1.0 beta
• 処理したログ：Slack の Audit ログ（user.ini で定義したログで jsonl 形式）

状況

1. S3 → Lambda のトリガーを設定し PUT で反応するようにする
2. Slack のログを定期的に S3 へ PUT する
3. PUT により es-loader が Invoke されるが、処理対象のログが 0バイトの場合、上述したエラーが発生

想定

• 対象のログファイルのサイズが 0 バイトや中身がない場合は処理をスキップする。

Refactor and tune up ETL process of es-loader

Unable to parse CloudFront Standard access log requested from IPv6 address

内容

es_loader において CloudFront の Standard access log のパースに失敗します。
IPv6 アドレスからのリクエストである場合、正規表現のパターンにマッチせずに失敗しているようです。

https://github.com/aws-samples/siem-on-amazon-elasticsearch/blob/main/source/lambda/es_loader/aws.ini#L515
こちらにおける c_ip のパターンを以下の様に修正したところ手元の環境ではエラーが解消しました。

- (?P<c_ip>[0-9.:]+)
+ (?P<c_ip>[0-9a-f.:]+) 

IPv6 アドレスからのアクセスログの例

2021-01-20	07:06:13	KIX56-C2	233918	2001:db8:85a3:8d3:1319:8a2e:370:7348	GET	xxxxxxxx.cloudfront.net	/test	200	https://example.com/	Mozilla/5.0%20(Macintosh;%20Intel%20Mac%20OS%20X%2010_15_6)%20AppleWebKit/605.1.15%20(KHTML,%20like%20Gecko)%20Version/14.0.2%20Safari/605.1.15	-	-	Hit	aYaCQgZzC1uaCG__Jc9a0FM0qGa2T96J2TKXPGP3NiOn48oODhcdAQ==	example.com	https	40	0.004	-	TLSv1.3	TLS_AES_128_GCM_SHA256	Hit	HTTP/2.0	-	-	56258	0.001	Hit	application/javascript	-	-	-