pavan3401/alb-logs-to-elasticsearch

Sending ALB Logs from S3 to Elasticsearch

ALB-Logs-to-Elasticsearch

#Send ALB logs from S3 bucket to ElasticSearch using AWS Lambda. This project based on awslabs/amazon-elasticsearch-lambda-samples,blmr/aws-elb-logs-to-elasticsearch Sample code for AWS Lambda to get AWS ELB log files from S3, parse and add them to an Amazon Elasticsearch Service domain.

Deployment Package Creation

1. On your development machine, download and install Node.js.

2. Go to root folder of the repository and install node dependencies by running:

   npm install
   

   Verify that these are installed within the node_modules subdirectory.

3. Create a zip file to package the index.js and the node_modules directory

The zip file thus created is the Lambda Deployment Package.

AWS Configuration

Set up the Lambda function and the S3 bucket. You can reffer to for more details > Lambda-S3 Walkthrough.

Please keep in mind the following notes and configuration overrides:

• The S3 bucket must be created in the same region as Lambda is, so that it can push events to Lambda.

• When registering the S3 bucket as the data-source in Lambda, add a filter for files having .log.gz suffix, so that Lambda picks up only apache log files.

• The following authorizations are required:

  1. Lambda permits S3 to push event notification to it
  2. S3 permits Lambda to fetch the created objects from a given bucket
  3. ES permits Lambda to add documents to the given domain
  4. Lambda handler is set to index.handler
  5. Don't forget the ES domain parameters in index.js

  The Lambda console provides a simple way to create an IAM role with policies for (1).
  For (2), when creating the IAM role, choose the "S3 execution role" option; this will load the role with permissions to read from the S3 bucket.
  For (3), add the following access policy to permit ES operations to the role.

{
      "Sid": "AllowLambdaAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/lambda_s3_exec_role"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-west-1:123456789012:domain/elastic-search-domain/*"
}

For (5)

var esDomain = {
    endpoint: 'elastic-search-domain-123456.us-east-1.es.amazonaws.com',
    region: 'us-east-1',
    index: 'alb-access-logs-' + indexTimestamp,
    doctype: 'alb-access-logs'
};

Event source Add Event source for your lambda function

Event source type: S3

Bucket: s3-elb-access-logs

Event type: Object Created (All)

Suffix: .log.gz

#License ASL

https://aws.amazon.com/asl/

alb-logs-to-elasticsearch's Issues

S3 bucket delay

Have you noticed parsing partial files? We have a similar script that we tried to use with Lambda, but was seeing partial log files. Meaning 1 or 2 records would be initially written and then a few more lines a bit later. Our guess was that AWS was continue to write to the file after it was initially loaded into S3. The docs say they upload file every 5 minutes, but maybe it's more of stream for 5 minutes per file.

Our solution was to use SQS to trigger the initial S3 file PUT event and then wait 5-6 minutes to parse the file. Then use a Node worker to parse the files into Elasticsearch

I was just wondering if you noticed anything similar?

Request Header Content-Type application/json is missing

In index.js Content-Type application/json is missing in index.js in request part

req.headers['Content-Type'] = 'application/json';

Now ES is require Content-Type in header of request.
Thank you.

Failed to parse json

I cloned this repo on my server.
After running "npm install" i got this error

npm WARN Failed to parse json
npm WARN Trailing comma in object at 8:5
npm WARN     }
npm WARN     ^
npm WARN File: /root/ALB-Logs-to-Elasticsearch/package.json
npm WARN ALB-Logs-to-Elasticsearch No description
npm WARN ALB-Logs-to-Elasticsearch No repository field.
npm WARN ALB-Logs-to-Elasticsearch No README data
npm WARN ALB-Logs-to-Elasticsearch No license field.

Please tell me how to fix this.