theburningmonk/lambda-logging-kinesis-demo

Demo of how to ship logs from CloudWatch Logs to a Kinesis stream, and then ship them to Logz.io

lambda-logging-kinesis-demo

Demo of how to ship logs from CloudWatch Logs to a Kinesis stream, and then ship them to Logz.io

A group of Lambda functions for:

• shipping logs from Kinesis stream to Logz.io (hosted ELK stack)
• auto-subscribe new log groups to the configured Kinesis stream so you don't have to subscribe them manually
• auto-updates the retention policy of new log groups to 7 days (configurable)

Deployment

1. insert the logstash_host, logstash_port and token in the serverless.yml file (under the ship-logs-to-logzio function's environment variables).

token: your Logz.io account token. Can be retrieved on the Settings page in the Logz.io UI. logstash_host: if you are in the EU region insert listener-eu.logz.io, otherwise, use listener.logz.io. You can tell which region you are in by checking your login URL - app.logz.io means you are in the US. app-eu.logz.io means you are in the EU. logstash_port: this should be 5050, but is subject to change. See this page for details.

for example:

ship-logs-to-logzio:
  handler: functions/ship-logs/handler.handler
  description: Sends CloudWatch logs from Kinesis to Logz.io
  environment:
    logstash_host: listener.logz.io
    logstash_port: 5050
    token: CduNgGwuFFeUVzbXvqVDXoGkjxEdKzc9

2. run ./build.sh deploy dev to deploy to a stage called "dev"

Updating existing log groups

1. open the process_all.js script

2. fill in the missing configuration values

3. run node process_all.js

lambda-logging-kinesis-demo's Issues

Missing build.sh

Hello

Your installation instructions mention running "build.sh". Should this be present in the repo?

DestinationArn for vendor kinesis cannot be used without roleArn

Hi @theburningmonk.

I am getting the following error in the subscribe function when I create a new log group.

{
    "errorMessage": "destinationArn for vendor kinesis cannot be used without roleArn",
    "errorType": "InvalidParameterException",
    "stackTrace": [
        "Request.extractError (/var/task/node_modules/aws-sdk/lib/protocol/json.js:48:27)",
        "Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:105:20)",
        "Request.emit (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:77:10)",
        "Request.emit (/var/task/node_modules/aws-sdk/lib/request.js:683:14)",
        "Request.transition (/var/task/node_modules/aws-sdk/lib/request.js:22:10)",
        "AcceptorStateMachine.runTo (/var/task/node_modules/aws-sdk/lib/state_machine.js:14:12)",
        "/var/task/node_modules/aws-sdk/lib/state_machine.js:26:10",
        "Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:38:9)",
        "Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:685:12)",
        "Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:115:18)"
    ]
}

If I add the roleArn property as:

let options = {
    destinationArn: arn,
    logGroupName: logGroupName,
    filterName: 'ship-logs',
    filterPattern: '[timestamp=*Z, request_id="*-*", event]',
    roleArn: role_arn,
  }

and

      role_arn:
        Fn::GetAtt:
          - CloudWatchLogsRole
          - Arn

Then I get the following error.

{
    "errorMessage": "User: arn:aws:sts::275307759108:assumed-role/cloudwatch-logs-to-logzio-dev-subscribe-eu-west-1-lambdaRole/cloudwatch-logs-to-logzio-dev-subscribe is not authorized to perform: iam:PassRole on resource: arn:aws:iam::275307759108:role/cloudwatch-logs-to-logzio-dev-cloudwatchrole",
    "errorType": "AccessDeniedException",
    "stackTrace": [
        "Request.extractError (/var/task/node_modules/aws-sdk/lib/protocol/json.js:48:27)",
        "Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:105:20)",
        "Request.emit (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:77:10)",
        "Request.emit (/var/task/node_modules/aws-sdk/lib/request.js:683:14)",
        "Request.transition (/var/task/node_modules/aws-sdk/lib/request.js:22:10)",
        "AcceptorStateMachine.runTo (/var/task/node_modules/aws-sdk/lib/state_machine.js:14:12)",
        "/var/task/node_modules/aws-sdk/lib/state_machine.js:26:10",
        "Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:38:9)",
        "Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:685:12)",
        "Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:115:18)"
    ]
}

Any thought what can it be? I haven't modify the code from the repo.

Thanks,
Pedro