lukewaite/logstash-input-cloudwatch-logs

Input plugin for Logstash to stream events from CloudWatch Logs

Logstash Input for CloudWatch Logs

Stream events from CloudWatch Logs.

Update July 2023

Hello, this project has not received any maintenance since May of 2018. It is no longer a path I am using professionally, and I can not devote the time to maintaining it.

Purpose

Specify an individual log group or array of groups, and this plugin will scan all log streams in that group, and pull in any new log events.

Optionally, you may set the log_group_prefix parameter to true which will scan for all log groups matching the specified prefix(s) and ingest all logs available in all of the matching groups.

Usage

Parameters

Parameter	Input Type	Required	Default
log_group	string or Array of strings	Yes
log_group_prefix	boolean	No	false
start_position	beginning, end, or an Integer	No	beginning
sincedb_path	string	No	$HOME/.sincedb*
interval	number	No	60
aws_credentials_file	string	No
access_key_id	string	No
secret_access_key	string	No
session_token	string	No
region	string	No	us-east-1
codec	string	No	plain

start_position

The start_position setting allows you to specify where to begin processing a newly encountered log group on plugin boot. Whether the group is 'new' is determined by whether or not the log group has a previously existing entry in the sincedb file.

Valid options for start_position are:

• beginning - Reads from the beginning of the group (default)
• end - Sets the sincedb to now, and reads any new messages going forward
• Integer - Number of seconds in the past to begin reading at

Logstash Default config params

Other standard logstash parameters are available such as:

• add_field
• type
• tags

Example

input {
    cloudwatch_logs {
        log_group => [ "/aws/lambda/my-lambda" ]
        access_key_id => "AKIAXXXXXX" 
        secret_access_key => "SECRET"
    }
}

Development

The default logstash README which contains development directions and other information has been moved to DEVELOPER.md.

Contributing

All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.

Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.

It is more important to the community that you are able to contribute.

For more information about contributing, see the CONTRIBUTING file.

logstash-input-cloudwatch-logs's Issues

Submit to Logstash-Plugins

https://www.elastic.co/guide/en/logstash/current/submitting-plugin.html

Sincedb file won't be updated until getting all the logs

Hi, Yesterday I restarted the logstash for some reasons, and after that it couldn't update the sincedb files anymore, because the log groups contains a huge amount of logs and they're growing every minute.

It seems the sincedb file won't be updated until getting all the logs in the specified Log Groups. So, when you want to read a large amount of logs with this plugin, if the logstash/plugin stops or restarts before getting all the logs, it will start getting the logs from the beginning (from the date/time that's written in sincedb file). And it takes a long time to reach the end or maybe it never reaches the end because of the growing number of logs, regular restarts and errors like this (execution expired):

{:timestamp=>"2016-01-19T02:03:30.537000+0000", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n  Plugin: <LogStash::Inputs::CloudWatch_Logs log_group=>\"sampleLogGroup\", region=>\"us-east-1\", sincedb_path=>\"/tmp/sincedb_something\", interval=>55, codec=><LogStash::Codecs::JSON charset=>\"UTF-8\">, debug=>false>\n  Error: execution expired", :level=>:error}

~almost similar issue in logstash-input-file plugin:
https://discuss.elastic.co/t/sincedb-doesnt-updated-until-tail-reaches-the-end-of-the-file/36116

start_postion is not working

https://github.com/lukewaite/logstash-input-cloudwatch-logs/blob/master/lib/logstash/inputs/cloudwatch_logs.rb#L173

this line of code failed to convert DateTime.now.strftime('%Q') to integer, which is causing problem "-" not defined. I have a fix locally
timeNow = DateTime.now.strftime('%Q') sincedb[group] = timeNow.to_i - (@start_position * 1000)
could you please take a look and fix it?

How to handle if logstash restart/reinstall or i do have multi logstash ?

What i would like to know if i reinstall logstash / i need upgrade logstash verison
how it will know which log i already send to ELK

or
if i have more then one logstash on server (such as if i have ELB and two logstash behind it for HA feature)

Error && "Initialize: name or service not known"

First, thanks for the plugin. Second, I just tried it with logstash 5.4.2 running in a container (the official one from elastic.co, with the plugin installed as described in the blog post), and ran into the following issue:

Jul 07 15:19:32 ip-10-10-15-13 docker[27906]: 13:19:32.212 [[main]<cloudwatch_logs] ERROR logstash.pipeline - A plugin had an unrecoverable error. Will restart this plugin.
Jul 07 15:19:32 ip-10-10-15-13 docker[27906]:   Plugin: <LogStash::Inputs::CloudWatch_Logs type=>"elasticbeanstalk", region=>"eu-central-1", log_group=>["/aws/elasticbeanstalk/"], log_group_prefix=>true, interval=>30, id=>"c667b4b34c50753165191cd8ff63292b06b5a5c7-2", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_2acad102-83e1-4aa2-accf-6ae5ffad62f9", enable_metric=>true, charset=>"UTF-8">, start_position=>"beginning">
Jul 07 15:19:32 ip-10-10-15-13 docker[27906]:   Error: initialize: name or service not known

I have no clue what this is, but I guess it's not working. :)

Plugin crashed when reading cw logs streams

Hi
I am facing an issue with plugin crashing when reading cw logs
See attached error
this is happeing periodically with different cw logs - instances are using iam roles and permissions are ok

Make repo and plugin name match?

A request - given that logstash-plugin doesn't have a search function, it took me longer than it should have to realise that the git repo name (-cloudwatch-logs) and the plugin name (-cloudwatch_logs) don't match. I use a year-old version of ELK (abomination, I know!) so I don't know if 5.x is different to the current release plugin name in 6.x.

Perhaps change the repo name to match the plugin name? Or at the very least, document the plugin name difference in the readme :)

Execute in parallel 10 requests per seconds

Hello,

According to the official AWS API documentation, calls are limited to 1 Mo of logs per requests and 10 requests per seconds (Source).

Is this plugin (can / will) implement multi "threads" in order to execute 10 requests per seconds and collect previous logs more quickly?

Thanks.

Plugin doesn't support role assumption

Any chance we can add role assumption into this plugin?

Multiple entries for for log_group

Hi,

We have the following log_group patterns _/aws/lambda/app-name
Can we have input
log_group => ["logname1", "logname2"]
does not work

Logstash version: 5.1

Entering child directories in log groups

Sorry for posting my question here, but was hoping I could get some information about how to configure the plugin to enter any child directories. Right now, the plugin pulls only the first - highest directory under my group.

Rate exceeded error

Hi,

This plugin is very helpful thanks. Can you help me with the errors I'm seeing please? I got the following errors on the logs:

{:timestamp=>"2016-08-09T02:59:05.946000+0000", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::CloudWatch_Logs log_group=>"XXX-XXX_server_log", sincedb_path=>"/opt/logstash/sincedb/.sincedb_", region=>"us-east-1", codec=><LogStash::Codecs::Plain charset=>"UTF-8">, interval=>60>\n Error: Rate exceeded", :level=>:error}
{:timestamp=>"2016-08-09T02:59:13.045000+0000", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::CloudWatch_Logs log_group=>"XXX-XXX_server_log", sincedb_path=>"/opt/logstash/sincedb/.sincedb_", region=>"us-east-1", codec=><LogStash::Codecs::Plain charset=>"UTF-8">, interval=>60>\n Error: Rate exceeded", :level=>:error}

Thanks!

Not processing latest logs for some log groups

Hello,

We're using this plugin to consume logs from Cloudwatch (a mixture of ECS logs and logs that are streamed from standalone node apps). It works well for the most part, but it seems that there are a handful of log groups for which the latest streams are not consumed. I restarted the container a few times, then left it running over a 24 hour period and those particular groups are still behind:

$ grep -rho "149.*" sincedb/
1490323402065
1497849471489
1494266051700
1493998038662
1495695009294
1497888135459
1490267320025
1496250858292
1493943685098

Not sure if this is related to #24 - what are some steps we could take to debug? I'm not seeing the plugin fail (at least nothing helpful is output in the logs) and heartbeats (I have them at a 5 minute interval) have been logged throughout the period where the streams from these groups are not consumed.

Running on docker logstash:5.3.0 with memory usage consistently around 1.3 GB.

Install breaks on upgrade

First, I want to say thank you for writing an awesome plugin for logstash. The only issue I have is that after an upgrade I can't seem to get the plugin to reinstall. Using either the install method from the gem file or using logstash-plugin install logstash-input-cloudwatch_logs command. I have tried wiping all files on the system that reference cloudwatch_logs but that didn't help either.

I just get the following error when I try to restart logstash of a missing codec.

[2018-11-28T07:28:52,695][ERROR][logstash.plugins.registry] Tried to load a plugin's code, but failed. {:exception=>#<LoadError: no such file to load -- logstash/codecs/cloudtrail>, :path=>"logstash/codecs/cloudtrail", :type=>"codec", :name=>"cloudtrail"}
[2018-11-28T07:28:52,738][FATAL][logstash.runner ] The given configuration is invalid. Reason: Couldn't find any codec plugin named 'cloudtrail'. Are you sure this is correct? Trying to load the cloudtrail codec plugin resulted in this error: no such file to load -- logstash/codecs/cloudtrail

Using version 6.4.1 of logstash.

Error on start: uninitialized constant Aws::CloudWatchLogs

Hi!
I get the following error when trying to start the plugin with logstash-5.2.0:

[2017-04-12T19:39:30,513][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"input", :name=>"cloudwatch_logs", :path=>"logstash/inputs/cloudwatch_logs", :error_message=>"uninitialized constant Aws::CloudWatchLogs", :error_class=>NameError, :error_backtrace=>["org/jruby/RubyModule.java:2719:in const_missing'", "/home/ec2-user/logstash-5.2.0/vendor/bundle/jruby/1.9/gems/logstash-input-cloudwatch_logs-0.10.0/lib/logstash/inputs/cloudwatch/patch.rb:16:in Aws'", "/home/ec2-user/logstash-5.2.0/vendor/bundle/jruby/1.9/gems/logstash-input-cloudwatch_logs-0.10.0/lib/logstash/inputs/cloudwatch/patch.rb:15:in (root)'", "org/jruby/RubyKernel.java:1040:in require'", "/home/ec2-user/logstash-5.2.0/vendor/bundle/jruby/1.9/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require'", "/home/ec2-user/logstash-5.2.0/vendor/bundle/jruby/1.9/gems/logstash-input-cloudwatch_logs-0.10.0/lib/logstash/inputs/cloudwatch_logs.rb:1:in (root)'", "org/jruby/RubyKernel.java:1040:in require'", "/home/ec2-user/logstash-5.2.0/vendor/bundle/jruby/1.9/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require'", "/home/ec2-user/logstash-5.2.0/vendor/bundle/jruby/1.9/gems/logstash-input-cloudwatch_logs-0.10.0/lib/logstash/inputs/cloudwatch_logs.rb:10:in (root)'", "/home/ec2-user/logstash-5.2.0/logstash-core/lib/logstash/plugins/registry.rb:1:in (root)'", "/home/ec2-user/logstash-5.2.0/logstash-core/lib/logstash/plugins/registry.rb:151:in legacy_lookup'", "/home/ec2-user/logstash-5.2.0/logstash-core/lib/logstash/plugins/registry.rb:133:in lookup'", "/home/ec2-user/logstash-5.2.0/logstash-core/lib/logstash/plugins/registry.rb:175:in lookup_pipeline_plugin'", "/home/ec2-user/logstash-5.2.0/logstash-core/lib/logstash/plugin.rb:127:in lookup'", "org/jruby/RubyKernel.java:1079:in eval'", "/home/ec2-user/logstash-5.2.0/logstash-core/lib/logstash/pipeline.rb:452:in plugin'", "(eval):8:in initialize'", "/home/ec2-user/logstash-5.2.0/logstash-core/lib/logstash/pipeline.rb:98:in initialize'", "/home/ec2-user/logstash-5.2.0/logstash-core/lib/logstash/agent.rb:246:in create_pipeline'", "/home/ec2-user/logstash-5.2.0/logstash-core/lib/logstash/agent.rb:95:in register_pipeline'", "/home/ec2-user/logstash-5.2.0/logstash-core/lib/logstash/runner.rb:264:in execute'", "/home/ec2-user/logstash-5.2.0/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/home/ec2-user/logstash-5.2.0/lib/bootstrap/environment.rb:71:in `(root)'"]}

Allow providing aws profile name instead of aws key and secret key

Currently you need to provide access key and secret key to the account where cloudwatch is. Can we add an option to provide just the aws profile name of the profile setup on the logstash server and the plugin would be getting access_key and secret_key from that profile? Same way as terraform does it.
We store logstash.config in code repository and for obvious security reasons we don't want to store secret keys there.

sincedb_path does not seem to be effective

I have the following configuration:

cloudwatch_logs {
  type             => "elasticbeanstalk"
  region           => "eu-central-1"
  log_group        => "/aws/elasticbeanstalk/"
  log_group_prefix => true
  interval         => 30
  sincedb_path     => "/state/sincedb"
}

The sincedb_path is used. I know that because I did create it as a directory (I thought the files would be placed inside the path), and it complained and crashed. I removed the directory, I thought I would see a file appear now. It does not. Is that normal?

How to install this plugin in windows

Hi Lukewaite/Team,

Please let me know the installation steps for windows

Can we use wildcards to for multiple entries.

Mr. Luke,
Can we use wildcards to take multiple entries, which follows same pattern(at least some part), as input?

Not getting any CloudWatch logs

Hi, I'm unable to actually stream any logs. I'm sure my IAM permissions etc are set (I was seeing unauthorized errors before updating them)

Below is my Logstash configuration file

input {
  cloudwatch_logs {
    log_group => [ "/aws/prefix" ]
    log_group_prefix => true
  }
}

output {
  elasticsearch {
    hosts => "http://host:9200"
    index => "cloudwatch-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug}
}

This is output from my logstash logs

[2017-07-18T17:55:34,221][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://host:9200/]}}
[2017-07-18T17:55:34,224][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://host:9200/, :path=>"/"}
[2017-07-18T17:55:34,300][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<Java::JavaNet::URI:0x6c7da35a>}
[2017-07-18T17:55:34,302][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<Java::JavaNet::URI:0x490865f>]}
[2017-07-18T17:55:34,304][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-07-18T17:55:34,332][INFO ][logstash.pipeline        ] Pipeline main started
[2017-07-18T17:55:34,393][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

I've tried running Logstash with the --verbose flag but I'm not seeing any issues or errors. On the AWS side, I can confirm that my credentials were used to access logs. I've tried explicitly adding the key_id and secret_key in the cloudwatch_logs blob to no avail (currently using an IAM role associated with the instance).

Any ideas on how to debug would be appreciated. I tried adding the output to stdout to see if there was an issue with Elasticsearch receiving the logs, but the issue seems to be with Logstash receiving the logs.
FWIW, I was using Filebeat previously and had no issues receiving logs, creating the index, and adding the log data to the ES index.

Thanks!

logstash-mixin-aws to 4.3.0

Hi,

Looks like there's a higher version of logstash-mixin-aws (4.3.0).
Can you please bump the version so we can use the latest features (assume role).

Alternatively, can you please list the steps involved in someone like me changing the gemspec and perhaps creating a new gemfile and installing the plugin. I don't have a clue how these are done.

Thanks,
Arun

How to install?

Hey guys!

Sorry for ignorance, but how do I install the plugin?

not compatible with 5.x.x

With a very minimal config and the official Logstash docker containers, this plugin isn't compatible with logstash 5.x. Here's the 5.0.0 error:

...
22:31:58.185 [[main]<cloudwatch_logs] ERROR logstash.pipeline - A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::CloudWatch_Logs log_group=>"staging5-docker", type=>"docker_cloudwatchlogs", id=>"96031aebedd7bbedf8fe2af725162a5ebb66f694-1", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_bf72ffa9-05f4-47d5-8412-3e5af3e9f35e", enable_metric=>true, charset=>"UTF-8">, region=>"us-east-1", interval=>60, log_group_prefix=>false>
  Error: Direct event field references (i.e. event['field'] = 'value') have been disabled in favor of using event get and set methods (e.g. event.set('field', 'value')). Please consult the Logstash 5.0 breaking changes documentation for more details.
  Exception: NoMethodError
  Stack: /usr/share/logstash/logstash-core-event-java/lib/logstash/event.rb:44:in `method_missing'
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-cloudwatch_logs-0.10.1/lib/logstash/inputs/cloudwatch_logs.rb:124:in `process_log'
...

That's in the process_log function, line 127. It looks like you changed those lines to event.set in 6e7cc5d, but removed it two weeks ago with the eb38dfd merge.

Here's the error in 5.3.0:

[2017-04-20T22:41:40,344][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::CloudWatch_Logs log_group=>"staging5-docker", type=>"docker_cloudwatchlogs", id=>"96031aebedd7bbedf8fe2af725162a5ebb66f694-1", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_ab41f289-30d5-4f40-a07f-5a1895e071b5", enable_metric=>true, charset=>"UTF-8">, region=>"us-east-1", interval=>60, log_group_prefix=>false>```

Err for plain text and encoding charset=ISO-8859-1

[406] {"error":"Content-Type header [text/plain; charset=ISO-8859-1] is not supported","status":406} {:class=>"Elasticsearch::Transport::Transport::Errors::NotAcceptable"

[406] {"error":"Content-Type header [text/plain; charset=ISO-8859-1] is not supported","status":406} {:class=>"Elasticsearch::Transport::Transport::Errors::NotAcceptable", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:201:in `__raise_transport_error'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:312:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/client.rb:128:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.18/lib/elasticsearch/api/actions/bulk.rb:90:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in `non_threadsafe_bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:172:in `safe_bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying_submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in `multi_receive'", "org/jruby/RubyArray.java:1653:in `each_slice'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:130:in `worker_multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:114:in `multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:301:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:301:in `output_batch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:232:in `worker_loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:warn}
{"inflight_count"=>68, "stalling_thread_info"=>{"other"=>[{"thread_id"=>20, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:128:in `pop'"}, {"thread_id"=>21, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:50:in `call'"}]}} {:level=>:warn}
```

Support `start_position` setting

When the plugin is starting, if a log_group is encountered that is not in the SinceDB, the start_position setting will allow the user to specify where to begin ingestion.

Available settings would be:

• beginning (default)
• end
• integer (seconds ago from now)

While running, if new log groups are encountered (because you've set log_group_prefix to true) this setting will be ignored, and all log entries in the group will be ingested as we are making the presumption that the stream will all be desired.

This is mimicking the behaviour of the logstash-input-s3 plugin's start_position, while adding the ability to start at now()-n minutes.

Updated: Setback in seconds now, not minutes.

Through the aws role can not get cloudwatch logs

configuration:

input {
cloudwatch_logs {
log_group => "CloudTrail/DefaultLogGroup"
region => "us-east-1"
id => "CloudTrail_log"
tags => "CloudTrail_log"
}
}

output {
elasticsearch {
hosts => ["x.x.x.x"]
index => "test-%{+YYYY.MM.dd}"
}
}

Logstash log can be seen, log_group is correct

Unable to install plugin

I tried to build the gem from gemspec file attached with the project

1. git clone https://github.com/lukewaite/logstash-input-cloudwatch-logs.git
2. [root@xxxxx logstash-input-cloudwatch-logs]# gem build logstash-input-cloudwatch_logs.gemspec
   WARNING: no homepage specified
   Successfully built RubyGem
   Name: logstash-input-cloudwatch_logs
   Version: 0.10.3
   File: logstash-input-cloudwatch_logs-0.10.3.gem
3. Set proxy settings for JRUBY env:
   export JRUBY_OPTS="-J-Dhttp.proxyHost=proxy -J-Dhttp.proxyPort=3128"
4. On trying to install the plugin i get this error:
   [root@ip-10-93-73-114 logstash]# /usr/share/logstash/bin/logstash-plugin install /logstash-input-cloudwatch-logs/logstash-input-cloudwatch_logs-0.10.3.gem
   Validating /logstash-input-cloudwatch-logs/logstash-input-cloudwatch_logs-0.10.3.gem
   Installing logstash-input-cloudwatch_logs
   Error Bundler::InstallError, retrying 1/10
   An error occurred while installing logstash-core-event-java (5.2.1), and Bundler cannot continue.
   Make sure that gem install logstash-core-event-java -v '5.2.1' succeeds before bundling.

[root@ip-10-93-73-114 logstash]# gem install logstash-core-event-java -v '5.2.1'
ERROR: Could not find a valid gem 'logstash-core-event-java' (= 5.2.1), here is why:
Found logstash-core-event-java (5.2.1), but was for platform java

Error: uninitialized constant OpenSSL

Hello, getting the following error on the latest logstash-input-cloudwatch_logs (0.10.3):

logstash_1 | Error: uninitialized constant #Class:0x5192f7f2::OpenSSL
logstash_1 | [2017-05-19T15:30:36,360][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
logstash_1 | Plugin: <LogStash::Inputs::CloudWatch_Logs sincedb_path=>"###", log_group=>"###", tags=>["###"], id=>"#####", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_a82f1752-347a-4484-b6b8-8fbc5c2680c1", enable_metric=>true, charset=>"UTF-8">, region=>"us-east-1", interval=>60, log_group_prefix=>false>

Tests

Need to add test coverage before submission to logstash-plugins group.

Blocks #5

Need to pull recent logs first.

Mr.Luke, Does this input plugin has any setting to enable, so that it can the pull the recent logs first?

Duplicate logs in Elasticsearch?

My ELK cluster is currently distributed across multiple containers. I have multiple Logstash containers, to help deal with Filebeat inputs, along with Cloudwatch Log inputs. Would there be any problem if I was running the cloudwatch_logs input on multiple Logstash containers going to the same Elasticsearch cluster?

I've noticed multiple entries in my Elasticsearch cluster with the same cloudwatch_logs event_id. Without defining a document_id, I've seen multiple Elasticsearch entries for the same Cloudwatch log, based on duplicate cloudwatch_logs event_id.

For now, I've mitigated the duplication by using the cloudwatch_logs event_id as a document_id for Elasticsearch, which would prevent duplicate entries. Is this behavior intended, or is there some other way to prevent duplicate Cloudwatch logs from being entered into my Elasticsearch cluster?

Thanks!

Provide Logstash 5.x Compatibility

Great plugin, thank you for your contribution! We have noticed, however that there are perhaps some (hopefully trivial) dependency issues:

Validating /Users/jay/Downloads/logstash-input-cloudwatch_logs-0.9.1.gem
Installing logstash-input-cloudwatch_logs
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-core":
  In snapshot (Gemfile.lock):
    logstash-core (= 5.0.0)

  In Gemfile:
    logstash-core-plugin-api (>= 0) java depends on
      logstash-core (= 5.0.0) java

    logstash-input-cloudwatch_logs (= 0.9.1) java depends on
      logstash-core (< 3.0.0, >= 1.4.0) java

    logstash-core (>= 0) java

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.

Please let me know if there's any way that Elastic can assist.

Handle Rate Limiting

Currently API rate limiting crashes the plugin. We should catch rate limiting errors and retry after a period.

This causes neither data loss nor duplication, as the entire plugin simply restarts, but can be troublesome if you are running multiple pipelines and they all hit at the same time. A decent workaround currently is to use different IAM credentials for each log group.

Possible Improvement - Bucket S3

Hello,

3 days ago AWS add an option to VPC Flow Logs, it is now possible to send logs into CloudWatch or a bucket S3.

In order to collect very large amount of log files, it can be faster than using the AWS API which limit at 1 Mb logs per request.

https://aws.amazon.com/about-aws/whats-new/2018/08/amazon-vpc-flow-logs-can-now-be-delivered-to-s3/?nc1=h_ls

dynamic log group discovery?

I think it would be great if the plugin would discover all log groups or take a wildcard for log groups (I am thinking something like /aws/lambda/*) so that all new lambdas that are deployed will automatically be consumed.

Do you see any issues with this functionality? I see the open issue about rate-limiting that I suspect would be even bigger of a concern with something like this.

EC2 - not finding role credentials

I' am getting the "Error: unable to sign request without credentials set" error if I do not supply an AWS access key id/secret access key (even though the instance is in a role). I have been researching the issue and have a theory but not the time to verify unfortunately. I also understand the issue may live upstream with the logstash-mixin-aws gem.

My servers are in a VPC with an HTTP proxy (no direct access to the internet). I have tried supplying proxy information via LS_JAVA_OPTIONS env var and /etc/logstash/jvm.options.

I also set
-DnonProxyHost=169.254.169.254 so that the service should NOT use the proxy for Ec2 instance metadata queries. Anything that needs the role credentials would query http://169.254.169.254/latest/meta-data/iam/security-credentials/$ROLENAME to get the access key id and secretaccesskey which leads me to my theory: Either this plugin or the logstash-mixin-aws gem is not respecting the nonProxyHost jvm option.

Let me know if this sounds sane. Thanks for your time

I start logstash with this:
sudo -Hu logstash bin/logstash -f /etc/logstash/conf.d/logstash.conf --path.settings /etc/logstash

Is this plugin added to Logstash Plugin Repo

Hello Team,

Is this plugin added to Logstash Plugin Repo. Also, can you please provide the information on using the plugin. How we will need to install the plug in and ho we can use it to take the cloudwatch logs.

Regards,

Error: can't convert nil into String

I keep getting this error when I start up logstash.
Error: can't convert nil into String

I am also trying to do Docker AWS Logger like #24. I have a single log group example below
input { cloudwatch_logs { log_group => [ "myservice" ] region => "eu-west-1" access_key_id => "" secret_access_key => "" } }

The logs have a prefix like looks like this SERVICE/SERVICE/ID. Am I doing something wrong?

Thanks.

Docker AWS Logging driver

When using the AWS Logging driver, each container creates a log stream within a log group, if a container moves to a different host or gets re provisioned it creates a new log stream, what this results in is lots of log streams being generated.

Whilst running logstash in debug mode I can see that the recursive statement tries to read and parse all the log streams created before it runs out of memory and fails

:message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::CloudWatch_Logs region=>"eu-west-1"

How have others solved this problem?

Do not read all log streams

For example when my log-group contains than 31 log-streams than cloud-watch-logs take only random 30 log stream. It seem to me plugin ignore last log-stream
my configuration:
cloudwatch_logs { log_group => [ "test-app" ] log_group_prefix => true aws_credentials_file => "/aws/crednetials/AWSCredentials.yaml" region => "eu-west-1" type => "test" }
Is there read limit log-stream?

Should use SinceDB time to set start_time parameter for get_log_events call

Rather than re-listing the entire stream content and checking if each event is newer than last_read, you should pass the last_read timestamp into the get_log_events call and let AWS do the filtering for you server-side. Doing the filtering client-side is incredibly wasteful, since you have to re-read and discard ALL the old events every polling interval.

Doesn't shut down nicely

When running service logstash stop the cloudwatch_logs input keeps reading events until the init system kills logstash with 9/SIGKILL.

Can a nice shutdown with proper sincedb support be achieved?

Process new log entries only

Since I'm running logstash in a container, which could potentially die for reasons, it would be good if this plugin would allow you to process new logs only otherwise log messages will get sent to the "output" every time logstash is restarted.

I haven't looked at the code closely yet, but would this be a simple matter of setting last_read to last_ingestion_time for each stream when the app is first started? If process_new_events_only is true, that is.

Clarify documentation on start_position / sincedb

I'm a bit confused by the documentation on how to use start_position.

If I want to re-run logstash and pick up every event from the beginning of my log group, it looks like I need to set start_position to "beginning."

However, the field is marked default, which is confusing to me. Wouldn't that mean every time I run logstash (without defining any value for that field) it would start from the first entry in my log group?

Wouldn't it be better to say the default for start_position is blank/none, meaning it will just pick up on the last date saved in sincedb?

I've just started using this (awesome) plugin so I could be way off in my understanding. Thanks for clarifying.

Events less than 1 ms apart can be missed?

The code saves log.timestamp + 1 in the sincedb, and uses that as start_time for @cloudwatch.filter_log_events. Doesn't that mean that between each time that new logs are fetched, there's a window of up to 1 millisecond where events can be ignored?

As an example, let's say that an event is logged at exactly midnight, and this code fetches logs at 1 microsecond past midnight. Aren't all events from 0:00.000001 to 0:00.000999+ ignored the next time logs are fetched?

If the raw data from Cloudwatch includes some sort of unique ID for each event, it would be best to remove that "+ 1" and be prepared to filter out duplicates based on the ID(s) of the latest seen events having that same timestamp.

Issue with ECS events

Seems like the plugin uniting multiple events into one.
Is there an easy way I can split this to multiple events based on the "cloudwatch_logs.ingestion_time" field?
Thanks

determine_start_position barfs with a number

I tried using this for a start position. Note there are no quotes on the number:

  # start one hour ago
  start_position => 3600

Here's the error/fail:

Error: undefined method `-' for "1504201621152":String
Exception: NoMethodError
Stack: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-cloudwatch_logs-1.0.1/lib/logstash/inputs/cloudwatch_logs.rb:173:in `determine_start_position'
Stack: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-cloudwatch_logs-1.0.1/lib/logstash/inputs/cloudwatch_logs.rb:173:in `determine_start_position'

If I quote it, I get this error:

[2017-08-31T18:07:54,634][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>#<LogStash::ConfigurationError: start_position '3600' is invalid! Must be `beginning`, `end`, or an integer.>,

Documentation

Can you update the README file on how the plugin should be configured/used? Thanks!

How to install this plugin

How to install this plugin?

Error: The request signature we calculated does not match the signature you provided

For some reason I cannot authenticate using inline access_key_id and secret_access_key and get error like below:

[2018-04-30T11:39:36,964][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main Plugin: <LogStash::Inputs::CloudWatch_Logs log_group=>["/aws/lambda/xxxxxxxxx"], log_group_prefix=>true, access_key_id=>"Axxxx", secret_access_key=><password>, region=>"us-east-1", type=>"lambda", id=>"3a9eca949288562397c794252c12cf269461c5c5286f0af4a007fe352ce8ea78", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_47402051-7611-4745-9383-8098c39648c1", enable_metric=>true, charset=>"UTF-8">, role_session_name=>"logstash", interval=>60, start_position=>"beginning"> Error: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. Exception: Aws::CloudWatchLogs::Errors::InvalidSignatureException Stack: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.32/lib/seahorse/client/plugins/raise_response_errors.rb:15:in call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.32/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in call' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.32/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.32/lib/aws-sdk-core/plugins/param_converter.rb:20:in call' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.32/lib/aws-sdk-core/plugins/response_paging.rb:26:in call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.32/lib/seahorse/client/plugins/response_target.rb:21:in call' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.32/lib/seahorse/client/request.rb:70:in send_request'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-core-2.11.32/lib/seahorse/client/base.rb:207:in block in describe_log_groups' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-cloudwatch_logs-1.0.2/lib/logstash/inputs/cloudwatch_logs.rb:141:in block in find_log_groups'
org/jruby/RubyKernel.java:1292:in loop' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-cloudwatch_logs-1.0.2/lib/logstash/inputs/cloudwatch_logs.rb:140:in block in find_log_groups'
org/jruby/RubyArray.java:1734:in each' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-cloudwatch_logs-1.0.2/lib/logstash/inputs/cloudwatch_logs.rb:139:in find_log_groups'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-cloudwatch_logs-1.0.2/lib/logstash/inputs/cloudwatch_logs.rb:115:in run' /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:514:in inputworker'
`

My input conf:
cloudwatch_logs { log_group => "/aws/lambda/xxxx" log_group_prefix => "true" access_key_id => "A....." secret_access_key => "Sxxxxxx...." region => "us-east-1" type => "lambda" }
Any ideas ?