Increase verbosity?

[logstash-plugins]

Increase verbosity?

[DanielRedOak]

If you throw --debug on the end of the startup for logstash, you'll get quite a bit more information. Also for the enumeration issues, I am 90% done with moving this plugin from aws-sdk v1 to v2 which should increase speeds quite a bit. Just waiting on testing

[jmreicha]

@DanielRedOak Thanks for following up. Here is what I get with the debug flag turned on. It just sits there, does it mean it is processing? The bucket is pretty huge.

sing version 0.1.x input plugin 's3'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}
Using version 0.1.x codec plugin 'plain'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}
Using version 0.1.x output plugin 'stdout'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}
Using version 0.1.x codec plugin 'rubydebug'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}
Registering s3 input {:bucket=>"my-cloudtrai-bucket", :region=>"us-east-1", :level=>:info}
Pipeline started {:level=>:info}
Logstash startup completed
Using default generated file for the sincedb {:filename=>"/root/.sincedb_dc760775953947267f2d81029009a3a0", :level=>:info}

Here is my logstash config to test this out.

nput {
    s3 {
        bucket => "my-cloudtrai-bucket"
        access_key_id => "ID"
        secret_access_key => "KEY"
        type => "cloudtrail"
    }
}

output {
    stdout {
        codec => rubydebug
    }
}

[jmreicha]

Just an update. I tried testing with a much smaller bucket and the plugin responded so I think it is at least working. Interestingly enough it is throwing some errors now. I will look in to the errors but I figured I would at least post an update.

Here's what the errors are reporting:

Error: No such file or directory - /tmp/logstash/ELBAccessLogTestFile {:level=>:error}
A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::S3 bucket=>"my-cloudtrail-bucket", access_key_id=>"ID", secret_access_key=>"KEY", type=>"cloudtrail", region=>"us-east-1", temporary_directory=>"/tmp/logstash">
  Error: No such file or directory - /tmp/logstash/ELBAccessLogTestFile {:level=>:error}
A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::S3 bucket=>"my-cloudtrail-bucket", access_key_id=>"ID", secret_access_key=>"KEY", type=>"cloudtrail", region=>"us-east-1", temporary_directory=>"/tmp/logstash">
  Error: No such file or directory - /tmp/logstash/ELBAccessLogTestFile {:level=>:error}
A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::S3 bucket=>"my-cloudtrail-bucket", access_key_id=>"ID", secret_access_key=>"KEY", type=>"cloudtrail", region=>"us-east-1", temporary_directory=>"/tmp/logstash">

Got it working by explicity creating /tmp/logstash.

[vanga]

I wonder if there is any way to change this location /tmp/logstash to something else, so it persists with system reboots. tmp/ files are getting cleaned up in a reboot.