running logcourier through a loadbalancer (ELB)? debugging?

[driskell]

running logcourier through a loadbalancer (ELB)? debugging?

[driskell]

Hello

As far as I know you cannot use AWS ELB for TCP load balancing. It will only do HTTP.

But just in case I'm wrong, you can debug by enabling verbose logging on logstash. The courier plugin will log all new connections and closures to the logstash log. I think it's "--verbose" when you run logstash. Failing that enable debug mode with "--debug" but it can get noisy that one.

Jason

[tedder]

No, ELB works for TCP. Docs here. I've been running lumberjack through it.

Here's the only interesting lines given with --verbose. (the only other lines are for grok patterns).

{:timestamp=>"2014-09-03T18:33:11.446000+0000", :message=>"Using milestone 1 input plugin 'lumberjack'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see http://logstash.net/docs/1.4.1.dev/plugin-milestones", :level=>:warn}
{:timestamp=>"2014-09-03T18:33:11.615000+0000", :message=>"Using milestone 2 output plugin 'elasticsearch_http'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.1.dev/plugin-milestones", :level=>:warn}
{:timestamp=>"2014-09-03T18:33:11.654000+0000", :message=>"Using milestone 1 input plugin 'courier'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see http://logstash.net/docs/1.4.1.dev/plugin-milestones", :level=>:warn}
{:timestamp=>"2014-09-03T18:33:15.489000+0000", :message=>"Starting lumberjack input listener", :address=>"0.0.0.0:5554", :level=>:info}
{:timestamp=>"2014-09-03T18:33:15.670000+0000", :message=>"Starting courier input listener", :address=>"0.0.0.0:5550", :level=>:info}

Here's the output with --debug.

{:timestamp=>"2014-09-03T18:38:49.390000+0000", :message=>"Using milestone 1 input plugin 'courier'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see http://logstash.net/docs/1.4.1.dev/plugin-milestones", :level=>:warn, :file=>"logstash/config/mixin.rb", :line=>"209"}
{:timestamp=>"2014-09-03T18:38:49.394000+0000", :message=>"config LogStash::Codecs::Plain/@charset = \"UTF-8\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
{:timestamp=>"2014-09-03T18:38:49.395000+0000", :message=>"config LogStash::Inputs::Courier/@port = 5550", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
{:timestamp=>"2014-09-03T18:38:49.396000+0000", :message=>"config LogStash::Inputs::Courier/@transport = \"tcp\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
{:timestamp=>"2014-09-03T18:38:49.397000+0000", :message=>"config LogStash::Inputs::Courier/@ssl_key = \"/etc/ssl/private/logstash-forwarder.key\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
{:timestamp=>"2014-09-03T18:38:49.398000+0000", :message=>"config LogStash::Inputs::Courier/@debug = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
{:timestamp=>"2014-09-03T18:38:49.399000+0000", :message=>"config LogStash::Inputs::Courier/@codec = <LogStash::Codecs::Plain charset=>\"UTF-8\">", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
{:timestamp=>"2014-09-03T18:38:49.400000+0000", :message=>"config LogStash::Inputs::Courier/@add_field = {}", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
{:timestamp=>"2014-09-03T18:38:49.401000+0000", :message=>"config LogStash::Inputs::Courier/@host = \"0.0.0.0\"", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
{:timestamp=>"2014-09-03T18:38:49.402000+0000", :message=>"config LogStash::Inputs::Courier/@ssl_verify = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
{:timestamp=>"2014-09-03T18:38:49.403000+0000", :message=>"config LogStash::Inputs::Courier/@ssl_verify_default_ca = false", :level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
{:timestamp=>"2014-09-03T18:38:53.121000+0000", :message=>"Starting lumberjack input listener", :address=>"0.0.0.0:5554", :level=>:info, :file=>"logstash/inputs/lumberjack.rb", :line=>"38"}
{:timestamp=>"2014-09-03T18:38:53.323000+0000", :message=>"Starting courier input listener", :address=>"0.0.0.0:5550", :level=>:info, :file=>"logstash/inputs/courier.rb", :line=>"66"}

I don't see anything really interesting in there.

When I set the logcourier forwarder to 'localhost' I get the expected debug lines; when it's set to go through the ELB I get.. nothing from the server side.

Ugh- solved. I was missing one security group setting (the ELB to the instances from the point of view of the instance). Wow. Well, at least I have learned a lot by reading through both the lumberjack and courier server code!

[driskell]

Ah! That explains the immediate EOF then!

I stand corrected on the ELB usage then! Never knew it could. I wonder how it works with the max connection time (unless that's specific to HTTP) - I'll have to read the docs!

Let us know if you do have any other issues though and I'll be glad to help.

Jason