Protocol error on connection with very long log lines

[driskell]

Protocol error on connection with very long log lines

[rwilliamselab]

On the log-courier side I get this when it tries to connect. I've restarted both the log-courier forwarder and the logstash receiver.
Connected to xx.xx.xx.xx
2014/07/30 08:53:42.707864 Transport error, will try again: write tcp xx.xx.xx.xx:pp: connection reset by peer

[driskell]

Hi @rwilliamselab

Does this occur constantly or after a period of time?
When you restart the client does it occur immediately or does it send logs and then do it after a bit?
I'll see about getting more logging on ProtocolError.

At first glance it looks like something is killing the connection. Maybe a firewall or something.

Jason

[rwilliamselab]

It worked fine for about 6 hours and now won¹t restart without the error

Richard Williams
Director of Product Development
http://elabsolutions.com/
610 Airport Road€ Suite 200 € Huntsville, AL 35802
256.337.9952 (Direct) €270.747.9952 (Fax)

http://www.linkedin.com/pub/elab-solutions/84/194/723
https://twitter.com/eLabSolutions
https://www.facebook.com/pages/eLab-Solutions/156689697863760
[email protected] | www.elabsolutions
http://elabsolutions.com/ | Map
<https://www.google.com/maps/preview/place/610+Airport+Rd+SW,+Huntsville,+AL
+35802/@34.691661,-86.571944,17z/data=!3m1!4b1!4m2!3m1!1s0x8862132fc72c7b07:
0xb1748584e03640e3>
This communication may contain information or materials that are subject to
confidentiality, privilege or intellectual property restrictions. Please
refer to our legal notices http://elabsolutions.com/legalnotices for more
detail.

From: driskell [email protected]
Reply-To: driskell/log-courier
<r[e-mail]
hub.com>
Date: Wednesday, July 30, 2014 at 12:16 PM
To: driskell/log-courier [email protected]
Cc: Richard Williams [email protected]
Subject: Re: [log-courier] Protocol error on connection (#28)

Hi @rwilliamselab https://github.com/rwilliamselab

Does this occur constantly or after a period of time?
When you restart the client does it occur immediately or does it send logs
and then do it after a bit?
I'll see about getting more logging on ProtocolError.

At first glance it looks like something is killing the connection. Maybe a
firewall or something.

Jason

‹
Reply to this email directly or view it on GitHub
#28 (comment) .

[rwilliamselab]

Have the same thing occurring on other instances too on different machines

Richard Williams
Director of Product Development
http://elabsolutions.com/
610 Airport Road€ Suite 200 € Huntsville, AL 35802
256.337.9952 (Direct) €270.747.9952 (Fax)

http://www.linkedin.com/pub/elab-solutions/84/194/723
https://twitter.com/eLabSolutions
https://www.facebook.com/pages/eLab-Solutions/156689697863760
[email protected] | www.elabsolutions
http://elabsolutions.com/ | Map
<https://www.google.com/maps/preview/place/610+Airport+Rd+SW,+Huntsville,+AL
+35802/@34.691661,-86.571944,17z/data=!3m1!4b1!4m2!3m1!1s0x8862132fc72c7b07:
0xb1748584e03640e3>
This communication may contain information or materials that are subject to
confidentiality, privilege or intellectual property restrictions. Please
refer to our legal notices http://elabsolutions.com/legalnotices for more
detail.

From: driskell [email protected]
Reply-To: driskell/log-courier
<r[e-mail]
hub.com>
Date: Wednesday, July 30, 2014 at 12:16 PM
To: driskell/log-courier [email protected]
Cc: Richard Williams [email protected]
Subject: Re: [log-courier] Protocol error on connection (#28)

Hi @rwilliamselab https://github.com/rwilliamselab

Does this occur constantly or after a period of time?
When you restart the client does it occur immediately or does it send logs
and then do it after a bit?
I'll see about getting more logging on ProtocolError.

At first glance it looks like something is killing the connection. Maybe a
firewall or something.

Jason

‹
Reply to this email directly or view it on GitHub
#28 (comment) .

[rwilliamselab]

Maybe this will help. We tried using both logstash-forwarder and logstash
with lumberjack output and began to get client shutdowns much like with
log-courier. However the reported errors on the logstash server were:

Zlib::DataError: invalid distance code and logstash Zlib::DataError:
invalid block type that causes the logstash service to stop.

Richard Williams
Director of Product Development
http://elabsolutions.com/
610 Airport Road€ Suite 200 € Huntsville, AL 35802
256.337.9952 (Direct) €270.747.9952 (Fax)

http://www.linkedin.com/pub/elab-solutions/84/194/723
https://twitter.com/eLabSolutions
https://www.facebook.com/pages/eLab-Solutions/156689697863760
[email protected] | www.elabsolutions
http://elabsolutions.com/ | Map
<https://www.google.com/maps/preview/place/610+Airport+Rd+SW,+Huntsville,+AL
+35802/@34.691661,-86.571944,17z/data=!3m1!4b1!4m2!3m1!1s0x8862132fc72c7b07:
0xb1748584e03640e3>
This communication may contain information or materials that are subject to
confidentiality, privilege or intellectual property restrictions. Please
refer to our legal notices http://elabsolutions.com/legalnotices for more
detail.

From: driskell [email protected]
Reply-To: driskell/log-courier
<r[e-mail]
hub.com>
Date: Wednesday, July 30, 2014 at 12:16 PM
To: driskell/log-courier [email protected]
Cc: Richard Williams [email protected]
Subject: Re: [log-courier] Protocol error on connection (#28)

Hi @rwilliamselab https://github.com/rwilliamselab

Does this occur constantly or after a period of time?
When you restart the client does it occur immediately or does it send logs
and then do it after a bit?
I'll see about getting more logging on ProtocolError.

At first glance it looks like something is killing the connection. Maybe a
firewall or something.

Jason

‹
Reply to this email directly or view it on GitHub
#28 (comment) .

[rwilliamselab]

One last clue

I wondered how you kept a marker on the current files you were shipping and
found a .log-courier file which contains offsets in our logs. If I remove
the file the service starts up again with no problem. Here is the content of
the placeholder

{"/opt/jboss/domain/servers/server-four/log/server.log":{"inode":133638,"dev
ice":64768,"source":"/opt/jboss/domain/servers/server-four/log/server.log","
offset":87849449}
,"/opt/node/quikshell/rotating.log":{"inode":666812,"device":64768,"source":
"/opt/node/quikshell/rotating.log","offset":2716}}

Thanks
Richard

Richard Williams
Director of Product Development
http://elabsolutions.com/
610 Airport Road€ Suite 200 € Huntsville, AL 35802
256.337.9952 (Direct) €270.747.9952 (Fax)

http://www.linkedin.com/pub/elab-solutions/84/194/723
https://twitter.com/eLabSolutions
https://www.facebook.com/pages/eLab-Solutions/156689697863760
[email protected] | www.elabsolutions
http://elabsolutions.com/ | Map
<https://www.google.com/maps/preview/place/610+Airport+Rd+SW,+Huntsville,+AL
+35802/@34.691661,-86.571944,17z/data=!3m1!4b1!4m2!3m1!1s0x8862132fc72c7b07:
0xb1748584e03640e3>
This communication may contain information or materials that are subject to
confidentiality, privilege or intellectual property restrictions. Please
refer to our legal notices http://elabsolutions.com/legalnotices for more
detail.

From: driskell [email protected]
Reply-To: driskell/log-courier
<r[e-mail]
hub.com>
Date: Wednesday, July 30, 2014 at 12:16 PM
To: driskell/log-courier [email protected]
Cc: Richard Williams [email protected]
Subject: Re: [log-courier] Protocol error on connection (#28)

Hi @rwilliamselab https://github.com/rwilliamselab

Does this occur constantly or after a period of time?
When you restart the client does it occur immediately or does it send logs
and then do it after a bit?
I'll see about getting more logging on ProtocolError.

At first glance it looks like something is killing the connection. Maybe a
firewall or something.

Jason

‹
Reply to this email directly or view it on GitHub
#28 (comment) .

[rwilliamselab]

Hey

Your offset marker was very helpful in nailing down the offender. It looks
like our developers have blasted a single line in the log that contains an
encoded pdf image. The problem is apparently the size of the line. Is there
anyway to prefilter the lines to eliminate the overrun caused by bad log
files? Apparently all of the logstash cllients suffer from the problem.

Thanks
Richard

Richard Williams
Director of Product Development
http://elabsolutions.com/
610 Airport Road€ Suite 200 € Huntsville, AL 35802
256.337.9952 (Direct) €270.747.9952 (Fax)

http://www.linkedin.com/pub/elab-solutions/84/194/723
https://twitter.com/eLabSolutions
https://www.facebook.com/pages/eLab-Solutions/156689697863760
[email protected] | www.elabsolutions
http://elabsolutions.com/ | Map
<https://www.google.com/maps/preview/place/610+Airport+Rd+SW,+Huntsville,+AL
+35802/@34.691661,-86.571944,17z/data=!3m1!4b1!4m2!3m1!1s0x8862132fc72c7b07:
0xb1748584e03640e3>
This communication may contain information or materials that are subject to
confidentiality, privilege or intellectual property restrictions. Please
refer to our legal notices http://elabsolutions.com/legalnotices for more
detail.

From: driskell [email protected]
Reply-To: driskell/log-courier
<r[e-mail]
hub.com>
Date: Wednesday, July 30, 2014 at 12:16 PM
To: driskell/log-courier [email protected]
Cc: Richard Williams [email protected]
Subject: Re: [log-courier] Protocol error on connection (#28)

Hi @rwilliamselab https://github.com/rwilliamselab

Does this occur constantly or after a period of time?
When you restart the client does it occur immediately or does it send logs
and then do it after a bit?
I'll see about getting more logging on ProtocolError.

At first glance it looks like something is killing the connection. Maybe a
firewall or something.

Jason

‹
Reply to this email directly or view it on GitHub
#28 (comment) .

[driskell]

@rwilliamselab Nice find! Thanks!

This is something that we should try to handle as gracefully as possible. I will check the code soon. I know there are 1MB hard coded limits in places but they may not be interacting nicely and causing this issue as opposed to failing gracefully to the next new line.

Jason

[driskell]

I'm going to close this in favour of #40 as the error handling has changed slightly so it now reports "Protocol error - packet too large" to show the problem more explicitly.

I'll be working on the changes required for this next.