zwzch/fluent-plugin-pulsar

fluent pulsar input

fluent-plugin-pulsar's Issues

Frequent connection drop | delay in message read when try to read bulk messages | Garbage string attached in start of the message

I found frequent connection drop while using this plugin
When I published bulk messages, plugin reads around 1k and seats ideal for a while and then again starts reading messages
Garbage data attached in front of the message

Fluentd is restarting , because of the unhandled exception while socket connection reading bytes of data on PulsarClient.rb

Function 'command_message' on PulsarClient.rb, is reading the messages byte by byte. But even though , there is no message published by Pulsar, it is trying to unpack the bytes of data , which is nil. So because of this exception, fluentd restarting continuosly
Line no: 273-275 on /plugin/lib/pulsar_client/PulsarClient.rb. Stack trace is added below :

2020-01-02 12:38:11 +0530 [error]: #0 unexpected error error_class=NoMethodError error="undefined method unpack' for nil:NilClass" 2020-01-02 12:38:11 +0530 [error]: #0 fluentd-plugins/plugins/pulsar_client/PulsarClient.rb:273:in command_message'
2020-01-02 12:38:11 +0530 [error]: #0 fluentd-plugins/plugins/pulsar_client/PulsarClient.rb:367:in get_message' 2020-01-02 12:38:11 +0530 [error]: #0 fluentd-plugins/plugins/in_pulsar.rb:58:in start'
2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.8.0/lib/fluent/root_agent.rb:201:in block in start' 2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.8.0/lib/fluent/root_agent.rb:190:in block (2 levels) in lifecycle'
2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.8.0/lib/fluent/root_agent.rb:189:in each' 2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.8.0/lib/fluent/root_agent.rb:189:in block in lifecycle'
2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.8.0/lib/fluent/root_agent.rb:176:in each' 2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.8.0/lib/fluent/root_agent.rb:176:in lifecycle'
2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.8.0/lib/fluent/root_agent.rb:200:in start' 2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.8.0/lib/fluent/engine.rb:193:in start'
2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.8.0/lib/fluent/engine.rb:146:in run' 2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.8.0/lib/fluent/supervisor.rb:534:in block in run_worker'
2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.8.0/lib/fluent/supervisor.rb:739:in main_process' 2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.8.0/lib/fluent/supervisor.rb:528:in run_worker'
2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.8.0/lib/fluent/command/fluentd.rb:338:in <top (required)>' 2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in require'
2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in require' 2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.8.0/bin/fluentd:8:in <top (required)>'
2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in load' 2020-01-02 12:38:11 +0530 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in <top (required)>'
2020-01-02 12:38:11 +0530 [error]: #0 /usr/sbin/td-agent:7:in load' 2020-01-02 12:38:11 +0530 [error]: #0 /usr/sbin/td-agent:7:in

'