emsearcy / fluent-plugin-gelf Goto Github PK
View Code? Open in Web Editor NEWBuffered fluentd output plugin to GELF (Graylog2)
License: Apache License 2.0
Buffered fluentd output plugin to GELF (Graylog2)
License: Apache License 2.0
When incoming JSON message contains "level":6
-- numerical level, I get
... [warn]: emit transaction failed: error_class=NoMethodError error="undefined method `downcase' for 6:Fixnum" ...
proposed change in line 49:
case "#{v}".downcase
I'm getting the following type of stacktrace when running under fluentd 0.14.7.
2016-10-08 13:51:01 +0000 [warn]: failed to flush the buffer. plugin_id="object:62d79288514" retry_time=0 next_retry=2016-10-08 13:51:02 +0000 chunk="53e5ad1234b711b05fd1857ecc4320c0" error_class=NoMethodError error="undefined method `msgpack_each' for #<Fluent::Plugin::Buffer::FileChunk:0x0071f4d85c1fe0>\nDid you mean? msgpack_packer"
2016-10-08 13:51:01 +0000 [warn]: /usr/lib64/ruby/gems/2.3.0/gems/fluent-plugin-gelf-0.1.2/lib/fluent/plugin/out_gelf.rb:101:in `write'
2016-10-08 13:51:01 +0000 [warn]: /usr/lib64/ruby/gems/2.3.0/gems/fluentd-0.14.7/lib/fluent/compat/output.rb:127:in `write'
2016-10-08 13:51:01 +0000 [warn]: /usr/lib64/ruby/gems/2.3.0/gems/fluentd-0.14.7/lib/fluent/plugin/output.rb:995:in `try_flush'
2016-10-08 13:51:01 +0000 [warn]: /usr/lib64/ruby/gems/2.3.0/gems/fluentd-0.14.7/lib/fluent/plugin/output.rb:1188:in `flush_thread_run'
2016-10-08 13:51:01 +0000 [warn]: /usr/lib64/ruby/gems/2.3.0/gems/fluentd-0.14.7/lib/fluent/plugin/output.rb:393:in `block (2 levels) in start'
2016-10-08 13:51:01 +0000 [warn]: /usr/lib64/ruby/gems/2.3.0/gems/fluentd-0.14.7/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
I haven't looked into the root cause yet, but it seems this line isn't working anymore: https://github.com/emsearcy/fluent-plugin-gelf/blob/master/lib/fluent/plugin/out_gelf.rb#L101
When fluentd is configured with a tcp gelf output, that cannot connect,
fluentd errors out, failing to start.
Centos 7 td-agent init scripts keep restarting fluentd, but it is not the expected behavior, as it halts all fluentd log processing and possibly results in message loss even in unrelated pipelines.
It may be related to #7, as fluentd core seems to be interacting improperly with the gelf library output connection. The problem may be in a difference between what fluentd expects and how gelf library behaves, that the plugin doesn't account for, but I don't know much about them.
2017-01-02 23:09:24 +0100 [error]: unexpected error error_class=Errno::ECONNREFUSED error="Connection refused - connect(2) for \"192.168.36.4\" port 12201"
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gelf-3.0.0/lib/gelf/transport/tcp.rb:31:in `initialize'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gelf-3.0.0/lib/gelf/transport/tcp.rb:31:in `new'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gelf-3.0.0/lib/gelf/transport/tcp.rb:31:in `connect'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gelf-3.0.0/lib/gelf/transport/tcp.rb:15:in `block (2 levels) in addresses='
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gelf-3.0.0/lib/gelf/transport/tcp.rb:15:in `map'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gelf-3.0.0/lib/gelf/transport/tcp.rb:15:in `block in addresses='
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gelf-3.0.0/lib/gelf/transport/tcp.rb:13:in `tap'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gelf-3.0.0/lib/gelf/transport/tcp.rb:13:in `addresses='
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gelf-3.0.0/lib/gelf/transport/tcp.rb:9:in `initialize'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gelf-3.0.0/lib/gelf/notifier.rb:138:in `new'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gelf-3.0.0/lib/gelf/notifier.rb:138:in `create_sender'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/gelf-3.0.0/lib/gelf/notifier.rb:36:in `initialize'
2017-01-02 23:09:24 +0100 [error]: /etc/td-agent/plugin/out_gelf.rb:42:in `new'
2017-01-02 23:09:24 +0100 [error]: /etc/td-agent/plugin/out_gelf.rb:42:in `start'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/compat/call_super_mixin.rb:42:in `start'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/root_agent.rb:138:in `block in start'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/root_agent.rb:117:in `block (2 levels) in lifecycle'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/agent.rb:117:in `block (2 levels) in lifecycle'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/agent.rb:116:in `each'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/agent.rb:116:in `block in lifecycle'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/agent.rb:109:in `each'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/agent.rb:109:in `lifecycle'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/root_agent.rb:116:in `block in lifecycle'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/root_agent.rb:113:in `each'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/root_agent.rb:113:in `lifecycle'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/root_agent.rb:137:in `start'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/engine.rb:211:in `start'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/engine.rb:175:in `run'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/supervisor.rb:697:in `run_engine'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/supervisor.rb:454:in `block in run_worker'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/supervisor.rb:626:in `call'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/supervisor.rb:626:in `main_process'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/supervisor.rb:449:in `run_worker'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/lib/fluent/command/fluentd.rb:288:in `<top (required)>'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.7/bin/fluentd:5:in `<top (required)>'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/bin/fluentd:23:in `load'
2017-01-02 23:09:24 +0100 [error]: /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
2017-01-02 23:09:24 +0100 [error]: /usr/sbin/td-agent:7:in `load'
2017-01-02 23:09:24 +0100 [error]: /usr/sbin/td-agent:7:in `<main>'
2017-01-02 23:09:24 +0100 [error]: unexpected error error="Connection refused - connect(2) for \"192.168.36.4\" port 12201"
2017-01-02 23:09:24 +0100 [error]: suppressed same stacktrace
hi i have trouble when im clone source and copy to path plugin fluentd like documentation official, when im trying debug i have error still no have found type=gelf ??? what i've missing ?
2016-09-21 08:26:43 +0700 [debug]: plugin/out_copy.rb:44:block in configure: adding store type="gelf"
2016-09-21 08:26:43 +0700 [error]: fluent/supervisor.rb:369:rescue in main_process: config error file="./fluent/fluent.conf" error="Unknown output plugin 'gelf'. Run 'gem search -rd fluent-plugin' to find plugins"
thanks
env : freebsd 10.x amd64
fluend v0.12 (install by gem)
I could not find the license information this repository is released under.
Could you add a file to describe the license, or put it in README?
Starting with commit d3d1992, I see errors like the below when using a fluentd configuration that includes a time_format
parameter and the log line never makes it to the target (Graylog). Reverting to 1377c3e resolved this issue for me.
Versions:
# dpkg --list | grep td-agent
ii td-agent 2.3.2-0 amd64 Treasure Agent: A data collector for Treasure Data
# td-agent --version
td-agent 0.12.26
# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 7.9 (wheezy)
Release: 7.9
Codename: wheezy
2016-09-22 02:43:28 +0000 [warn]: emit transaction failed: error_class=NameError error="uninitialized constant Fluent::EventTime" tag="graylog.foo"
2016-09-22 02:43:28 +0000 [warn]: /etc/td-agent/plugin/out_gelf.rb:48:in `format'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/output.rb:282:in `block in format_stream'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/event.rb:130:in `call'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/event.rb:130:in `block in each'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/event.rb:129:in `each'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/event.rb:129:in `each'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/output.rb:281:in `format_stream'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/output.rb:267:in `emit'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/output.rb:42:in `next'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/plugin/out_copy.rb:78:in `emit'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/event_router.rb:90:in `emit_stream'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/plugin/in_tail.rb:269:in `receive_lines'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/plugin/in_tail.rb:373:in `call'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/plugin/in_tail.rb:373:in `wrap_receive_lines'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/plugin/in_tail.rb:568:in `call'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/plugin/in_tail.rb:568:in `on_notify'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/plugin/in_tail.rb:399:in `on_notify'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/plugin/in_tail.rb:500:in `call'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/plugin/in_tail.rb:500:in `on_change'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.4/lib/cool.io/loop.rb:88:in `run_once'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.4/lib/cool.io/loop.rb:88:in `run'
2016-09-22 02:43:28 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/plugin/in_tail.rb:253:in `run'
Here is my source configuration:
<source>
type tail
format /^(?<client_ip>[^ ]*) - (?<user>[^ ]*) \[(?<time>[^\]]*)\]\s+"(?<method>\S+)(?: +(?<path>[^\"]*) +\S*)? (?<status>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<user-agent>[^\"]*)" (?<hostname>[^ "]*) (?<request-time>[^ "]*) (?<upstream-time>[^ "]*)/
time_format %d/%b/%Y:%T %z
path /var/log/nginx/*.log
pos_file /var/log/td-agent/nginx.pos
tag graylog.foo
</source>
And my match:
<match graylog.**>
type copy
<store>
type gelf
host graylog.jana.dev
port 12201
flush_interval 5s
</store>
</match>
And a log line:
10.11.12.13 - - [22/Sep/2016:02:27:35 +0000] "POST /foo/foo HTTP/1.1" 200 14 "-" "Dalvik/2.1.0 (Linux; U; Android 6.0.1; SM-J700M Build/MMB29K)" foo 0.014 0.014 .
Looks like when 'host' to '_host' is not working in graylog-server 1.1.5.
graylog-web-interface 1.1.5 cant show _host and _version.
Hi,
I am experiencing UndefinedConversionError
errors when emitting log messages containing non-ASCII characters, such as Russian or Japanese text.
For example:
Log message contains "привет"...
2015-01-31 17:31:58 +0000 [warn]: emit transaction failed error_class=Encoding::UndefinedConversionError error=#<Encoding::UndefinedConversionError: "\xD0" from ASCII-8BIT to UTF-8>
2015-01-31 17:31:58 +0000 [warn]: /etc/td-agent/plugin/out_gelf.rb:77:in `encode'
2015-01-31 17:31:58 +0000 [warn]: /etc/td-agent/plugin/out_gelf.rb:77:in `to_json'
2015-01-31 17:31:58 +0000 [warn]: /etc/td-agent/plugin/out_gelf.rb:77:in `format'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/output.rb:259:in `block in format_stream'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `call'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:55:in `each'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/output.rb:258:in `format_stream'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/output.rb:244:in `emit'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/match.rb:36:in `emit'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:160:in `emit_stream'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:140:in `emit'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-reformer-0.4.0/lib/fluent/plugin/out_record_reformer.rb:90:in `block in emit'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:129:in `call'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:129:in `block in each'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:128:in `each'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/event.rb:128:in `each'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-reformer-0.4.0/lib/fluent/plugin/out_record_reformer.rb:87:in `emit'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/output.rb:33:in `next'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-reformer-0.4.0/lib/fluent/plugin/out_record_reformer.rb:92:in `emit'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/output.rb:33:in `next'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/out_copy.rb:73:in `emit'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/match.rb:36:in `emit'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/engine.rb:160:in `emit_stream'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-mongo-slow-query-0.0.5/lib/fluent/plugin/in_mongo_slow_query.rb:71:in `receive_lines'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_tail.rb:1031:in `call'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_tail.rb:1031:in `on_notify'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_tail.rb:855:in `on_notify'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_tail.rb:966:in `call'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_tail.rb:966:in `on_change'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.2.4/lib/cool.io/loop.rb:88:in `run_once'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.2.4/lib/cool.io/loop.rb:88:in `run'
2015-01-31 17:31:58 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/in_tail.rb:768:in `run'
If my input is turned off to where fluentd is sending the chunks, fluentd doesn't retry the chunks and anyways if using buf_file, the file is deleted after flush_interval. I am assuming fluentd doen't mark the write as failed in this case and deletes the buffer file.
IllegalArgumentException[Document contains at least one immense term in field="response" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[123, 34, 115, 116, 97, 116, 117, 115, 34, 58, 50, 48, 48, 44, 34, 100, 97, 116, 97, 34, 58, 91, 123, 34, 105, 100, 34, 58, 49, 48]...', original message: bytes can be at most 32766 in length; got 465638]; nested: MaxBytesLengthExceededException[bytes can be at most 32766 in length; got 465638];
Can you have optional for truncate message or put original data to non-indexed field and truncate the message in indexed field?
Proposed change in line 72:
gelfentry[k[0] == '_'? k: '_' + k] = v
I install this plugin in /etc/td-agent/out_gelf.rb, but when i run service td-agent restart, Then, this is error
Restarting td-agent: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- gelf (LoadError)
from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
from /etc/td-agent/plugin/out_gelf.rb:15:in `initialize'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/plugin.rb:128:in `new'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/plugin.rb:128:in `new_impl'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/plugin.rb:59:in `new_output'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/plugin/out_copy.rb:46:in `block in configure'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/plugin/out_copy.rb:39:in `each'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/plugin/out_copy.rb:39:in `configure'
look like it require gelf, can someone help me
Currently, in_gelftail
inherits Fluent::TailInput
which is v0.12 manner of in_tail
.
But Fluentd v0.14 does not provide Fluent::TailInput
class.
It had been migrated into Fluent::Plugin::TailInput
: https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/in_tail.rb#L31-L32
@emsearcy Any thoughts?
Hi,
Can you start generating Git tags that correspond with the versions you plan on releasing to Rubygems? (Thanks, for the new gemspec)
I'd like to be able to get a particular version of this repository... but without having to reference an entire Git commit hash.
I'm sorry if this sounds trivial, but it's the pattern I've come to expect from the majority of open-source projects I work with.
Hello @emsearcy,
I am getting the following error.
2016-11-29 16:33:07 +0300 [warn]: emit transaction failed: error_class=NoMethodError error="undefined method
bytesize' for nil:NilClass" tag="requests.domain" 2016-11-29 16:33:07 +0300 [warn]: suppressed same stacktrace 2016-11-29 16:33:07 +0300 [error]: forward error error=#<NoMethodError: undefined method
bytesize' for nil:NilClass> error_class=NoMethodError
2016-11-29 16:33:07 +0300 [error]: suppressed same stacktrace
Hello. I am new with fluentd. I use td-agent v0.12 with ubuntu
Here is my td-agent.conf. My problem is , i want config fluent to buffer data if can not reach graylog in buffer_path and re-transfer to host when reached. But it auto purse chunk event can not reach my hosts. Plz help. Thank you
<source>
@type forward
</source>
<match docker>
@type gelf
host 172.17.0.4
port 12201
protocol tcp
flush_interval 10s
buffer_type file
buffer_path /var/log/td-agent/buffer/td.*.buffer
</match>
Hi guys, I have problem with your plugin. When I run fluentd with with gelf sending logs on graylog my kubernetes system is drastically slower. Is there a way to optimize it?
This is my td-agent conf file
<source>
type tail
path /var/log/containers/*.log
pos_file /var/log/containers.log.pos
time_format %Y-%m-%dT%H:%M:%S.%NZ
tag containers-*
format json
</source>
<match containers-**>
type copy
<store>
type gelf
host graylog.example.com
port 12201
flush_interval 10s
buffer_queue_limit 8
buffer_chunk_limit 2M
num_threads 8
</store>
</match>
Thanks in advance.
Hi Eric-
Thanks for the awesome work. I noticed that inside lib/fluent/plugin are three plugins. I would say in_gelftail
and out_gelf2file
can be yanked out of the repo for the following reasons:
in_gelftail
can be implement with a combination of in_tail (with custom parsers) and record_reformer now. Perhaps this was not the case when the plugin was written.out_gelf2file
seems to be a variant of out_file
. I believe it is for persisting log files in gelf-compatible formats locally, but it is not strictly to send data to Graylog2.I propose that this plugin focus on out_gelf
only (and I am more than happy to add tests and package it for Rubygems and send pull requests).
Let me know what you think.
Hi,
There is a problem with extracting Fields in Graylog.
You can see the issue in the following recipe: http://www.fluentd.org/guides/recipes/graylog2
The last picture shows the Message-Field, which looks very messed up: http://www.fluentd.org/images/graylog2-graph.png
I tried it at my own system and it looks exactly like in the last picture.
When using the following nc example message it looks all fine:
echo '{"version": "1.1","host":"example.org","short_message":"A short message that helps you identify what is going on","full_message":"Backtrace here\n\nmore stuff","level":1,"_user_id":9001,"_some_info":"foo","_some_env_var":"bar"}' | nc -w 1 -u localhost 12200
So my guess is that either out_gelf plugin is bugged or the ruby gem gelf.
regards.
I'm just getting started with Graylog and fluent but is there any way to ensure logs get delivered to the Graylog server even if the Graylog server is temporarily unreachable? Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.