repeatedly/fluent-plugin-multi-format-parser

Multi format parser for Fluentd

Multi format parser plugin for Fluentd

Parse format mixed logs.

Requirements

fluent-plugin-multi-format-parser	fluentd	ruby
>= 1.0.0	>= v0.14.0	>= 2.1
< 1.0.0	>= v0.12.0	>= 1.9

Installation

Use RubyGems:

fluent-gem install fluent-plugin-multi-format-parser

Configuration

This plugin is a parser plugin. After installed, you can use multi_format in <parse> supported plugins. Use multiple <pattern>s to specify multiple parser formats.

<source>
  @type udp
  tag logs.multi

  <parse>
    @type multi_format
    <pattern>
      format apache
    </pattern>
    <pattern>
      format json
      time_key timestamp
    </pattern>
    <pattern>
      format none
    </pattern>
  </parse>
</source>

multi_format tries pattern matching from top to bottom and returns parsed result when matched.

Available format patterns and parameters are depends on Fluentd parsers. See parser plugin document for more details.

For v1.0

Put <pattern>s inside <parse>.

<filter app.**>
  @type parser
  key_name message
  <parse>
    @type multi_format
    <pattern>
      format json
    </pattern>
    <pattern>
      format regexp
      expression /...your regexp pattern.../
    </pattern>
    <pattern>
      format none
    </pattern>
  </parse>
</filter>

For v0.12

Use format instead of <parse></parse>.

<filter app.**>
  @type parser
  key_name message

  format multi_format
  <pattern>
    format json
  </pattern>
  <pattern>
    format /...your regexp pattern.../
  </pattern>
  <pattern>
    format none
  </pattern>
</filter>

Adding format identity field

Sometimes it may be useful to know which pattern was used. Since pareser usage may not support retagging, there is an option to add a format name field and/or index field.

Example:

<filter app.**>
  @type parser
  key_name message
  <parse>
    @type multi_format

    # if set, add this key to record with value being pattern format name
    # (format_name key)
    format_key 'format'

    <pattern>
      format json
      # set format name for this pattern. If unset, uses format name
      # followed by index (in this case would be 'json#0')
      format_name 'json'
    </pattern>
    <pattern>
      format regexp
      format_name 'MyRefex'
      expression /...your regexp pattern.../
    </pattern>
    <pattern>
      format none
      format_name 'unparsed'
    </pattern>
  </parse>
</filter>

NOTE

This plugin doesn't work with multiline parsers because parser itself doesn't store previous lines.

Copyright

Author	Masahiro Nakagawa
Copyright	Copyright (c) 2014- Masahiro Nakagawa
License	Apache License (2.0)

fluent-plugin-multi-format-parser's Issues

Set pattern-specific tag

Is it possible to set a pattern-specific tag? I'd like to tag based on which pattern matched the message. The examples in the README show the tag being set at the top level, so maybe this can't be done, but it sure would be useful for what I'm trying to do.

Please confirm License of fluent-plugin-multi-format-parser

Please confirm if fluent-plugin-multi-format-parser is licensed under Apache as mentioned here : https://github.com/repeatedly/fluent-plugin-multi-format-parser/blob/master/fluent-plugin-multi-format-parser.gemspec

or MIT has mentioned in the README : https://github.com/repeatedly/fluent-plugin-multi-format-parser/blob/master/README.md

Can multi-format parse special json format？

Traefik log is like:

{"log":"{\"BackendAddr\":\"172.62.187.135:16686\",\"BackendName\":\"tracing-test.shannonai.com\",\"BackendURL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"172.62.187.135:16686\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"ClientAddr\":\"10.20.3.252:14432\",\"ClientHost\":\"10.20.11.156\",\"ClientPort\":\"14432\",\"ClientUsername\":\"-\",\"DownstreamContentSize\":530,\"DownstreamStatus\":200,\"DownstreamStatusLine\":\"200 OK\",\"Duration\":118040406,\"FrontendName\":\"tracing-test.shannonai.com\",\"OriginContentSize\":530,\"OriginDuration\":117940134,\"OriginStatus\":200,\"OriginStatusLine\":\"200 OK\",\"Overhead\":100272,\"RequestAddr\":\"tracing-test.shannonai.com\",\"RequestContentSize\":0,\"RequestCount\":174683,\"RequestHost\":\"tracing-test.shannonai.com\",\"RequestLine\":\"GET /api/traces?end=1576485011758000\\u0026limit=20\\u0026lookback=1h\\u0026maxDuration\\u0026minDuration\\u0026service=osprey\\u0026start=1576481411758000 HTTP/1.1\",\"RequestMethod\":\"GET\",\"RequestPath\":\"/api/traces?end=1576485011758000\\u0026limit=20\\u0026lookback=1h\\u0026maxDuration\\u0026minDuration\\u0026service=osprey\\u0026start=1576481411758000\",\"RequestPort\":\"-\",\"RequestProtocol\":\"HTTP/1.1\",\"RetryAttempts\":0,\"StartLocal\":\"2019-12-16T16:30:11.875918571+08:00\",\"level\":\"info\",\"msg\":\"\",\"time\":\"2019-12-16T16:30:11+08:00\"}\n","stream":"stdout","time":"2019-12-16T08:30:11.994250321Z"}

Notice that in "log"'s value, it have a subfield called "BackendURL". And it's value is also a json dictionary format.

I use below config, it cannot return expected result, but ignore the whole log

   <filter kubernetes.**>
      @id filter_parser
      @type parser
      key_name log
      reserve_data true
      remove_key_name_field true
      <parse>
        @type multi_format
        <pattern>
          format json
        </pattern>
        <pattern>
          format none
        </pattern>
      </parse>
    </filter>

But when I use another configuration(below), it can return a field called "message", which contain all the value of "log"
"json" and "none"‘s order is different than before

   <filter kubernetes.**>
      @id filter_parser
      @type parser
      key_name log
      reserve_data true
      remove_key_name_field true
      <parse>
        @type multi_format
        <pattern>
          format none
        </pattern>
        <pattern>
          format json
        </pattern>
      </parse>
    </filter>

result like:

"message": "{\"BackendAddr\":\"172.62.187.135:16686\",\"BackendName\":\"tracing-test.shannonai.com\",\"BackendURL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"172.62.187.135:16686\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"ClientAddr\":\"10.20.3.252:14432\",\"ClientHost\":\"10.20.11.156\",\"ClientPort\":\"14432\",\"ClientUsername\":\"-\",\"DownstreamContentSize\":530,\"DownstreamStatus\":200,\"DownstreamStatusLine\":\"200 OK\",\"Duration\":118040406,\"FrontendName\":\"tracing-test.shannonai.com\",\"OriginContentSize\":530,\"OriginDuration\":117940134,\"OriginStatus\":200,\"OriginStatusLine\":\"200 OK\",\"Overhead\":100272,\"RequestAddr\":\"tracing-test.shannonai.com\",\"RequestContentSize\":0,\"RequestCount\":174683,\"RequestHost\":\"tracing-test.shannonai.com\",\"RequestLine\":\"GET /api/traces?end=1576485011758000\\u0026limit=20\\u0026lookback=1h\\u0026maxDuration\\u0026minDuration\\u0026service=osprey\\u0026start=1576481411758000 HTTP/1.1\",\"RequestMethod\":\"GET\",\"RequestPath\":\"/api/traces?end=1576485011758000\\u0026limit=20\\u0026lookback=1h\\u0026maxDuration\\u0026minDuration\\u0026service=osprey\\u0026start=1576481411758000\",\"RequestPort\":\"-\",\"RequestProtocol\":\"HTTP/1.1\",\"RetryAttempts\":0,\"StartLocal\":\"2019-12-16T16:30:11.875918571+08:00\",\"level\":\"info\",\"msg\":\"\",\"time\":\"2019-12-16T16:30:11+08:00\"}\n"

My QUESTION :
Can json parse json in my certain log? Or what should I do to config a right parse operation?

Help me ! undefined method `parse'

When fluent-plugin-multi-format-parser is used, the next warning is issued. And it doesn't seem that plugin is used.

fluent.warn {"message":"parse failed undefined method `parse' for #Fluent::TextParser::MultiFormatParser:0x8db3f64"

How should it correspond?

About Environment

openSUSE 12.2 (i586)
-----RPMS-----------------------------------------------------------------------------------------------------
ruby19-1.9.3.p392-3.34.1.i586
ruby-common-2.1-3.1.noarch
ruby19-devel-1.9.3.p392-3.34.1.i586
ruby20-2.0.0.p247-14.2.i586
ruby-devel-2.0-109.1.i586
ruby-2.0-59.1.i586
ruby20-devel-2.0.0.p247-12.1.i586
-----GEMS-----------------------------------------------------------------------------------------------------
aws-sdk (1.59.1)
aws-sdk-v1 (1.59.1)
bigdecimal (1.2.5, 1.2.0)
bson (2.3.0, 1.11.1)
bson_ext (1.11.1)
bundler (1.7.9)
cool.io (1.2.4)
diff-lcs (1.2.5)
ffi (1.9.6)
ffi-yajl (1.3.1)
fluent-mixin-config-placeholders (0.3.0)
fluent-mixin-plaintextformatter (0.2.6)
fluent-plugin-config-expander (0.1.5)
fluent-plugin-ec2-metadata (0.0.4)
fluent-plugin-flume (0.1.1)
fluent-plugin-forest (0.3.0)
fluent-plugin-jsonbucket (0.0.2)
fluent-plugin-multi-format-parser (0.0.1)
fluent-plugin-parser (0.4.0)
fluent-plugin-record-reformer (0.4.0)
fluent-plugin-redshift (0.0.4)
fluent-plugin-rewrite-tag-filter (1.5.1)
fluent-plugin-s3 (0.5.1)
fluent-plugin-tail-ex (0.1.1)
fluent-plugin-tail-multiline (0.1.5)
fluent-plugin-td (0.10.23)
fluent-plugin-td-monitoring (0.2.0)
fluent-plugin-webhdfs (0.4.1)
fluentd (0.12.1)
http_parser.rb (0.6.0)
httpclient (2.4.0)
io-console (0.4.2)
iobuffer (1.1.2)
ipaddress (0.8.0)
json (1.8.1, 1.7.7)
libyajl2 (1.2.0)
ltsv (0.1.0)
mime-types (2.4.3, 1.25.1)
mini_portile (0.6.1)
minitest (5.5.0, 4.3.2)
mixlib-cli (1.5.0)
mixlib-config (2.1.0)
mixlib-log (1.6.0)
mixlib-shellout (2.0.0)
msgpack (0.5.9)
net-dhcp (1.3.2)
nokogiri (1.6.5)
ohai (8.0.1, 6.20.0)
parallel (1.3.3)
pg (0.17.1)
power_assert (0.2.2)
psych (2.0.8, 2.0.0)
rake (10.4.2, 0.9.6)
rdoc (4.2.0, 4.0.0)
rspec (3.1.0)
rspec-core (3.1.7)
rspec-expectations (3.1.2)
rspec-mocks (3.1.3)
rspec-support (3.1.2)
rubyzip (1.1.6)
sigdump (0.2.2)
string-scrub (0.0.5, 0.0.3)
systemu (2.6.4, 2.5.2)
td-client (0.8.67)
test-unit (3.0.8, 2.0.0.0)
thread_safe (0.3.4)
thrift (0.8.0)
tzinfo (1.2.2)
tzinfo-data (1.2014.10)
uuidtools (2.1.5)
webhdfs (0.6.0)
wmi-lite (1.0.0)

yajl-ruby (1.2.1)

Multiple regexes

My fluentd config has two regexes, but only the first one will match.

This is my .conf file.

<source>
  @type http
  port 8888
</source>

## live debugging agent
<source>
  @type debug_agent
  bind 127.0.0.1
  port 24230
</source>

<source>
	@type tail
	path /Users/pedro/.pm2/logs/fv-back-out*.log
	pos_file /Users/pedro/.pm2/logs/fv-back-response.log.pos
	tag fv-back
	<parse>
		@type multi_format
		<pattern>
			format regexp
			expression /(?<time>((\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})\.(\d{3})) (\-|\+)(\d{2}):(\d{2})), \[(?<tags>(.)*response(.)*)\] data: (?<method>[^ ]+) (?<endpoint>[^ ]+) (?<query>[^ ]+) (?<payload>[^ ]+) (?<statusCode>[^ ]+) \((?<responseTime>[^ ]+)\) (?<credentials>[^ ]+)/
			time_format %Y-%m-%d %H:%M:%S.%L
			types time:time,tags:array,statusCode:integer
		</pattern>
		<pattern>
			format regexp
			expression /(?<time>((\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})\.(\d{3})) (\-|\+)(\d{2}):(\d{2})), \[(?<tags>((.)*error(.)*))\] data: \[(?<statusCode>.*)\] (?<type>.*): (?<message>.*)/
			time_format %Y-%m-%d %H-%M-%S.%L
                	types time:time,tags:array,statusCode:integer
		</pattern>
	</parse>
</source>

<match fv-back*>
	@type copy

	<store>
		@type elasticsearch
		host localhost
		port 9200
		logstash_format true
	</store>

	<store>
		@type s3
		aws_key_id AKIAI3BKNP4PWLJW5PMQ
		aws_sec_key hoWhK+KzAKOwKuPszBkS8099Nuu8WZKJKo516XUY
		s3_bucket softruck.pm2.logs
		s3_region us-east-1
		path /

		<buffer time>
			@type file
			path /var/log/td-agent/s3_buffer
			timekey 3600
			timekey wait 10m
			timekey_use_utc true
			chunk_limit_size 256m
		</buffer>
	</store>
</match>

<match **>
	@type null
</match>

And this 2018-08-08 16:45:05 -0300 [warn]: #0 pattern not match: "2018-08-08 16:45:04.869 -03:00, [log,api,error] data: [500] SequelizeDatabaseError: relation \"daily_courses\" does not exist" is my Fluentd log.

Rubular says my regex should pass. Is there anything I'm doing wrong?

Support for format parameters inside <pattern>?

Does this plugin support format parameters, such as inject_key_prefix for format json? It does not appear to work:

fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: starting fluentd-1.4.2 pid=8 ruby="2.5.5"
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-v", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: gem 'fluent-plugin-docker_metadata_tb_filter' version '0.3.2'
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: gem 'fluent-plugin-elasticsearch' version '3.2.1'
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: gem 'fluent-plugin-record-modifier' version '2.0.1'
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: gem 'fluentd' version '1.4.2'
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: adding filter pattern="docker.**" type="parser"
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: adding match pattern="docker.**" type="stdout"
fluentd-tag | 2019-05-15 21:51:08 +0000 [info]: fluent/log.rb:322:info: adding source type="forward"
fluentd-tag | 2019-05-15 21:51:08 +0000 [warn]: fluent/log.rb:342:warn: parameter 'inject_key_prefix' in <pattern>
fluentd-tag |   format json
fluentd-tag |   inject_key_prefix jsonlog.
fluentd-tag |   reserve_data true
fluentd-tag |   reserve_time true
fluentd-tag | </pattern> is not used.
fluentd-tag | 2019-05-15 21:51:08 +0000 [warn]: fluent/log.rb:342:warn: parameter 'reserve_data' in <pattern>
fluentd-tag |   format json
fluentd-tag |   inject_key_prefix jsonlog.
fluentd-tag |   reserve_data true
fluentd-tag |   reserve_time true
fluentd-tag | </pattern> is not used.
fluentd-tag | 2019-05-15 21:51:08 +0000 [warn]: fluent/log.rb:342:warn: parameter 'reserve_time' in <pattern>
fluentd-tag |   format json
fluentd-tag |   inject_key_prefix jsonlog.
fluentd-tag |   reserve_data true
fluentd-tag |   reserve_time true
fluentd-tag | </pattern> is not used.

Minimum config file to demonstrate the error:

<system>
@log_level debug
</system>

<source>
  @type forward
  port 24224
</source>

<filter docker.**>
  @type parser
  key_name log
  <parse>
    @type multi_format
    <pattern>
      format json
      inject_key_prefix jsonlog.
      reserve_data true
      reserve_time true
    </pattern>
    <pattern>
      format none
    </pattern>
  </parse>
</filter>

<match docker.**>
  @type stdout
</match>

Actual record overridden after parsing

In the fluentd input, I am using a parser in the syslog input plugin to break the messages using regular expression. After the separation, I am trying to parse one of the field again using the multi-format-parser.

Output after input plugin - {"field1":"abc", "field2":"xyz", "message":"{"field3":"def", "field4":"ghi"}"}
Output after multi-format-parser - {"field3":"def", "field4":"ghi"}

Is there a way to retain the actual fields also.

fluentd-1.14.1/lib/fluent/plugin.rb:139:in `new_parser': undefined method `[]' for nil:NilClass (NoMethodError)

Fluent.Conf

<source>
      @type tail
      @label @tcpstream
      tag proxy.log
      path /fluentd/log/proxy-service-*.log
      pos_file /fluentd/log/fluentd_posfile.log.pos
      pos_file_compaction_interval 72h
    <parse>
      @type multi_format
      <pattern>
        expression /^\[(?<logtime>[^\]]*)\] (?<level>[^ ]*) (?<pod_name>[^ ]*) (?<app_name>\d*) (?<custom_dimensions>\d*) $/
        time_key logtime
        time_format %Y-%m-%d %H:%M:%S %z
      </pattern>
    </parse>
    </source>

ERROR:

/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/plugin.rb:139:in `new_parser': undefined method `[]' for nil:NilClass (NoMethodError)
from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-multi-format-parser-1.0.0/lib/fluent/plugin/parser_multi_format.rb:21:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-multi-format-parser-1.0.0/lib/fluent/plugin/parser_multi_format.rb:17:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-multi-format-parser-1.0.0/lib/fluent/plugin/parser_multi_format.rb:17:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/plugin.rb:187:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/plugin_helper/parser.rb:90:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/plugin_helper/parser.rb:85:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/plugin_helper/parser.rb:85:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/plugin/in_tail.rb:136:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/plugin.rb:187:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/root_agent.rb:320:in `add_source'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/root_agent.rb:161:in `block in configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/root_agent.rb:155:in `each'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/root_agent.rb:155:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/engine.rb:105:in `configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/engine.rb:80:in `run_configure'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/supervisor.rb:665:in `run_supervisor'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/lib/fluent/command/fluentd.rb:352:in `'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.1/bin/fluentd:15:in `

section <pattern> is not used

I'm using Fluent 0.14.9 from gem source and trying to use multi_format to allow message that don't parse to fall back to being stored unformatted, but no config appears to work.

Although Fluent starts correctly, it doesn't load the patterns in format_multi. I get this output:

Dec 07 11:51:57 consul-server-a fluentd[2008]: 2016-12-07 11:51:57 +0000 [warn]: section <pattern> is not used in <filter consul.log> of parser plugin Dec 07 11:51:57 consul-server-a fluentd[2008]: 2016-12-07 11:51:57 +0000 [warn]: section <pattern> is not used in <filter consul.log> of parser plugin

Is this to do with formatting changes in newer versions of Fluent? I've tried a few different ways of writing the config but nothing has worked.

This is my config:

<source>
  @type systemd
  filters [{ "_SYSTEMD_UNIT": "consul.service" }]
  tag consul.log
  path /run/log/journal
</source>

<filter consul.log>
  format multi_format
  <pattern>
    format /^(?<time>[0-9]{4}\/[0-9]{2}\/[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}) \[(?<severity>[A-Z]*)\] (?<source>[a-z]*): (?<message>[[
^$]*)?$/
  </pattern>
  <pattern>
    format none
  </pattern>
  key_name MESSAGE
  @type parser
</filter>

<match consul.*>
  @type kafka
  brokers 172.17.0.3:9092
  default_topic logstash
  output_include_tag true
  output_include_time true
</match>

Error while installing fluent-plugin-multi-format-parser plugin on MapR Cluster 5.2

I have followed GitHub page to install the plugin and got following error:

./fluent-gem install fluent-plugin-multi-format-parser

Successfully installed fluent-plugin-multi-format-parser-0.1.1
/opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rdoc/rdoc.rb:572:in require': cannot load such file -- rdoc/generator/darkfish (LoadError) from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rdoc/rdoc.rb:572:in <top (required)>'
from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rdoc/rubygems_hook.rb:74:in require' from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rdoc/rubygems_hook.rb:74:in load_rdoc'
from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rdoc/rubygems_hook.rb:245:in setup' from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rdoc/rubygems_hook.rb:152:in generate'
from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rdoc/rubygems_hook.rb:57:in block in generation_hook' from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rdoc/rubygems_hook.rb:56:in each'
from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rdoc/rubygems_hook.rb:56:in generation_hook' from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rubygems/request_set.rb:186:in block in install'
from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rubygems/request_set.rb:185:in each' from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rubygems/request_set.rb:185:in install'
from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rubygems/commands/install_command.rb:250:in install_gem' from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rubygems/commands/install_command.rb:300:in block in install_gems'
from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rubygems/commands/install_command.rb:296:in each' from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rubygems/commands/install_command.rb:296:in install_gems'
from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rubygems/commands/install_command.rb:203:in execute' from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rubygems/command.rb:308:in invoke_with_build_args'
from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rubygems/command_manager.rb:169:in process_args' from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rubygems/command_manager.rb:139:in run'
from /opt/mapr/fluentd/fluentd-0.14.00/lib/fluentd-0.14.00-linux-x86_64/lib/ruby/lib/ruby/2.3.0/rubygems/gem_runner.rb:55:in run' from /opt/mapr/fluentd/fluentd-0.14.00/bin/../lib/fluentd-0.14.00-linux-x86_64/lib/app/bin/fluent-gem:6:in

'

Initially, it says "Successfully installed" but then throws the bunch of errors.

Please help

how to use multiparser using multiformat plugin

Can you please tell me how to use parse the tomcat logs(debus,error which is a multiline) and access logs from the stdout..
It will be helpfull if you share the logic

Store parsed inline json in same/new field as an json object

I have a json
{
"name":"something",
"details":"{"appName":"my-app","message":"new message"}"
}

When I parsed this data, inline json fields gets added at parent level
{
"name":"something",
"appName":"my-app",
"message":"new message"
}

Is it possible to store the parsed data within same field as an json object??
{
"name":"something",
"details":{
"appName":"my-app",
"message":"new message"
}
}

Possible incompatibility with grok parser's grok XML elements

Hi,

Thanks for the great plugin!

We may be attempting to configure it improperly esp. around the v0.14 changes, but are at a loss as to how to use <grok> elements from the grok plugin with this plugin?

For reference, the basic grok_pattern works well with separate individual <pattern>format grok</pattern>s.

Fluent Version: 0.14.14

Given the following config block:

<filter **>
  @type parser
  key_name log
  reserve_data false
  <parse>
    @type multi_format
    <pattern>
        format grok
        <grok>
          # e.g. 2017-04-10 08:07:49.062 DEBUG [doot-boop,d0ea3905d59025c2,d0ea3905d59025c2,false] 17206 --- [  XNIO-2 task-5] io.undertow.session                      : Created session ... snip ...
          pattern %{TIMESTAMP_ISO8601:time}%{SPACE}%{LOGLEVEL:level}%{SPACE}\[%{DATA:app_name},%{DATA:trace_id},%{DATA:span_id},%{DATA:export_to_zipkin}\]%{SPACE}%{NUMBER:pid}%{SPACE}---%{SPACE}\[%{SPACE}%{DATA:threadname}\]%{SPACE}%{JAVACLASS:classname}%{SPACE}:%{SPACE}%{GREEDYDATA:logmessage}
        </grok>
        <grok>
          # e.g. 2017-03-27 22:15:59.861 DEBUG 5 --- [t-dispatcher-37] com.acme.aspects.LogAspect              : snip\n
          pattern %{GREEDYDATA:logmessage}
        </grok>
    </pattern>
    <pattern>
        format grok
    </pattern>
  </parse>
</filter>

When starting fluent we get:

2017-04-11 22:22:13 +0000 [info]: adding filter pattern="**" type="parser"
2017-04-11 22:22:13 +0000 [info]: #0 Expanded the pattern  into (?<time>(?>\d\d){1,2}-(?:0?[1-9]|1[0-2])-(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])[T ](?:2[0123]|[01]?[0-9]):?(?:[0-5][0-9])(?::?(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))?(?:Z|[+-](?:2[0123]|[01]?[0-9])(?::?(?:[0-5][0-9])))?)\s*(?<level>([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?))\s*\[(?<app_name>.*?),(?<trace_id>.*?),(?<span_id>.*?),(?<export_to_zipkin>.*?)\]\s*(?<pid>(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))))\s*---\s*\[\s*(?<threadname>.*?)\]\s*(?<classname>(?:[a-zA-Z0-9-]+\.)+[A-Za-z0-9$]+)\s*:\s*(?<logmessage>.*)
2017-04-11 22:22:13 +0000 [info]: #0 Expanded the pattern  into (?<logmessage>.*)
2017-04-11 22:22:13 +0000 [error]: #0 config error file="/etc/fluent/fluent.conf" error_class=Fluent::ConfigError error="no grok patterns. Check configuration, e.g. typo, configuration syntax, etc"

The requirements are met but a dependency error occurs.

The requirements are met but a dependency error occurs.

-My env are as the following.
Ruby: 2.0.0
Fluentd: 0.12.0

-I expected "ruby version 2.0.0" to be ok when i use a "fluent-plugin-multi-format-parser 0.1.1". But i got a dependency error to require "ruby version >= 2.1.0".

# ruby --version
ruby 2.0.0p648 (2015-12-16) [x86_64-linux]
 
# fluentd --version
fluentd 0.12.43

# fluent-gem install fluent-plugin-multi-format-parser -v 0.1.1
Fetching: fluent-plugin-multi-format-parser-0.1.1.gem (100%)
Successfully installed fluent-plugin-multi-format-parser-0.1.1
ERROR:  Error installing fluent-plugin-multi-format-parser:
        serverengine requires Ruby version >= 2.1.0.

use multiline parser

I am trying to use multiline parser within this plugin and It seems that doesnt work

Ive tested it also with Java stack trace example from documentation...

My configuration:

<source>
  @type tcp
  tag "sometag.#{Socket.gethostname}"
  port 1234
  <parse>
    @type multi_format
    <pattern>
      format regexp
      expression /\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<message>.*)$/
      time_format %b %d %H:%M:%S
    </pattern>
    <pattern>
      format regexp
      expression /\<(?<pri>[0-9]{1,3})\>(?<time>[^ ]+) (?<host>[!-~]{1,255}) (?<message>.*)$/
      time_format %Y-%m-%dT%H:%M:%S.%L%z
    </pattern>
    <pattern>
      format regexp
      expression /\<(?<pri>[0-9]{1,3})\>(?<time>[^ ]+) (?<host>[!-~]{1,255}) (?<message>.*)$/
      time_format %Y-%m-%dT%H:%M:%S%z
    </pattern>
    <pattern>
      format multiline
      format_firstline /\d{4}-\d{1,2}-\d{1,2}/
      format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
    </pattern>
  </parse>
</source>

convert_line_to_event: pattern not match

Hi! I have some trouble

2016-12-30 16:46:16 +0500 [warn]: plugin/in_tail.rb:318:block in convert_line_to_event: pattern not match: "GET hello world"
2016-12-30 16:46:16 +0500 [warn]: plugin/in_tail.rb:318:block in convert_line_to_event: pattern not match: "complited"

config

2016-12-30 16:46:14 +0500 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2016-12-30 16:46:14 +0500 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-12-30 16:46:14 +0500 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-kafka' version '0.3.1'
2016-12-30 16:46:14 +0500 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-mongo' version '0.7.15'
2016-12-30 16:46:14 +0500 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-multi-format-parser' version '0.0.2'
2016-12-30 16:46:14 +0500 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2016-12-30 16:46:14 +0500 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-s3' version '0.7.1'
2016-12-30 16:46:14 +0500 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-scribe' version '0.10.14'
2016-12-30 16:46:14 +0500 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-td' version '0.10.29'
2016-12-30 16:46:14 +0500 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-12-30 16:46:14 +0500 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-webhdfs' version '0.4.2'
2016-12-30 16:46:14 +0500 [info]: fluent/engine.rb:126:block in configure: gem 'fluentd' version '0.12.29'
2016-12-30 16:46:14 +0500 [info]: fluent/agent.rb:129:add_match: adding match pattern="**" type="forward"
2016-12-30 16:46:14 +0500 [info]: plugin/out_forward.rb:148:block in configure: adding forwarding server 'fluentd' host="192.168.99.245" port=24224 weight=60 plugin_id="object:3fd183b85ddc"
2016-12-30 16:46:14 +0500 [info]: fluent/root_agent.rb:147:add_source: adding source type="tail"
2016-12-30 16:46:14 +0500 [info]: fluent/engine.rb:133:configure: using configuration file: <ROOT>
  <source>
    @type tail
    format multi_format
    path /tmp/test.log
    pos_file /var/log/td-agent/fpm.error.pos
    tag fpm-error.*
    <pattern>
      format multiline
      format_firstline /^GET/
      format1 /GET (?<message>.*)/
      format2 /complited/
    </pattern>
  </source>

and simple log lines

GET hello world
complited

my env

3.13.0-77-generic
Description: Ubuntu 14.04.3 LTS

Use time_format and types on multi_format parse

I currently parse one tag with a simple filter but I need to add a different pattern to the filter and I'm planning to migrate to a multi_format plugin.

My question is if it's possible to use the time_format and types fields on each the pattern like this:

<filter kubernetes.var.log.containers.traefik-ingress-**.log>
  @type parser
  key_name log
  reserve_data yes
  <parse>
    @type multi_format
    <pattern>
      format regexp
      expression /^(?<ip>[^-]*) - - \[(?<datetime>[^\]]*)\] "(?<method>[^ ]*) (?<path>[^ ]*) (?<http_version>[^"]*)" (?<status_code>[^ ]*) (?<body_bytes>[^ ]*) "(?<referer>[^"]*)" "(?<user_agent>[^"]*)" (?<seq>[^ ]*) "(?<domain>[^ ]*)" "(?<dest_url>[^ ]*)" (?<response_time>[^ ms]*)/
      time_key datetime
      time_format %d/%b/%Y:%H:%M:%S %z
      types status_code:integer,body_bytes:integer,seq:integer,response_time:integer
    </pattern>
    <pattern>
      format regexp
      expression /^\[(?<datetime>[^\]]*)\] - (?<data>[^ ]*)/
    </pattern>
  </parse>
</filter>

Not working under Ubuntu xenial, with td-agent 3.2v

Hello,
It seems like this plugin is not working on Unbuntu Xenial.

I installed this version :

http://packages.treasuredata.com.s3.amazonaws.com/3/ubuntu/xenial/pool/contrib/t/td-agent/td-agent_3.2.0-0_amd64.deb

td-agent.log

2021-02-18 18:38:07 +0900 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf" 2021-02-18 18:38:07 +0900 [info]: brokers has been set: ["10.232.0.134:9092", "10.232.0.94:9092", "10.232.0.78:9092"] 2021-02-18 18:38:07 +0900 [info]: brokers has been set: ["10.232.0.134:9092", "10.232.0.94:9092", "10.232.0.78:9092"] 2021-02-18 18:38:07 +0900 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Unknown parser plugin 'multi_format'. Run 'gem search -rd fluent-plugin' to find plugins" 2021-02-18 18:38:07 +0900 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf" 2021-02-18 18:38:07 +0900 [info]: brokers has been set: ["10.232.0.134:9092", "10.232.0.94:9092", "10.232.0.78:9092"] 2021-02-18 18:38:07 +0900 [info]: brokers has been set: ["10.232.0.134:9092", "10.232.0.94:9092", "10.232.0.78:9092"] 2021-02-18 18:38:07 +0900 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Unknown parser plugin 'multi_format'. Run 'gem search -rd fluent-plugin' to find plugins" 2021-02-18 18:38:08 +0900 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf" 2021-02-18 18:38:08 +0900 [info]: brokers has been set: ["10.232.0.134:9092", "10.232.0.94:9092", "10.232.0.78:9092"] 2021-02-18 18:38:08 +0900 [info]: brokers has been set: ["10.232.0.134:9092", "10.232.0.94:9092", "10.232.0.78:9092"] 2021-02-18 18:38:08 +0900 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Unknown parser plugin 'multi_format'. Run 'gem search -rd fluent-plugin' to find plugins" 2021-02-18 18:38:08 +0900 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf" 2021-02-18 18:38:08 +0900 [info]: brokers has been set: ["10.232.0.134:9092", "10.232.0.94:9092", "10.232.0.78:9092"] 2021-02-18 18:38:08 +0900 [info]: brokers has been set: ["10.232.0.134:9092", "10.232.0.94:9092", "10.232.0.78:9092"] 2021-02-18 18:38:08 +0900 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Unknown parser plugin 'multi_format'. Run 'gem search -rd fluent-plugin' to find plugins" 2021-02-18 18:38:09 +0900 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf" 2021-02-18 18:38:09 +0900 [info]: brokers has been set: ["10.232.0.134:9092", "10.232.0.94:9092", "10.232.0.78:9092"] 2021-02-18 18:38:09 +0900 [info]: brokers has been set: ["10.232.0.134:9092", "10.232.0.94:9092", "10.232.0.78:9092"] 2021-02-18 18:38:09 +0900 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Unknown parser plugin 'multi_format'. Run 'gem search -rd fluent-plugin' to find plugins"

td-agent-gem list|grep fluent

fluent-plugin-elasticsearch (2.10.3, 1.17.1) fluent-plugin-kafka (0.16.0, 0.7.3) fluent-plugin-multi-format-parser (1.0.0) fluent-plugin-record-modifier (1.1.0) fluent-plugin-rewrite-tag-filter (2.1.0) fluent-plugin-s3 (1.1.3) fluent-plugin-td (1.0.0) fluent-plugin-td-monitoring (0.2.4) fluent-plugin-webhdfs (1.2.3)

lsb_release -a

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.7 LTS Release: 16.04 Codename: xenial

Is there any solution for this ?

Thanks.

Is it possible to retag inside a pattern?

For example like this:

@type multi_format
  <pattern>
    format json
    tag json
  </pattern>

add diff value of key-type key<new key> for each pattern matched.

In multi_format, add diff value of keytype key for each pattern matched.

<parse>
     @type multi_format
       <pattern>
           format /.*reg1.*/
          #keytype type1
       </pattern>
       <pattern>
           format /.*reg2.*/
          #keytype type2
       </pattern>
       <pattern>
           format json
           #keytype type3
       </pattern>
   </parse>

My purpose is use this for later filter/ or search/ ...