Giter Site home page Giter Site logo

sergelogvinov / fluent-plugin-rewrite-tag-filter Goto Github PK

View Code? Open in Web Editor NEW

This project forked from fluent/fluent-plugin-rewrite-tag-filter

0.0 1.0 0.0 443 KB

Fluentd Output filter plugin to rewrite tags that matches specified attribute.

Home Page: http://rubygems.org/gems/fluent-plugin-rewrite-tag-filter

License: Other

Ruby 100.00%

fluent-plugin-rewrite-tag-filter's Introduction

fluent-plugin-rewrite-tag-filter Build Status

Overview

Rewrite Tag Filter for Fluentd. It is designed to rewrite tags like mod_rewrite.
Re-emit the record with rewrited tag when a value matches/unmatches with a regular expression.
Also you can change a tag from Apache log by domain, status code (ex. 500 error),
user-agent, request-uri, regex-backreference and so on with regular expression.

Installation

Install with gem, fluent-gem or td-agent-gem command as:

# for system installed fluentd
$ gem install fluent-plugin-rewrite-tag-filter

# for td-agent (Legacy)
$ sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-rewrite-tag-filter -v 1.5.6

# for td-agent2 (with fluentd v0.12)
$ sudo td-agent-gem install fluent-plugin-rewrite-tag-filter -v 1.5.6

Configuration

Syntax

rewriterule<num> <attribute> <regex_pattern> <new_tag>

# Optional: Capitalize letter for every matched regex backreference. (ex: maps -> Maps)
# for more details, see usage.
capitalize_regex_backreference <yes/no> (default no)

# Optional: remove tag prefix for tag placeholder. (see the section of "Tag placeholder")
remove_tag_prefix <string>

# Optional: override hostname command for placeholder. (see the section of "Tag placeholder")
hostname_command <string>

# Optional: Set log level for this plugin. (ex: trace, debug, info, warn, error, fatal)
log_level        <string> (default info)

Usage

It's a sample to exclude some static file log before split tag by domain.

<source>
  @type tail
  path /var/log/httpd/access_log
  format apache2
  time_format %d/%b/%Y:%H:%M:%S %z
  tag td.apache.access
  pos_file /var/log/td-agent/apache_access.pos
</source>

# "capitalize_regex_backreference yes" affects converting every matched first letter of backreference to upper case. ex: maps -> Maps
# At rewriterule2, redirect to tag named "clear" which unmatched for status code 200.
# At rewriterule3, redirect to tag named "clear" which is not end with ".com"
# At rewriterule6, "site.$2$1" to be "site.ExampleMail" by capitalize_regex_backreference option.
<match td.apache.access>
  @type rewrite_tag_filter
  capitalize_regex_backreference yes
  rewriterule1 path   \.(gif|jpe?g|png|pdf|zip)$  clear
  rewriterule2 status !^200$                      clear
  rewriterule3 domain !^.+\.com$                  clear
  rewriterule4 domain ^maps\.example\.com$        site.ExampleMaps
  rewriterule5 domain ^news\.example\.com$        site.ExampleNews
  rewriterule6 domain ^(mail)\.(example)\.com$    site.$2$1
  rewriterule7 domain .+                          site.unmatched
</match>

<match site.*>
  @type mongo
  host localhost
  database apache_access
  remove_tag_prefix site
  tag_mapped
  capped
  capped_size 100m
</match>

<match clear>
  @type null
</match>

Result

$ mongo
MongoDB shell version: 2.2.0
> use apache_access
switched to db apache_access
> show collections
ExampleMaps
ExampleNews
ExampleMail
unmatched

Debug

On starting td-agent, Logging supported like below.

$ tailf /var/log/td-agent/td-agent.log
2012-09-16 18:10:51 +0900: adding match pattern="td.apache.access" type="rewrite_tag_filter"
2012-09-16 18:10:51 +0900: adding rewrite_tag_filter rule: [1, "path", /\.(gif|jpe?g|png|pdf|zip)$/, "clear"]
2012-09-16 18:10:51 +0900: adding rewrite_tag_filter rule: [2, "domain", /^maps\.example\.com$/, "site.ExampleMaps"]
2012-09-16 18:10:51 +0900: adding rewrite_tag_filter rule: [3, "domain", /^news\.example\.com$/, "site.ExampleNews"]
2012-09-16 18:10:51 +0900: adding rewrite_tag_filter rule: [4, "domain", /^(mail)\.(example)\.com$/, "site.$2$1"]
2012-09-16 18:10:51 +0900: adding rewrite_tag_filter rule: [5, "domain", /.+/, "site.unmatched"]

Tag placeholder

It is supported these placeholder for new_tag (rewrited tag).

  • ${tag}
  • __TAG__
  • ${tag_parts[n]}
  • __TAG_PARTS[n]__
  • ${hostname}
  • __HOSTNAME__

The placeholder of ${tag_parts[n]} and __TAG_PARTS[n]__ acts accessing the index which split the tag with "." (dot).
For example with td.apache.access tag, it will get td by ${tag_parts[0]} and apache by ${tag_parts[1]}.

Note Currently, range expression ${tag_parts[0..2]} is not supported.

Placeholder Option

  • remove_tag_prefix

This option adds removing tag prefix for ${tag} or __TAG__ in placeholder.

  • hostname_command

By default, execute command as hostname to get full hostname.
On your needs, it could override hostname command using hostname_command option.
It comes short hostname with hostname_command hostname -s configuration specified.

Placeholder Usage

It's a sample to rewrite a tag with placeholder.

# It will get "rewrited.access.ExampleMail"
<match apache.access>
  @type rewrite_tag_filter
  rewriterule1  domain  ^(mail)\.(example)\.com$  rewrited.${tag}.$2$1
  remove_tag_prefix apache
</match>

# It will get "rewrited.ExampleMail.app30-124.foo.com" when hostname is "app30-124.foo.com"
<match apache.access>
  @type rewrite_tag_filter
  rewriterule1  domain  ^(mail)\.(example)\.com$  rewrited.$2$1.${hostname}
</match>

# It will get "rewrited.ExampleMail.app30-124" when hostname is "app30-124.foo.com"
<match apache.access>
  @type rewrite_tag_filter
  rewriterule1  domain  ^(mail)\.(example)\.com$  rewrited.$2$1.${hostname}
  hostname_command hostname -s
</match>

# It will get "rewrited.game.pool"
<match app.game.pool.activity>
  @type rewrite_tag_filter
  rewriterule1  domain  ^.+$  rewrited.${tag_parts[1]}.${tag_parts[2]}
</match>

Example

Related Articles

TODO

Pull requests are very welcome!!

Copyright

Copyright : Copyright (c) 2012- Kentaro Yoshida (@yoshi_ken)
License : Apache License, Version 2.0

fluent-plugin-rewrite-tag-filter's People

Watchers

avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.