y-ken/fluent-plugin-anonymizer

Fluentd filter output plugin to anonymize records with MD5/SHA1/SHA256/SHA384/SHA512 algorithms. This data masking plugin protects privacy data such as ID, email, phone number, IPv4/IPv6 address and so on.

fluent-plugin-anonymizer's Issues

Can't anonymize fields containing <ip-addr>:<port>

I'm having multiple logs entries with ipPort field with values like this: 192.168.0.1:8080. Trying to use anonymizer with the following config does not produce desired result:

<source>
  @type dummy
  tag raw.dummy
  dummy [ {"ipPort":"10.102.3.80:5555","member_id":"12345", "mail":"[email protected]"} ]
</source>

<filter raw.**>
  @type anonymizer

  # Specify rounding address keys with comma and subnet mask
  <mask network>
    key_pattern .*
    ipv4_mask_bits  0
    ipv6_mask_bits  0
  </mask>
</filter>

<match raw.**>
  @type stdout
</match>

Gives:

2020-06-22 17:17:46.099515554 +0000 raw.dummy: {"ipPort":"10.102.3.80:5555","member_id":"12345","mail":"[email protected]"}

Probably plugin does not recognize value 10.102.3.80:5555 as an IP address, hence skips it. Is there any workaround to handle such cases?

Implementing a filter plugin support

This plugin is a great example of a useful output plugin that should really be a filter plugin. I suggest that

1. the anonymization functionality is factored out into a re-usable library used both by the existing out_anonymizer.rb and filter_anonymizer.rb
2. Implement the filter plugin version (filter_anonymizer)

Support for masking a part of string within a value

I am using the value_pattern to anonymize the value of the msg key from log. It is observed that the fluent-plugin-anonymizer hashes the entire string which ‘has’ the matching pattern.

Can you add support for hashing only the part of matching string and not the entire value?

Sample Config:
<filter **>
  @type anonymizer
  <mask sha1>
     value_pattern UserId:\d{3}-?\d{3}
  </mask>
 </filter>

Log Sample:
{ "level": "INFO", "timezone": "UTC", "threadname": "main", "msg": "Logged in UserId:123-456"}

Output:
{ "level": "INFO", "timezone": "UTC", "threadname": "main", "msg": "<hashed value>"}

Expected Output:
{"level": "INFO", "timezone": "UTC", "threadname": "main", "msg": "Logged in <hashed value>"}

installation of the gem fails. plugin can't be found

gem install fluent-plugin-anonymizer

ERROR: Could not find a valid gem 'fluent-plugin-anonymizer' (>= 0), here is why:
Unable to download data from https://rubygems.org/ - no such name (https://rubygems.org/specs.4.8.gz)

Any suggestions?
Thank you!
-D.

Question, Sending both filtered and unfiltered data to two different elastic search hosts

I am trying to use this plugin to anonymize some records, using this plugin, the only caveat being I want to send the anonymized data to one es host and the unanonymized data to a different host, anyone has ideas as to how this can be accomplished

Anonymizer does not handle multi-level keys.

I have a complex log entry that is tailed from a file in the JSON format such as:

 { "eventType": "logEvent", "context": { "email": "[email protected]" } }

When I try to anonymize the email field with the following config, the email field is not anonymized:

<match anonymize_event.**>
    type anonymizer
    remove_tag_prefix anonymize_event.
    sha1_keys context.email
    hash_salt salty_awesomeness
  </match>

Am I configuring the plugin incorrectly?

Is <mask ARGUMENT> style easy to mistake?

Copied from #25

@y-ken said:

---

I suggest <mask> section won't have ARGUMENT like <mask ARGUMENT>.
Because It does not often used section which have argument as like of other plugins.
In other point of view, It can be mistake to masking network key when using<mask network> style.
How do you think about this?

I suggest this style. It is hard to mistake setting mask type.

  <mask>
    type sha1
    keys user_id, member_id, mail
    # Set hash salt with any strings for more security
    salt mysaltstring
  </mask>

Current sample is below.

  <mask sha1>
    keys user_id, member_id, mail
    # Set hash salt with any strings for more security
    salt mysaltstring
  </mask>

---

@y-ken
In v0.14/v1, buffer configuration is also using argument to specify chunk_keys:
https://docs.fluentd.org/v1.0/articles/buffer-section#chunk-keys
I think that this arguments feature is not well known feature, though...

Support label feature

This plugin seems to be using Fluent::Engine.emit.
It should be using router.emit instead.

I'll start to work this a bit of improvement after #8 is merged.

Drop out_anonymizer

Because out_anonymizer is not migrated to v1 API and uses some deprecated APIs. (using compat layer)
Therefore we cannot keep maintaining out_anonymizer anymore.

How do you think about this? @y-ken

Improve filter_anonymizer configurations

Current configuration is redundant and very complicated.
We can simplify parameters using record_accessor plugin helper.

My plan is followings:

• organize configurations
  • remove mask/{key,key_chain,key_chanis}
  • use record_accessor plugin helper
  • add :deprecated option or :obsoleted options to obsolete configuration parameters
  • refactor code
  • update README.md

If you are OK, I will create patches for this issue.

endless loop caused by HandleTagNameMixin miss usage

On using HandleTagNameMixin for multiple filter chain, we must use tag.dup.

ref. kentaro/fluent-plugin-extract_query_params#1

sample config

<match app.postfix.sent>
  type anonymizer
  sha1_keys         mail
  remove_tag_prefix app.postfix.
  add_tag_prefix    mail.
</match>

sample log

2013-12-03 18:35:09 +0900 [warn]: no patterns matched tag="mail.mail.sent"
2013-12-03 18:35:09 +0900 [warn]: no patterns matched tag="mail.mail.mail.sent"
2013-12-03 18:35:09 +0900 [warn]: no patterns matched tag="mail.mail.mail.mail.sent"
2013-12-03 18:35:09 +0900 [warn]: no patterns matched tag="mail.mail.mail.mail.mail.sent"
2013-12-03 18:35:09 +0900 [warn]: no patterns matched tag="mail.mail.mail.mail.mail.mail.sent"
2013-12-03 18:35:09 +0900 [warn]: no patterns matched tag="mail.mail.mail.mail.mail.mail.mail.sent"