The logstash-deduplication's intro from imweijh

logstash-deduplication's Introduction

Link do wpisu: https://wiadrodanych.pl/big-data/deduplikacja-zdarzen-logstash-redis/

本来想借鉴此方法，后面发现用logstash内置filter：fingerprint + memcached，可以实现去重功能


  ### fingerprint 和 memcached 实现去重(dedupe)功能begin
  fingerprint {
    source => [ "cjdate", "gddm", "zqdm", "wtid", "cjid" ]
    target => "[fingerprint]"
    concatenate_sources => true
    method => "SHA1"
  }

  memcached {
      id => "memcached-getfp"
      hosts => ["127.0.0.1:11211"]
      namespace => "dedupe"
      get => {"%{fingerprint}" => "[cjexists]"}
  }

  if [cjexists] {
    mutate { add_tag => [ "dedupe" ] }
    # memcached 根据 fingerprint 如果查回来cjexists字段，表明为重复记录，加tag，也可以直接drop{}
  } else {
    memcached {
      hosts => ["127.0.0.1:11211"]
      namespace => "dedupe"
      ttl => 28800                          # ttl一定要配否则内存爆，28800秒=8小时
      set => { "gddm" => "%{fingerprint}" } # 存什么字段不重要，这里是存的股东代码
      id => "memcached-setfp"
   }
  }
  ### fingerprint 和 memcached 实现去重(dedupe)功能end

Recommend Projects