ingafeick/logstash-filter-denormalize

A logstash filter to split an event into many. Like the "split filter" but it splits on objects, not strings.

Ohai 👋

• 🎓 Software developer / SRE / Observability Engineer
• 💪🏻 I'm currently working on: ...
• 🌱 I'm currently learning: Haskell and Rust
• 📫 How to reach me: LinkedIn

logstash-filter-denormalize's Issues

Tag for successful denormalization gets added multiple times

Input:

{"name": "foo","response_time": 30,"c_tests": ["a", "b"]}

{"name": "bar","response_time": 10,"c_tests": ["a", "b", "c"]}

Config:

  {
    source => "c_tests" 
    target => "it_works_"
    add_tag => "denormalized"
  }

Output:

{
             "name" => "foo",
    "response_time" => 30,
        "it_works_" => "a",
             "tags" => [
        [0] "denormalized"
    ]
}
{
             "name" => "foo",
    "response_time" => 30,
        "it_works_" => "b",
             "tags" => [
        [0] "denormalized",
        [1] "denormalized"
    ]
}
{
             "name" => "bar",
    "response_time" => 10,
        "it_works_" => "a",
             "tags" => [
        [0] "denormalized"
    ]
}
{
             "name" => "bar",
    "response_time" => 10,
        "it_works_" => "b",
             "tags" => [
        [0] "denormalized",
        [1] "denormalized"
    ]
}
{
             "name" => "bar",
    "response_time" => 10,
        "it_works_" => "c",
             "tags" => [
        [0] "denormalized",
        [1] "denormalized",
        [2] "denormalized"
    ]
}

The tag should be added only once per new event.

Aggregation error when averaging over fields in upper document layer

Example input:

{
"name": "foo",
"response_time": 30,
"c_tests": ["a", "b", "c"]
}
{
"name": "bar",
"response_time": 10,
"c_tests": ["a", "b", "c", "e", "f"]
}

creates output (assuming a config that sets the target field name to "c_test")

{ "name": "foo", "response_time": 30, "c_test": "a" }
{ "name": "foo", "response_time": 30, "c_test": "b" }
{ "name": "foo", "response_time": 30, "c_test": "c" }
{ "name": "bar", "response_time": 10, "c_test": "a" }
{ "name": "bar", "response_time": 10, "c_test": "b" }
{ "name": "bar", "response_time": 10, "c_test": "c" }
{ "name": "bar", "response_time": 10, "c_test": "d" }
{ "name": "bar", "response_time": 10, "c_test": "e" }

Problem: the average over the response time is 20 second on the original data.
On the modified data, the average is 17.5 due to the fact that the propertions of the uppermost document 'level' get distorted by the amount of elements in the denormalisation field.

Suggested solution was:
mark one of the spawned documents with a flag such as "first" or "master": true and add that as a filter criterion to any application on the uppermost level fields.

Rspec fails / uncomplete

• cannot execute rspec locally due to some log4j import error
• the rspec does not cover the code properly.