ixixi/fluent-plugin-record_splitter

Output split array plugin for fluentd

fluent-plugin-record_splitter's Issues

Have split_key still be present in the record

First - thanks for this module. It has been quite useful.

I was wondering if there was an option for when the record was split, if the original record field could be preserved? To be more specific, I have a record that looks like this:

{
    "key1": "data1",
    "key2": "data2",
    "key3": {
        "key4": "data4",
        "key5": "data5"
    },
    "items": [ 
                {
                  "key6": "data6",
                  "key7": "data7"
                }, 
                {
                  "key8": "data8",
                  "key9": "data9",
                  "key10": "data10"
                }
            ],
    "key11": "data11"
}

A few notes: The elements inside the 'items' key are dictionaries, and do not neccessarily have the same keys (or even number of keys) inside them. They may or may not be overlap in the keys between dictionaries. In this example, they do not, but there is nothing stopping 'key6' also being present in the key8-10 dictionary with the same or different value than 'data6'.

I am trying to split_key on 'items'. If I do so, then I think this is the result (with keep_other_key true):

{
    "key1": "data1",
    "key2": "data2",
    "key3": {
        "key4": "data4",
        "key5": "data5"
    },
   "key6": "data6",
   "key7": "data7",
    "key11": "data11"
},
{
    "key1": "data1",
    "key2": "data2",
    "key3": {
        "key4": "data4",
        "key5": "data5"
    },
     "key8": "data8",
     "key9": "data9",
     "key10": "data10",
    "key11": "data11"
}

However, it totally removed all reference to 'items'. Ideally, I would just swap that to a dictionary (which now only has one 'entry'), like so:

{
    "key1": "data1",
    "key2": "data2",
    "key3": {
        "key4": "data4",
        "key5": "data5"
    },
    "items": {
                  "key6": "data6",
                  "key7": "data7"
                }, 
    "key11": "data11"
},
{
    "key1": "data1",
    "key2": "data2",
    "key3": {
        "key4": "data4",
        "key5": "data5"
    },
    "items": {
                  "key8": "data8",
                  "key9": "data9",
                  "key10": "data10"
                },
    "key11": "data11"
}

Do you have any suggestion on how I might do that or some way of enhancing this module to support that?

Cannot use the plugin with fluentd v1.4

When I follow the instructions to use the plugin with fluentd v1.4, I get this error message:

fluentd_1        | 2019-05-23 12:06:22 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Unknown filter plugin 'record_splitter'. Run 'gem search -rd fluent-plugin' to find plugins"
docker-elk_fluentd_1 exited with code 1

Is it maybe related to fluent/fluentd#1826?

BUG: use router.emit instead of Engine.emit

Appears due to the new fluentd , this error now pops up:

[warn]: emit transaction failed: error_class=RuntimeError error="BUG: use router.emit instead of Engine.emit"