feature-request: ability to drop some log lines

[driskell]

feature-request: ability to drop some log lines

[sysmonk]

By the way, in my case it's 10+gb of logs per day per server, so that makes up some cpu/traffic

[deviantony]

Dropping useless lines by applying pre-filters on the node rather than on the indexer (logstash) seems logical for me too.

Could be configured this way:

/etc/log-courier/conf.d/smtp.conf

[ {
    "paths": [ "/var/log/smtp.log" ],
    "fields": [ "type": "smtp_log" ],
    "filters": ["^(.*connect from.*)$", "^(.*status=sent.*)$"]
} ]

Thus, applying pre-filters to grab only the following lines in a specified log file:

Jul 29 06:25:03 smtp1 postfix/smtpd[3533]: connect from unknown[x.x.x.x]
Jul 29 06:25:03 smtp1 postfix/smtp[3641]: ABCQUEUEID: to=<[email protected]>, relay=127.0.0.1[127.0.0.1]:10037, conn_use=72, delay=0.14, delays=0.04/0/0/0.09, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as SECONDQID)

[sysmonk]

I'd also like to have a negate option in that filter :)

[driskell]

This would be best implemented as a codec like the multiline one. It can receive events and throw out ones that don't match. That sound ok?

[sysmonk]

sounds perfect to me!

[driskell]

@sysmonk @deviantony

I've pushed a filter codec if you'd like to give it a try before we close the request? :)

[deviantony]

Hey @driskell, I'm currently on vacation for the week. I'll give it a try next week!

[sysmonk]

I'll try to run it on my servers today. By the way, is there an option to use multiple filters (both filter AND multiline in example) ?

[driskell]

No, sorry. It's going to be a fair bit more work to chain codecs together and then the purpose becomes grey as they stop being codecs and become filters. The existence of this filter codec is already odd terminology wise but I feel its a justified feature to prevent excessive shipping where its not wanted.

Now I think about it, maybe a codec is the wrong implementation here. Maybe we should be simply adding "filter" as a property of the existing codecs (plain and multiline) - this solves terminology problem and fixes the "entering grey area". And then it becomes usable for multiline. If we had other codecs for none "line" data (like JSON) we'd just leave the option out on it.

What do you think?

[sysmonk]

Makes sense to me, although i don't think there's a need to change the naming - codec is fine, just say that only one codec is possible at a time.

Now as for the filter codec itself - i fail to get it working at all...
My setup:

[
        {
                "paths": [ "/tmp/test.log" ],
                "fields": { "type": "testlog" },
                "codec": {
                                        "name": "filter",
                                        "patterns": [ 
                                                        ".*statistics.*",
                                                        "^(.*DKIM signing.*)$"
                                                        ],
                                        "negate": true
                                }
        }
]

Test data:

Aug 13 11:46:07 smtp1 postfix/smtpd[24787]: statistics: testing
Aug 13 11:46:07 smtp1 dkimproxy.out[18974]: DKIM signing - signed; message-id=<[email protected]>, signer=<[email protected]>, from=<[email protected]>
Aug 13 11:46:07 smtp1 postfix/smtpd[24787]: connect from unknown[1.1.1.1]
Aug 13 11:46:07 smtp1 postfix/smtpd[24787]: C64D26E0D72: client=unknown[1.1.1.1]

I see all 4 lines going to logstash/elasticsearch ( i don't want the DKIM signing and statistics line ). I tried using ".statistics.", "^.statistics.$", "statistics" and other similar regexps - they all still get through.

[driskell]

Ah yes. The negate applies to each pattern separately, not as a whole.

So its going "does it NOT match statistics?" if it does match it moves to next pattern, which goes "does it NOT match DKIN signing?" so essentially all lines will match :)

I'll adjust it later so the negate applies to the result. So instead it goes "does it match statistics?" if it doesn't, next pattern. Once finished, did it match any pattern? Then take negate into account.

Sorry for the confusion :)

[driskell]

I will leave things as they are once I change the negate behaviour and docs.

If there is a future demand for filtering AND multiline combined, I can remove the filter codec and instead make "filter_pattern" an option embedded in the plain codec and the multiline codec, so the filtering is part of the existing codecs as opposed to separate.

[driskell]

OK so negate behaviour is now as one expects. If you just have patterns, it only ships if it matches at least one of the patterns. If you then add negate, it only ships if it matches none of the patterns.

[driskell]

I will assume this functionality is working successfully! Thanks.