grosser/logrecycler

Re-process logs from applications you cannot modify to convert them to json and add prometheus/stats metrics

Logrecycler

Re-process logs from applications you cannot modify to:

• convert plaintext or glog logs from stdin (or command) to json on stdout
• remove noise
• add log levels / timestamp / details / captured values
• emit prometheus metric
• emit statsd metric

Example

stdin: I0530 10:13:00.740596      33 foo.go:132] error connecting to remote host foobar.com:12345
stdout: {"ts":"2020-05-30 10:13:00","level":"error","message":"error connecting to remote host","host":"foobar.com","port":"1234","pattern":"connection-error"}
/metrics: log_total{level="error",host="foobar.com",port="1234",pattern="connection-error"} 1

Setup

Install

Download the latest binary:

curl -sfL <PICK URL FROM RELEASES PAGE> | tar -zx && chmod +x logrecycler && ./logrecycler --version

Configure

Configure a logrecycler.yaml in your project root:

# optional settings
# timestampKey: ts # what to call the timestamp in the logs (for example @timestamp, ts, leave empty for no timestamp)
# levelKey: level # what to call the level in the logs (for example level/lvl/severity, leave empty for no level)
# messageKey: msg # what to call the message in the logs (leave empty for 'message')
# glog: simple # convert glog style prefix ([IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] message) into timestamp/level/message
# json: simple # assume input starting with `{` and ending with `}` as json and merge it, also set allowMetricLabels to avoid metric spam and match the level+message+timestamp keys with the input
# preprocess: '[^\]]+\] (?P<message>.*)' # reduce noise from message by replacing it with captured (for example remove, leave empty for none)
# allowMetricLabels: [foo] # ignore everything but these

# enable prometheus /metrics
# when using: try to use the same `add` value and the same named regex captures in patterns below
# to avoid running out of memory
# prometheus:
#   port: 1234

# enable statsd metric
# statsd:
#   address: 0.0.0.0:8125
#   metric: my_app.logs

# patterns to match ... each log line only match the first matching pattern
patterns:
# simple match
- regex: 'error.*parsing' # log line needs to match this
  level: ERROR
  add: # will appear in log and metric
    pattern: parsing-error # using the same pattern key here, so we can group by pattern when reporting
# named captures go into logs, replacing message here too
- regex: '(?P<message>error connecting .*) (?P<host>\S+):(?P<port>\d+)'
  level: ERROR
  add:
    pattern: connection-error
  ignoreMetricLabels: ["host"] # do not use "host" as metric
# override message if it includes secrets
- regex: 'secret key is'
  level: INFO
  add:
    pattern: secret
    message: secret key redacted # override message
- regex: 'Waited for .* due to client-side throttling'
  level: INFO
  sampleRate: 0.01 # sample only 1%
  add:
    pattern: throttle
# discard spam
- regex: 'todays weather is'
  discard: true
# mark all unmatched as unknown so we can alert on it
- pattern: '' # catch all
  level: WARN
  add:
    pattern: unknown

Use

Pipe your logs to the recycler:

set -o pipefail; <your-program-here> | logrecycler

or make the recycler call your command:

logrecycler -- <your-program-here>

SVM

The released go binary includes dependency metadata, if you do not use prometheus or do not expose the prometheus endpoint to the public, you should strip the logrecycler binary, to avoid false positive vulnerability detection.

Development

Test

• go get github.com/grosser/go-testcov
• install any version of ruby (used for integration tests)
• make test

Release

• manually tag on master
• create a new release via github UI
• GA workflow will automatically build a new binary

TODO

• glog: full to also capture location and thread
• support json log parsing and rewriting
• basic benchmark of memory/cpu overhead (without counting startup time)
• more examples

Author

Michael Grosser
[email protected]
License: MIT

logrecycler's Issues

Bug: doesn't escape json

Actual:

[~/Desktop] ➔ echo "foo\"bar" | ./logrecycler
{"message":"foo"bar"}

Expected:

[~/Desktop] ➔ echo "foo\"bar" | ./logrecycler
{"message":"foo\"bar"}

@grosser

Sample log patterns

Currently, you can only preserve or discard logs. It would be nice to have a random sample in the event that you want to care about something but not every entry of it.

Something like:

- regex: 'Waited for .* due to client-side throttling'
  sample: 0.01 # keep 1% of it (can be random)