hoffie/multilog_exporter

Watches log files and exposes data as Prometheus metrics

multilog_exporter

Watches one or more configured log file paths, analyzes them based on configured patterns and exposes the result in the Prometheus metrics format.

The files are watched using inotify and can handle log file rotations (based on grok_exporter code).

The patterns are regular expressions. The pattern may contain groups which can later be used as label values or metric value.

Currently, the following actions are supported:

• add an arbitrary number (Counter, Gauge)
• set to a new value (Gauge)
• dec to decrease by an arbitrary number (Gauge)

Supported values (for metric results and label values):

• Static numbers, such as 1, 10.3e5
• Regex group references, such as $pool from the regexp pool=(?P<pool>[^ ]+) msg
• now() which returns the current Unix time

Metrics are registered at startup so that the relevant time series are generated even if no relevant log lines have been processed yet (only works for those without labels).

Building

Tested on Linux 4.x using go 1.10: go get -u github.com/hoffie/multilog_exporter. Run unit and integration tests using: make

Configuration

multilog_exporter is mainly configured using a configuration file in yaml syntax.

The following options are command line flags, though:

• --metrics.listen-addr, which specifies the ip and port for listening.
• --config.file, the path to the mentioned config file
• --debug, which helps diagnosing potential issues

The configuration file syntax is best described by the annotated example configuration. In general, the config consists of a list of log file paths. For each log file path, a list of patterns is configured. The pattern describes how to map log lines to Prometheus metrics.

Reloading configuration

Configuration can be reloaded without stopping the application, by means of a SIGHUP signal. This can be useful to change the configuration (for instance, adding new log files or patterns) without losing current exposed metrics.

$ kill -SIGHUP $(pidof multilog_exporter)

When doing so, the application log should reflect the reload

INFO[0137] SIGHUP received, reloading config file        configFile=doc/example.yaml
INFO[0137] Loading config file                           configFile=doc/example.yaml

Limitations

Multi-cardinality

There is no support for working with multi-cardinality in log files. In other words, the log line errors in the following pools: prod, test, dev cannot generate multiple time series with the same name, such as: errors{pool="prod"}, errors{pool="test"}, errors{pool="dev"}. If it turns out there is a use case for that, support might be added.

Arithmetic

There is no support for any calculations or variables (except for references to regex groups). If it turns out this is required and cannot be solved on the scraping side, support might be added.

Similar tools

You may have noticed that there are two very popular tools which seem to have similar goals to this project: Google's mtail and @fstab's grok_exporter. There is basically two reasons, why this tool exists:

• multilog_exporter provides an easy way to monitor multiple log files while still handling them as independent files. For instance, you could use it to monitor your system log and count the frequency of certain events while concurrently monitoring an application log with totally different data.
• multilog_exporter aims to provide a simple, integrated way to configure it, while the other tools implement own domain-specific languages or require knowledge of the grok format.

Both named tools seem to have a large user base and if their feature set matches your requirements, you are encouraged to use them. In fact, multilog_exporter even uses grok_exporter's library for handling file system watches.

License

This software is released under the Apache 2.0 license.

Author

multilog_exporter has been created by Christian Hoffmann. If you find this project useful, please star it or drop me a short mail.

multilog_exporter's Issues

Allow issuing a metric having as value the timestamp of the last update of the metric

Because of the nature of multilog_exporter (parsing logs and issuing metrics on their contents), the metrics endpoint will issue the same value until the logs triggers again the condition (if this log is generated by a cron job, then it might be its next execution).

But there are scenarios where this value might not change:

• The metric might not change, the log generates again the same metric value
• The log doesn´t trigger the metric (job not executed for some reason, job failing and not triggering the appropiate log entry...)

In such scenarios, {{multilog_endpoint}} doesn´t give enough visibility, as if we are in the second scenario we probably will want to investigate the root cause.

As discussed in #1 (comment) , a possible workaround is to have a sidecar metric added to the configuration that reports the timestamp when the metric is pushed. But given that this seems a useful thing to have, it makes sense to have it built-in, so that we can just tell {{multilog_exporter}} that we want to have that metric issued.

As a reference, pushgateway presents this behaviour through:

In order to make it easier to alert on failed pushers or those that have not run recently, the Pushgateway will add in the metrics push_time_seconds and push_failure_time_seconds with the Unix timestamp of the last successful and failed POST/PUT to each group. This will override any pushed metric by that name. A value of zero for either metric implies that the group has never seen a successful or failed POST/PUT.

(push_failure_time doesn´t make much sense in this scenario)

There are several approaches (speaking of user facing side) to expose this.

I was thinking of something like

timestamp: false
timestamp_metric: push_time_seconds  
logs:
- path: /logs/subtitles_download.log
  timestamp: false
  timestamp_metric: push_time_seconds
  patterns:
  - match: .*total time execution:\s*(?P<seconds>\d+)\s*seconds

Where:

• timestamp: Enables of disables such metric. Defaults to false for backwards compatibility
• timestamp_metric: Sets the name for the metric. Defaults to push_time_seconds for consistency with pushgateway
• Both can be set at the top configuration level, or at "path level", in case some wants to have different configuration for some logs and not for others (maybe this scenario is quite rare and not worth it, not sure)
• The metric will be issues for every metric issued by the log (as long as it is configured this to be enabled)
• The metric will have as
  • name: timestamp_metric
  • value: the epoch in seconds
  • labels: the same labels as the log related metric. Not sure if an extra label with the related log label should be added, pushgateway doesn´t do this so I would follow the same approach and just keep the same labels as the original metric

I can work on this issue, let´s just agree on the description.

error: reference to undefined identifier ‘syscall.NAME_MAX’

go get -u github.com/hoffie/multilog_exporter

github.com/hoffie/multilog_exporter/vendor/github.com/fstab/grok_exporter/tailer/fswatcher

/home/yva/go/src/github.com/hoffie/multilog_exporter/vendor/github.com/fstab/grok_exporter/tailer/fswatcher/fseventProducerLoop_linux.go:64:24: error: reference to undefined identifier ‘syscall.NAME_MAX’
bytes *[syscall.NAME_MAX]byte
^
/home/yva/go/src/github.com/hoffie/multilog_exporter/vendor/github.com/fstab/grok_exporter/tailer/fswatcher/fseventProducerLoop_linux.go:65:65: error: reference to undefined identifier ‘syscall.NAME_MAX’
buf = make([]byte, (syscall.SizeofInotifyEvent+syscall.NAME_MAX+1)10)
^
/home/yva/go/src/github.com/hoffie/multilog_exporter/vendor/github.com/fstab/grok_exporter/tailer/fswatcher/fseventProducerLoop_linux.go:97:25: error: reference to undefined identifier ‘syscall.NAME_MAX’
bytes = ([syscall.NAME_MAX]byte)(unsafe.Pointer(&buf[offset+syscall.SizeofInotifyEvent]))