trustpilot/beat-exporter

Elastic beat-exporter for Prometheus

beat-exporter for Prometheus

Exposes (file|metric)beat statistics from beats statistics endpoint to prometheus format, automaticly configuring collectors for apporiate beat type.

Current coverage

• filebeat
• metricbeat
• packetbeat - partial
• auditbeat - partial

Setup

Edit your *beat configuration and add following:

http:
  enabled: true
  host: localhost
  port: 5066

This will expose (file|metrics|*)beat http endpoint at given port.

Run beat-exporter:

$ ./beat-exporter

beat-exported default port for prometheus is: 9479

Point your Prometheus to 0.0.0.0:9479/metrics

Configuration reference

$ ./beat-exporter -help
Usage of ./beat-exporter:
  -beat.system
    	Expose system stats
  -beat.timeout duration
    	Timeout for trying to get stats from beat. (default 10s)
  -beat.uri string
    	HTTP API address of beat. (default "http://localhost:5066")
  -tls.certfile string
    	TLS certs file if you want to use tls instead of http
  -tls.keyfile string
    	TLS key file if you want to use tls instead of http
  -version
    	Show version and exit
  -web.listen-address string
    	Address to listen on for web interface and telemetry. (default ":9479")
  -web.telemetry-path string
    	Path under which to expose metrics. (default "/metrics")

Contribution

Please use pull requests, issues

beat-exporter's Issues

Beat exporter setup doesn't work

Hello,

I wanted to export filebeat metrics via beat exporter.

filebeat has a conf that exposes the metrics https://www.elastic.co/guide/en/beats/filebeat/master/http-endpoint.html

beat exporter's documentation also tells me to do the same https://github.com/trustpilot/beat-exporter/blob/master/readme.md#setup

Filebeat and beat-exporter are running on the same host in the same docker bridge network

Out of the box this config doesn't work. To make it work I had to change

http: 
  enabled: true 
  host: localhost 
  port: 5066

to

http: 
  enabled: true 
  host: 0.0.0.0 
  port: 5066

Why localhost config doesn't work : If the process inside the container is bound to localhost, localhost uses a special loopback interface that is distinct from regular networking. I am able to access the service from within the container (host) but you won't be able to route its traffic externally

In doing so I make the setup work for filebeat and beat-exporter, but cause issues between logstash and filebeat.(filebeat and logstash running on different server) filebeat logs

2020-06-23T10:50:06.823+0200	ERROR	logstash/async.go:256	Failed to publish events caused by: read tcp X.X.X.X:40590->X.X.X.X:5050: read: connection reset by peer 
2020-06-23T10:50:06.894+0200	ERROR	logstash/async.go:256	Failed to publish events caused by: client is not connected 2020-06-23T10:50:08.708+0200 ERROR	pipeline/output.go:121	Failed to publish events: client is not connected

[24th June 2020 14:36 PM] Noticed something :
As soon as I stop beat exporter and restart the filebeat instance, the above mentioned error do not appear and filebeat/logstash work as expected

So the setup has a container of filebeat and beat-exporter running on application-server and logstash running on a my log-server
With the below config in filebeat. filebeat and logstash work (so i will rule out the possibility that 0.0.0.0 doesn't work )

http:
  enabled: true
  host: 0.0.0.0
  port: 5066

Now when i start beat-exporter, I start to see the above errors (not sure what could cause it)
But even after stopping beat-exporter the above errors persist and it only gets resolved once i restart filebeat
Regards
Ashish

beat_exporter error

I am trying to setup beat_exporter as a systemd service. However I am seeing the following errors when I start it. I created its own user and told the group to be root. I am running CentOS 7 latest.

Apr 19 14:38:58 server1 filebeat_exporter[21087]: {"level":"error","message":"Could not load beat type, with error: Get 10.150.10.10: unsupported protocol scheme \"\", retrying in 5s","time":"2019-04-19T14:38:58-05:00"}
Apr 19 14:39:03 server1 filebeat_exporter[21087]: {"level":"error","message":"Could not load beat type, with error: Get 10.150.10.10: unsupported protocol scheme \"\", retrying in 5s","time":"2019-04-19T14:39:03-05:00"}
Apr 19 14:39:08 server1 filebeat_exporter[21087]: {"level":"error","message":"Could not load beat type, with error: Get 10.150.10.10: unsupported protocol scheme \"\", retrying in 5s","time":"2019-04-19T14:39:08-05:00"}
Apr 19 14:39:13 server1 filebeat_exporter[21087]: {"level":"error","message":"Could not load beat type, with error: Get 10.150.10.10: unsupported protocol scheme \"\", retrying in 5s","time":"2019-04-19T14:39:13-05:00"}

Should I run this as root same as filebeat?

Beat Exporter Could Not Parse JSON Response for Target

It seems, in the previous version of beat registrar stats writes didn't have subdivisions of fail, success and total. In newer version of beat they do. This is causing this error:

{"level":"error","message":"Could not parse JSON response for target","time":"2018-09-13T10:55:52Z"}
{"level":"error","message":"Failed getting /stats endpoint of target","time":"2018-09-13T10:55:52Z"}

Upgrade dependencies

prometheus/client_golang to v0.9.2
sirupsen/logrus to v1.2.0

Check prometheus metrics

When I test the metrics form exporter I'm getting this result

curl -qs 0.0.0.0:9479/metrics | ./promtool check metrics
filebeat_cpu_ticks counter metrics should have "_total" suffix
filebeat_cpu_time_miliseconds counter metrics should have "_total" suffix
filebeat_libbeat_config_reloads counter metrics should have "_total" suffix
filebeat_libbeat_output counter metrics should have "_total" suffix
filebeat_libbeat_output_read_bytes counter metrics should have "_total" suffix
filebeat_libbeat_output_read_errors counter metrics should have "_total" suffix
filebeat_libbeat_output_write_bytes counter metrics should have "_total" suffix
filebeat_libbeat_output_write_errors counter metrics should have "_total" suffix
filebeat_memstats_gc_next counter metrics should have "_total" suffix
filebeat_memstats_memory_total non-counter metrics should not have "_total" suffix
filebeat_system_cpu_cores counter metrics should have "_total" suffix
filebeat_uptime_seconds counter metrics should have "_total" suffix

Can you please check?

Rename `type` label

There's an overly generic type label on the events output that should probably be renamed. I would suggest event or event_name.

Beat Exporter 0.1.1 still doesn't work with Filebeat 6.3.0

I still have problem exporting FB 6.3.0 even with the latest BE 0.1.1.

First, FB 6.3.0 http had been enabled. I could see stats at 0.0.0.0:5066.

# curl http://localhost:5066/stats
{"beat":{"cpu":{"system":{"ticks":4180,"time":{"ms":4181}},"total":{"ticks":11370,"time":{"ms":11377},"value":11370},"user":{"ticks":7190,"time":{"ms":7196}}},"info":{"ephemeral_id":"4b46befe-9f6d-4c90-821a-aee743445b97","uptime":{"ms":2270145}},"memstats":{"gc_next":4743984,"memory_alloc":2974376,"memory_total":743705048,"rss":47874048}},"filebeat":{"events":{"active":10,"added":25606,"done":25596},"harvester":{"closed":5,"open_files":5,"running":5,"skipped":0,"started":10},"input":{"log":{"files":{"renamed":0,"truncated":0}}}},"libbeat":{"config":{"module":{"running":0,"starts":0,"stops":0},"reloads":0},"output":{"events":{"acked":25563,"active":0,"batches":2262,"dropped":0,"duplicates":0,"failed":0,"total":25563},"read":{"bytes":7296398,"errors":0},"type":"elasticsearch","write":{"bytes":17887252,"errors":0}},"pipeline":{"clients":1,"events":{"active":10,"dropped":0,"failed":0,"filtered":33,"published":25573,"retry":32,"total":25606},"queue":{"acked":25563}}},"registrar":{"states":{"cleanup":3,"current":14,"update":25596},"writes":2278},"system":{"cpu":{"cores":32},"load":{"1":0.16,"15":0.11,"5":0.11,"norm":{"1":0.005,"15":0.0034,"5":0.0034}}}}

However, BE 0.1.1 couldn't export any metric about FB 6.3.0 except "filebeat_up 0".

# curl 0.0.0.0:9479/metrics
# HELP beat_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which beat_exporter was built.
# TYPE beat_exporter_build_info gauge
beat_exporter_build_info{branch="master",goversion="go1.10.3",revision="67af338c0fed4d18831566b9301e812277efbe60",version="0.1.1"} 1
# HELP filebeat_up Target up
# TYPE filebeat_up gauge
filebeat_up 0

Here is my BE 0.1.1 cmdline and output showing the exact error.

# /opt/beat_exporter/beat-exporter-0.1.1.linux-amd64/beat-exporter -beat.timeout 10s
{"level":"info","message":"Exploring target for beat type","time":"2018-09-22T00:20:52Z"}
{"beat":"filebeat","hostname":"myhost","level":"info","message":"Target beat configuration loaded successfully!","name":"myhost","time":"2018-09-22T00:20:52Z","uuid":"3d7ab23d-a338-4e6e-b0ba-d739af6cf201","version":"6.3.0"}
{"addr":":9479","level":"info","message":"Starting exporter with configured type: filebeat","time":"2018-09-22T00:20:52Z"}
{"level":"error","message":"Could not parse JSON response for target","time":"2018-09-22T00:21:22Z"}
{"level":"error","message":"Failed getting /stats endpoint of target: json: cannot unmarshal number into Go struct field Registrar.writes of type struct { Fail float64 \"json:\\\"fail\\\"\"; Success float64 \"json:\\\"success\\\"\"; Total float64 \"json:\\\"total\\\"\" }","time":"2018-09-22T00:21:22Z"}

filebeat_up metric issue

Hi guys, first of all thanks for the great work and support that you've put to this project !

I just want to mention an issue that i witnessed by deploying the beat-exporter as a side container in a pod next to filebeat in a kubernetes environment :

If the filebeat pod which exports the metrics on the port 5066 is in a state different than CrashLoopBackOff - filbeat_up returns 0 - which is the expected behavior everything works fine.If the filebeat pod enters in a condition of a CrashLoopBackOff then beat-exporter doesn't register anything related to the pod hence filebeat_up is absent and all the metrics for this particular pod.
CrashLoopBackOff status of the filebeat pod - beat -exporter logs:

{"level":"error","message":"Could not load beat type, with error: Get http://localhost:5066: dial tcp 127.0.0.1:5066: connect: connection refused, retrying in 1s","time":"2021-01-05T09:58:45Z"}
{"level":"error","message":"Could not load beat type, with error: Get http://localhost:5066: dial tcp 127.0.0.1:5066: connect: connection refused, retrying in 1s","time":"2021-01-05T09:58:46Z"}

and here is the case when the POD is not in a CrashLoopBackOff / Error but in a different failed state and the filebeat_up is evaluated correctly to 0 :

{"level":"error","message":"Failed getting /stats endpoint of target: Get http://localhost:5066/stats: dial tcp 127.0.0.1:5066: connect: connection refused","time":"2021-01-05T09:59:04Z"}
{"level":"error","message":"Could not fetch stats endpoint of target: http://localhost:5066","time":"2021-01-05T09:59:25Z"}
{"level":"error","message":"Failed getting /stats endpoint of target: Get http://localhost:5066/stats: dial tcp 127.0.0.1:5066: connect: connection refused","time":"2021-01-05T09:59:25Z"}
{"level":"error","message":"Could not fetch stats endpoint of target: http://localhost:5066","time":"2021-01-05T09:59:34Z"}

Retries exceeded causes pod to wrongly restart

When launching a new pod the beats-exporter container will shut down with the following error message

2022-10-12 09:30:48,767 - __main__ - ERROR - Error connecting Beat at port 5066:
HTTPConnectionPool(host='localhost', port=5066): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f149d86b0d0>: Failed to establish a new connection: [Errno 111] Connection refused'))

However, after a couple of restarts (usually no more than 2) the pod will be alive and ready. I assume this is the filebeat container not being ready fast enough for the exporter to exhaust it's retries

is there some k8s way to handle this? i can add a delay by overriding the container CMD but it looks like kind of a hack for me
maybe we could set a startup delay using arguments? or increase interval between retries?

Crash while trying to export metrics for apm-server.

Hi,

I have setup beat-exporter as a sidecar to my apm-server in kubernetes but the beat-exporter crashes with the following log message:

{"level":"info","message":"Exploring target for beat type","time":"2020-11-23T17:56:52Z"}
{"beat":"apm-server","hostname":"apmserver-5668d79455-zwfp7","level":"info","message":"Target beat configuration loaded successfully!","name":"apmserver-5668d79455-zwfp7","time":"2020-11-23T17:56:53Z","uuid":"e2243c95-c7e6-47b8-8f04-b82edc41b069","version":"7.9.1"}
panic: descriptor Desc{fqName: "apm-server_up", help: "Target up", constLabels: {}, variableLabels: []} is invalid: "apm-server_up" is not a valid metric name

My guess is that apm-server reports it's name containing a dash which in turn makes beat-exporter generate a non valid Prometheus metric name.
The solution might be to convert all invalid chars in metricnames to underscores, or simply just removing them. In my case I would have been fine with either apm_server_up or apmserver_up.

Inconsistant naming of version

Hello,

Previous version was tagged v0.1.2 but latest bersion is tagged 0.2.0 (without leading 'v').

Is it a mistake or did you change your naming convention (it puts off my updating tool).

(Also, in the packages, a "." was transformed into a "-" but it make more sense like that)

Thanks for the exporter

rename prometheus related flags'web.listen-address' to 'web.listen-port'

Currently the prometheus flags are under the web. prefix:

$ ./beat-exporter -help
Usage of ./beat-exporter:
  ...
  -web.listen-address string
        Address to listen on for web interface and telemetry. (default ":9479")
  -web.telemetry-path string
        Path under which to expose metrics. (default "/metrics")

I propose we put this under prom, and maybe sharpen the language a bit:

$ ./beat-exporter -help
Usage of ./beat-exporter:
  ...
  -prom.port string
        Port to listen on for web interface and telemetry. (default "9479") <-- leave out the quote
  -prom.path string
        Path under which to expose metrics. (default "/metrics")

Thoughts?

Connection refused while using beat-exporter and port forward.

Hello
I have created port forward from pod on cluster to locahost, which exposes filebeat metrics.
Under http://localhost:5066/stats?pretty I can see those metrics without any problems(curl or browser).
When I run beat-exporter i have error message connection refused:
{"level":"info","message":"Exploring target for beat type","time":"2021-11-30T09:57:45Z"} {"level":"error","message":"Could not load beat type, with error: Get http://localhost:5066: dial tcp 127.0.0.1:5066: connect: connection refused, retrying in 1s","time":"2021-11-30T09:57:46Z"} {"level":"error","message":"Could not load beat type, with error: Get http://localhost:5066: dial tcp 127.0.0.1:5066: connect: connection refused, retrying in 1s","time":"2021-11-30T09:57:47Z"} {"level":"error","message":"Could not load beat type, with error: Get http://localhost:5066: dial tcp 127.0.0.1:5066: connect: connection refused, retrying in 1s","time":"2021-11-30T09:57:48Z"} ^C{"level":"info","message":"Signal received: interrupt","time":"2021-11-30T09:57:48Z"}
Does anyone had similar problem? If yes, what's the solution?
Thanks in advance.

Provide a Dockerfile / Create Docker image on DockerHub

Thanks you tremendously for taking the effort to write this exporter! It really helps a lot to maintain a healthy logging infrastructure.

When using the exporter with Filebeat running containerized (i.e.inside Kubernetes) one would like to simply place your beat-exporter next to it as a sidecar. Is there any chance you would add a Dockerfile and potentially also use that to place an "official" Docker image on the Docker Hub?

metricbeat filesystem metrics not being output to prometheus

In my metricbeat console output I have this:

  "system": {
    "filesystem": {
      "type": "fuse.glusterfs",
      "device_name": "10.154.131.242:vol_f19380703ea925da14813fcfa05fa331",
      "available": 10524708864,
      "mount_point": "/mnt",
      "free_files": 5225819,
      "used": {
        "pct": 0.0189,
        "bytes": 202223616
      },
      "total": 10726932480,
      "files": 5242880,
      "free": 10524708864
    }
  },

But in metric output from beat-exporter I only have:
metricbeat_metricbeat_system_filesystem system.filesystem
metricbeat_metricbeat_system_filesystem counter

and event failures and success.. Nothing about the above stats - which is what I actually need :(

Explanation of metrics

Love the exporter, but is there somewhere with a good description of what each of the metrics tracks?

Expose the stats of several filebeat instances

Hi there,

I understand that beat.uri allows to configure one unique beat instance we want to expose statistics.

But I have a question: I have a use case where I need to expose the stats of several filebeat instances. Is there a mean to aggregate the metrics "by host" for those beat instances ?

If not, do you think it make sense to add that feature in beat-exporter ?

Best regards.

Add support to monitor multiple beats

It would be great if a single beat_exporter could monitor both a filebeat and a metricsbeat instance for example.

Miss type for System field

beat-exporter/collector/beat.go

Line 21 in 54b852f

Sytem CPUTimings `json:"system"`

Publish release checksums

It would be nice to publish sha256sums.txt for the release binaries for validation.

See this example in Prometheus.

Feature: add beat version info

Feature idea, add beat info metrics from the stats endpoint /

{
  "beat": "pubsubbeat",
  "hostname": "pubsub-sidekiq-inf-gstg",
  "name": "pubsub-sidekiq-inf-gstg",
  "uuid": "FOO",
  "version": "7.5.1"
}

Would make a metrics like this:

pubsubbeat_build_info{version="7.5.1"} 1
pubsubbeat_beat_info{beat_name="pubsub-sidekiq-inf-gstg",uuid="FOO",} 1

Beat exporter need support `filebeat.inputs - type: log`; but not the deprecated `filebeat.prospectors: - input_type: log`

beat-exporter/collector/filebeat.go

Line 30 in 67af338

} `json:"prospector"`

Basically, just need replace "prospector" with "input" at the above line.

Please note that filebeat.prospectors is now deprecated and people should start using filebeat.inputs in their filebeat.yml instead.

Deprecated:

filebeat.prospectors:
- input_type: log

New:

filebeat.inputs:
- type: log
  paths:
  - /var/log/*.log
  - /var/log/*/*.log
  - /var/log/*/*/*.log
  - /var/log/*/*/*/*.log

Here is what curl http://localhost:5066/stats will show for filebeat.input from FB 6.4.1

  "filebeat": {
    "events": {
      "active": 11,
      "added": 497,
      "done": 486
    },
    "harvester": {
      "closed": 0,
      "open_files": 6,
      "running": 6,
      "skipped": 0,
      "started": 6
    },
    "input": {
      "log": {
        "files": {
          "renamed": 0,
          "truncated": 0
        }
      }
    }
  }

No data when exporter started without beat

When the beat exporter is started but the beat (filebeat in my case) start failed, the exporter doesn't send back any data.

I would have expected to get the usual exporter internal metrics and a filebeat_up set to 0.
As it is, I have aggregate the up{job="beat"} == 0 and filebeat_up == 0 to have a unified view.

It is not a deal breaker but it would make querying easier.