sysulq/nginx-vts-exporter

Simple server that scrapes Nginx vts stats and exports them via HTTP for Prometheus consumption

nginx-vts-exporter

nginx-vts-exporter is powered by Kod, which is a dependency injection framework for Go.
It is designed to be simple and easy to use, and to provide a consistent way to manage dependencies across your application.

Simple server that scrapes Nginx vts stats and exports them via HTTP for Prometheus consumption

To support time related histogram metrics, please refer to sysulq/nginx-prometheus-metrics or #43.

ANN

It's hard to say that this project is not maintained any longer, and it is recommended to use nginx-vtx-module instead, which supports multiple vhost_traffic_status_display_format, like <json|html|jsonp|prometheus>.

Hope you guys enjoy it, and thanks for all the contributors and the issue finders. 😃

Table of Contents

• nginx-vts-exporter
  • ANN
  • Table of Contents
  • Dependency
  • Download
  • Compile
    • build binary
    • build RPM package
    • build docker image
  • Docker Hub Image
  • Run
    • run binary
    • run docker
  • Environment variables
  • Metrics
    • Server main
    • Server zones
    • Filter zones
    • Upstreams

Dependency

• nginx-module-vts
• Prometheus
• Golang

Download

Binary can be downloaded from Releases page.

Compile

build binary

make

build RPM package

make rpm

build docker image

make docker

Docker Hub Image

docker pull sophos/nginx-vts-exporter:latest

It can be used directly instead of having to build the image yourself. (Docker Hub sophos/nginx-vts-exporter)

Run

run binary

nohup /bin/nginx-vts-exporter -nginx.scrape_uri=http://localhost/status/format/json

run docker

docker run  -ti --rm --env NGINX_STATUS="http://localhost/status/format/json" sophos/nginx-vts-exporter

Environment variables

This image is configurable using different env variables

Variable name	Default	Description
NGINX_STATUS	http://localhost/status/format/json	Nginx JSON format status page
METRICS_ENDPOINT	/metrics	Metrics endpoint exportation URI
METRICS_ADDR	:9913	Metrics exportation address:port
METRICS_NS	nginx	Prometheus metrics Namespaces

Metrics

Documents about exposed Prometheus metrics.

For details on the underlying metrics please see nginx-module-vts

For grafana dashboard please see nginx-vts-exporter dashboard

Server main

Metrics details

Nginx data	Name	Exposed informations
Info	{NAMESPACE}_server_info	hostName, nginxVersion, uptimeSec
Connections	{NAMESPACE}_server_connections	status [active, reading, writing, waiting, accepted, handled]

Metrics output example

# Server Info
nginx_server_info{hostName="localhost", nginxVersion="1.11.1"} 9527
# Server Connections
nginx_server_connections{status="accepted"} 70606

Server zones

Metrics details

Nginx data	Name	Exposed informations
Requests	{NAMESPACE}_server_requests	code [2xx, 3xx, 4xx, 5xx, total], host (or domain name)
Bytes	{NAMESPACE}_server_bytes	direction [in, out], host (or domain name)
Cache	{NAMESPACE}_server_cache	status [bypass, expired, hit, miss, revalidated, scarce, stale, updating], host (or domain name)

Metrics output example

# Server Requests
nginx_server_requests{code="1xx",host="test.domain.com"} 0

# Server Bytes
nginx_server_bytes{direction="in",host="test.domain.com"} 21

# Server Cache
nginx_server_cache{host="test.domain.com",status="bypass"} 2

Filter zones

Metrics details

Nginx data	Name	Exposed informations
Requests	{NAMESPACE}_filter_requests	code [2xx, 3xx, 4xx, 5xx and total], filter, filter name
Bytes	{NAMESPACE}_filter_bytes	direction [in, out], filter, filter name
Response time	{NAMESPACE}_filter_responseMsec	filter, filter name

Metrics output example

# Filter Requests
nginx_upstream_requests{code="1xx", filter="country", filterName="BY"} 0

# Filter Bytes
nginx_upstream_bytes{direction="in", filter="country", filterName="BY"} 0

# Filter Response time
nginx_upstream_responseMsec{filter="country", filterName="BY"} 99

Upstreams

Metrics details

Nginx data	Name	Exposed informations
Requests	{NAMESPACE}_upstream_requests	code [2xx, 3xx, 4xx, 5xx and total], upstream (or upstream name)
Bytes	{NAMESPACE}_upstream_bytes	direction [in, out], upstream (or upstream name)
Response time	{NAMESPACE}_upstream_responseMsec	backend (or server), in_bytes, out_bytes, upstream (or upstream name)

Metrics output example

# Upstream Requests
nginx_upstream_requests{code="1xx",upstream="XXX-XXXXX-3000"} 0

# Upstream Bytes
nginx_upstream_bytes{direction="in",upstream="XXX-XXXXX-3000"} 0

# Upstream Response time
nginx_upstream_responseMsec{backend="10.2.15.10:3000",upstream="XXX-XXXXX-3000"} 99

nginx-vts-exporter's Issues

Feature: Return a histogram for response time

This exporter should have the option to expose response times as a histogram, rather than a simple average. The dataset to support this feature is already available in the upstreamZones. requestMsecs.msecs data structure from VTS.

Proposed Interface: An optional command line flag -nginx.response.histogram_buckets is added, and takes a comma separated list of integers as an argument. When declared, two additional metrics are implemented: {NAMESPACE}_filter_responses and {NAMESPACE}_upstream_responses. They are both histograms with bucket sizes from the input flag.

Note: I am actively building feature; any comments on the interface please let me know!

[Optimize] Health check scraping website page like "/health" or "/metrics" will contaminate the website visit statistics

Hi:
As I said in issue title, exporter scrape page like /health or /metrics/format/json will contaminate website visit statistics, Those visits have no relationship with real service.
I deem that this could be optimize in ngx-vts-module, but could we do sth in this exporter?

import template form grafana import dashboard ,data not show

import template form grafana import dashboard ,data not show

because every chart metric contains a invalid 'kubeingress' chars,

delete this chars ,the data will show

the template id is 1623

ENV var issues

Me again.

I noticed some issues/quirks with the env vars and I would normally submit a PR, but I'm not sure how to handle things in terms of backwards compatibility etc.

1. Is the NGIX_HOST env var supposed to be named NGINX_HOST? not a big issue, but might be confusing for some.

2. In the README, the example docker run command provides --env NGIX_HOST="http://localhost/status/format/json" (with appended status path), but in the docker-entrypoint.sh it has default_status="$NGIX_HOST/status/format/json" (which, again, appends the status path). I think it should either have --env NGINX_STATUS="http://localhost/status/format/json" or --env NGIX_HOST="http://localhost"

scrape insecure not working

Hi,

nice work! But i got a little error, it seems the ssl-certificate of the scrape-target is checked anyway..?

/usr/local/bin/prometheus_nginx_exporter -insecure -nginx.scrape_uri https://localhost/nginx_status_vts
2017/10/27 11:35:22 Starting nginx_vts_exporter (version=0.6, branch=HEAD, revision=e48bd6899d8864b11b9f71f852b049ec3ad2ae26)
2017/10/27 11:35:22 Build context (go=go1.8.3, user=root@3b0fc1d0acb6, date=20170808-07:09:08)
2017/10/27 11:35:22 Starting Server at : :9913
2017/10/27 11:35:22 Metrics endpoint: /metrics
2017/10/27 11:35:22 Metrics namespace: nginx
2017/10/27 11:35:22 Scraping information from : https://localhost/nginx_status_vts
2017/10/27 11:35:23 fetchHTTP failed Get https://localhost/nginx_status_vts: x509: certificate is valid for *.xyz.de, not localhost

Add grafana dashboard?

json.Unmarshal failed

[root@localhost nginx_export]# ./nginx-vts-exporter -nginx.scrape_uri=http://xxx/status/format/json
2019/05/20 18:57:51 Starting nginx_vts_exporter (version=0.10.3, branch=HEAD, revision=8aa2881c7050d9b28f2312d7ce99d93458611d04)
2019/05/20 18:57:51 Build context (go=go1.10, user=root@56ca8763ee48, date=20180328-05:47:47)
2019/05/20 18:57:51 Starting Server at : :9913
2019/05/20 18:57:51 Metrics endpoint: /metrics
2019/05/20 18:57:51 Metrics namespace: nginx
2019/05/20 18:57:51 Scraping information from : http://xxx/status/format/json
2019/05/20 18:58:28 json.Unmarshal failed invalid character '\x1f' in string literal

func (e *Exporter) Collect(ch chan<- prometheus.Metric) {
	body, err := fetchHTTP(e.URI, time.Duration(*nginxScrapeTimeout)*time.Second)()
	if err != nil {
		log.Println("fetchHTTP failed", err)
		return
	}
	defer body.Close()

	data, err := ioutil.ReadAll(body)
	if err != nil {
		log.Println("ioutil.ReadAll failed", err)
		return
	}
      // add this  code to remove character . now it worker 
	for index, char := range data {
		switch {
		case char > '~':
			data[index] = ' '
		case char == '\r':
		case char == '\n':
		case char == '\t':
		case char < ' ':
			data[index] = ' '
		}
	}
	var nginxVtx NginxVts
	err = json.Unmarshal(data, &nginxVtx)

how to start nginx-vts-exporter in gitlab9.4...

Here I'm so sorry..but I really can not solve it. my problem is, how to install and start nginx-vts-exporter correctly.
i install gitlab with rpm and it's version is 9.4.2, i do as the guide md but not get expected result, i add the config in : /var/opt/gitlab/nginx/conf/nginx.conf

server {
listen 9914;
server_name 10.21.20.124;

  location /status {
      vhost_traffic_status_display;
      vhost_traffic_status_display_format html;
  }   

}

but in this way i can not restart the nginx with gitlab-ctl restart nginx which will raise a timeout error.
due to not configing nginx correct so when i run nginx-vts-exporter:
nohup /bin/nginx-vts-exporter -nginx.scrape_uri=http://localhost:9914/status/format/json
it will tell get timeout error.

i am not sure it's a right place to ask this ques, but i really don't know when to find help. you know the doc is rare for me, thanks really much! waitting for your responce.
other: install nginx-vts-exporter with command go get github.com/hnlq715/nginx-vts-exporter and start nginx-vts-exporter in the dir GOPATH/bin/

unsupported protocol scheme ""

I have nginx with vts and using the exporter I see this in the logs

fetchHTTP failed Get /status/format/json: unsupported protocol scheme ""

I can manually go to the /status/format/json endpoint and see the json data. Here's a sample

{"hostName":test-2rb6h","nginxVersion":"1.13.1","loadMsec":1497388530496,"nowMsec":1497388857639,"connections":{"active":1,"reading":0,"writing":1,"waiting":0,"accepted":20,"handled":20,"requests":317},"sharedZones":{"name":"ngx_http_vhost_traffic_status","maxSize":1048575,"usedSize":2441,"usedNode":1},"serverZones":{"test":{"requestCounter":316,"inBytes":102925,"outBytes":901605,"responses":{"1xx":0,"2xx":316,
"3xx":0,"4xx":0,"5xx":0,"miss":0,"bypass":0,"expired":0,"stale":0,"updating":0,"revalidated":0,"hit":0,"scarce":0},"requestMsec":0,"requestMsecs":{"times":[1497388770070,1497388771072,1497388772070,1497388773072,1497388774071,1497388775071,1497388776071,1497388777071,1497388778070,1497388779071,1497388780073,1497388781071,1497388782071,1497388783071,1497388784071,1497388785071,1497388786071,1497388787071,1497388788071,1497
388789071,1497388790071,1497388791072,1497388792071,1497388793072,1497388794071,1497388795071,1497388796072,1497388797071,1497388798071,1497388799072,1497388800071,1497388801071,1497388802072,1497388803071,1497388804071,1497388805071,1497388806071,1497388807070,1497388808070,1497388809072,1497388810070,1497388811072,1497388812071,1497388813077,1497388814071,1497388815071,1497388816071,1497388817071,1497388818071,1497388819
072,1497388820071,1497388821071,1497388822071,1497388823071,1497388824078,1497388825071,1497388826076,1497388827073,1497388828071,1497388829071,1497388830071,1497388831071,1497388856221],"msecs":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},"overCounts":{"maxIntegerSize":18446744073709551615,"requestCounter":0,"inBytes":0,"outBytes":0,"1xx":
0,"2xx":0,"3xx":0,"4xx":0,"5xx":0,"miss":0,"bypass":0,"expired":0,"stale":0,"updating":0,"revalidated":0,"hit":0,"scarce":0}},"*":{"requestCounter":316,"inBytes":102925,"outBytes":901605,"responses":{"1xx":0,"2xx":316,"3xx":0,"4xx":0,"5xx":0,"miss":0,"bypass":0,"expired":0,"stale":0,"updating":0,"revalidated":0,"hit":0,"scarce":0},"requestMsec":0,"requestMsecs":{"times":[],"msecs":[]},"overCounts":{"maxIntegerSize":1844674
4073709551615,"requestCounter":0,"inBytes":0,"outBytes":0,"1xx":0,"2xx":0,"3xx":0,"4xx":0,"5xx":0,"miss":0,"bypass":0,"expired":0,"stale":0,"updating":0,"revalidated":0,"hit":0,"scarce":0}}}}

The metrics endpoint only shows

# TYPE nginx_vts_exporter_build_info gauge
nginx_vts_exporter_build_info{branch="fix-docker-error",goversion="go1.8",revision="0f3dbb44a86340d65bf3d6abbcc0ee88663cb419",version="0.4"} 1

I'm still troubleshooting but wanted to open an issue as there appears to be some configuration I've set up that doesn't work.

Grafana dashboard is off compared to metrics

The problem:
This Grafana dashboard doesn't seem to be reporting right numbers for ServerBytes

• https://grafana.com/dashboards/2949

My efforts so far:
Can the problem be that the nginx-vts-exporter outputs metrics in scientific notation, which then is misinterpreted by Grafana? This seems plausible from my numbers. A quick test with Go shows that printing float64 defaults to scientific notation, whereas printing uint64 results in only digits. Hoewever, changing types in nginx_vts_exporter.go isn't compatible with the function signature for prometheus.MustNewConstMetric.

Is there some way to force formatting in the nginx-vts-exporter metrics output to check if that's the problem?

My Grafana dashboard shows the following:

• ServerBytes for the all hosts peak at 12.92KiB out
• ServerBytes for the owncloud host peak at 4.1KiB out
• The JSON-file and the nginx-vts-exporter metrics are accumulated numbers, but even so the grafana graph seems far off.

http://nginx-vts-exporter:9913/metrics contains, among other data, the following lines:

nginx_server_bytes{direction="in",host="my.owncloud.host"} 6.7403731e+07
nginx_server_bytes{direction="out",host="my.owncloud.host"} 6.8422731e+07
nginx_upstream_bytes{backend="172.22.0.7:8080",direction="in",upstream="my.owncloud.host"} 6.7403731e+07
nginx_upstream_bytes{backend="172.22.0.7:8080",direction="out",upstream="my.owncloud.host"} 6.8422731e+07

• I note that there are float64 numbers formatted in the scientific notation, wondering if that could be an issue when reading the value into Grafana
• I have cross checked the numbers, and they seem to add up when comparing with http://nginx-vts-status/status/format/json

My nginx-vts-exporter on docker-compose.yml:

    nginx-vts-exporter:
    container_name: nginx-vts-exporter
    image: sophos/nginx-vts-exporter
    restart: always
    networks:
      - backend
      - frontend
    environment:
      - TZ=Europe/Oslo
      - NGINX_STATUS=http://nginx-vts-status/status/format/json

From prometheus.yml scrape_configs (scrape interval 5 seconds):

    - job_name: 'nginx-vts-exporter:'
      static_configs:
        - targets: ['nginx-vts-exporter:9913']

The prometheus data source seems to be working well in Grafana; the node exporter f.ex. seems to report correctly.

Best Regards,
Remi

why I got so little metrics

[root@node1 export]# curl http://10.96.174.195:9913/metrics

HELP nginx_vts_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which nginx_vts_exporter was built.

TYPE nginx_vts_exporter_build_info gauge

nginx_vts_exporter_build_info{branch="HEAD",goversion="go1.9.4",revision="43b45563691a369b8fa12f7383d185a6b28d18ea",version="0.10.3"} 1

json type error??

I added the arg "-nginx.scrape_uri http://127.0.0.1/nginx-status/format/json ".
but get no json:"""
Active connections: 2
server accepts handled requests
18 18 72
Reading: 0 Writing: 1 Waiting: 1
"""
which is exactly same as I use "-nginx.scrape_uri http://127.0.0.1/nginx-status/"
so the exporter log "2018/08/08 11:00:46 json.Unmarshal failed invalid character 'A' looking for beginning of value
"
why???
thanks

exporter does not work for mulitple nginx instances

Hi,

We have 2 nginx instances (with nginx-vts module installed) setup to serve the traffic.

We have setup one nginx-vts-exporter and configured it to scrape the metrics from "/status/format/json" endpoint.

Observation: metrics exposed from "/metrics" endpoint of nginx-vts-exporter doesn't include the information about the nginx instance from which the information was grabbed. Hence we are not able to plot the metrics against the nginx instance say in grafana/prometheus.

Does nginx-vts-exporter support to track metrics from multiple nginx instances? or Do we need to configure it differently to support multiple nginx instances setup?

VTS version: 0.1.14
nginx version: 1.12.0

for single instance of nginx we are getting proper data but for multiple it is causing issues.
Please help us asap!

cache size

Hi,

It's nice to be supported cache size (capacity, used, sent).

Upstream requests per backend

Hi,

currently the nginx-vts-exporter supports Server requests (1xx, 2xx etc.) and Upstream requests (1xx, 2xx etc.), but what would also be neat is feature already provided by vts plugin -- Upstream requests per backend. Backends are already working for nginx_upstream_requestMsec and nginx_upstream_responseMsec so, I'm assuming here, I think this should not be super difficult to implement.

Current state:

nginx_upstream_requests{code="1xx",instance="$INSTANCE",job="$JOB",upstream="$UPSTREAM"}

Feature request state:

nginx_upstream_requests{backend="$BACKEND",code="1xx",instance="$INSTANCE",job="$JOB",upstream="$UPSTREAM"}

Please let me know whether there's anything not clear about the issue.

All the best
Filip

refactor namespace

use global variable instead of pass args...

Add basic prometheus alerts

Some basic alerts to help others like:
increase of 4xx errors
Increase of waiting connections.
spikes on requests. ( could be an attack)

This is just some ideas, there is probably more that make sense!

If you have any, please add it here or do a PR.

I probably will work on something soon with this, so I will have some alerts to add it, but if someone already have soem it would be great!

Thanks

Collected before with the same name and label values

Hi,thanks for your exporter,it is very useful!
Recently,I think I have a few problems.
When I started the nginx_exporter,there were few errors:

then I curled http://localhost/nginx_status/format/json and I found that there were two servers in the esadmin upstream:

"esadmin":[{"server":"10.30.46.147:9100","requestCounter":0,"inByt
es":0,"outBytes":0,"responses":{"1xx":0,"2xx":0,"3xx":0,"4xx":0,"5xx":0},"requestMsecCounter":0,"requestMsec":0,"requestMsecs":{"times":[],"msecs":[]},"requestBuckets":{"msecs":[],"counters":[]},"responseMsecCounter":0,"responseMsec":0,"responseMsecs":{"times":[],"msecs":[]},"responseBuckets":{"msecs":[],"counters":[]},"weight":1,"maxFails":1,"failTimeout":10,"backup":false,"down":false,"overCounts":{"maxIntegerSize":18446744073709551615,"requestCounter":0,"inBytes":0,"outBytes":0,"1xx":0,"2xx":0,"3xx":0,"4xx":0,"5xx":0,"requestMsecCounter":0,"responseMsecCounter":0}},{"server":"10.30.46.147:9100","requestCounter":0,"inBytes":0,"outBytes":0,"responses":{"1xx":0,"2xx":0,"3xx":0,"4xx":0,"5xx":0},"requestMsecCounter":0,"requestMsec":0,"requestMsecs":{"times":[],"msecs":[]}

but there was only one in my profile:

So...what's my problem?
Thanks for any reply!

Commit "Support for filter zones (e40ee9)" broke nginx_upstream_xxx metrics

Was deleted merics nginx_upstream_requests and nginx_upstream_bytes from UpstreamZones

Payload metrics for each request

Hi,

Is it possible to get the metrics for payload size of each request hitting the nginx server using nginx-vts-exporter?

Thanks,
Sushant

Problems running in Kubernetes

I added the exporter to our Kubernetes Cluster at the nginx Controller.
At the metrics page i get the following errors

4 error(s) occurred:
* collected metric nginx_upstream_response label:<name:"backend" value:"xxx.xxx.xxx.xxx:80" > label:<name:"upstream" value:"xxx" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric nginx_upstream_response label:<name:"backend" value:"xxx.xxx.xxx.xxx:80" > label:<name:"upstream" value:"xxx" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric nginx_upstream_response label:<name:"backend" value:"xxx.xxx.xxx.xxx:80" > label:<name:"upstream" value:"xxx" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric nginx_upstream_response label:<name:"backend" value:"xxx.xxx.xxx.xxx:80" > label:<name:"upstream" value:"xxx" > gauge:<value:0 >  was collected before with the same name and label values

I have 4 endpoints behind this nginx.
I use the latest docker image. Do you have any suggestion for me to fix that errors?

Release v0.10.4 has no assets

Hi,
can you take a look at the latest release? It only contains the sourcecode, the binary assets are missing

Can't create binary

hi,

I'm no go expert but I followed your instructions and it does not build:

`[root@localhost ~]# mkdir go
[root@localhost ~]# cd go
[root@localhost go]# git clone https://github.com/hnlq715/nginx-vts-exporter.git
Klone nach 'nginx-vts-exporter'...
remote: Counting objects: 511, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 511 (delta 0), reused 2 (delta 0), pack-reused 507
Empfange Objekte: 100% (511/511), 18.18 MiB | 585.00 KiB/s, done.
Löse Unterschiede auf: 100% (237/237), done.
[root@localhost go]# cd nginx-vts-exporter/
[root@localhost nginx-vts-exporter]# make

formatting code
can't load package: package /root/go/nginx-vts-exporter: cannot find package "/root/go/nginx-vts-exporter" in any of:
/usr/lib/golang/src//root/go/nginx-vts-exporter (from $GOROOT)
/root/go/src//root/go/nginx-vts-exporter (from $GOPATH)
make: *** [format] Fehler 1
[root@localhost nginx-vts-exporter]#
`

Calculation bug in requests per backend

I think that PR #60 introduced a bug, the numbers do not add up:

They cumulate up to the total:

nginx_upstream_requests{backend="10.216.0.11:32154",code="total",redacted} 15110
nginx_upstream_requests{backend="10.216.0.12:32154",code="total",redacted} 30210
nginx_upstream_requests{backend="10.216.0.15:32154",code="total",redacted} 45306
nginx_upstream_requests{backend="10.216.0.16:32154",code="total",redacted} 60400
nginx_upstream_requests{backend="10.216.0.17:32154",code="total",redacted} 75493
nginx_upstream_requests{backend="10.216.0.18:32154",code="total",redacted} 90584
nginx_upstream_requests{backend="10.216.0.19:32154",code="total",redacted} 105674
nginx_upstream_requests{backend="10.216.0.21:32154",code="total",redacted} 120764
nginx_upstream_requests{backend="10.216.0.22:32154",code="total",redacted} 135862

nginx_server_requests{code="total",host="redacted"} 135987

can u provide a complete metrics for me ?

Hi, hnlq715. Nginx has many metrics, but i only get 3 useful metrics :nginx_server_bytes、nginx_server_cache、nginx_server_connections. Is there some metrics i missed ? like nginx_uptime

Choice to provide multiple URIs from several Nginx servers

Hey,

More like a feature request. It would be nice if I could specify several URIs (from several separate Nginx servers). To spin up dozen of these exporters for metrics does not seem right... :)

Problem with vts upstream module

Hi:

This exporter only work with vts module, but what about nginx vts stream module (https://github.com/vozlt/nginx-module-stream-sts)???

Only expose this metrics when scraping with the above module mentioned:

 HELP nginx_server_connections nginx connections
# TYPE nginx_server_connections gauge
nginx_server_connections{status="accepted"} 1
nginx_server_connections{status="active"} 1
nginx_server_connections{status="handled"} 1
nginx_server_connections{status="reading"} 0
nginx_server_connections{status="requests"} 29
nginx_server_connections{status="waiting"} 0
nginx_server_connections{status="writing"} 1
# HELP nginx_vts_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which nginx_vts_exporter was built.
# TYPE nginx_vts_exporter_build_info gauge
nginx_vts_exporter_build_info{branch="HEAD",goversion="go1.8.3",revision="82382edc1742c7308719c9f5050fba21fb62f87c",version="0.8.3"} 1

What about my upstream??

Running nginx-vts-exporter and got error:

./nginx-vts-exporter &

[1] 14394
[root@office-monitoring nginx-vts-exporter-0.6.linux-amd64]# 2017/09/05 08:58:48 Starting nginx_vts_exporter (version=0.6, branch=HEAD, revision=e48bd6899d8864b11b9f71f852b049ec3ad2ae26)
2017/09/05 08:58:48 Build context (go=go1.8.3, user=root@3b0fc1d0acb6, date=20170808-07:09:08)
2017/09/05 08:58:48 Starting Server at : :9913
2017/09/05 08:58:48 Metrics endpoint: /metrics
2017/09/05 08:58:48 Metrics namespace: nginx
2017/09/05 08:58:48 Scraping information from : http://localhost/status
2017/09/05 08:58:51 json.Unmarshal failed invalid character '<' looking for beginning of value
2017/09/05 08:59:06 json.Unmarshal failed invalid character '<' looking for beginning of value
2017/09/05 08:59:21 json.Unmarshal failed invalid character '<' looking for beginning of value
2017/09/05 08:59:36 json.Unmarshal failed invalid character '<' looking for beginning of value
2017/09/05 08:59:51 json.Unmarshal failed invalid character '<' looking for beginning of value
2017/09/05 09:00:06 json.Unmarshal failed invalid character '<' looking for beginning of value
2017/09/05 09:00:21 json.Unmarshal failed invalid character '<' looking for beginning of value
2017/09/05 09:00:36 json.Unmarshal failed invalid character '<' looking for beginning of value

Do you konw how to resolving the issue as above?

Thank you!

Build docker image as `latest` and `vx.x`

It would be great if we can access a more deterministic way to the latest version of this great exporter.

To access the latest changes the only way is using your docker image with the latest tag with the risk that we will pull the following release without knowing it and it might contain breaking changes.

This just happened to us 03e03cf

`nginx_filter_xxx` metrics are missing in `prometheus` graph console

I have:

• activated nginx_module_vts (see config below)
• configured and started nginx_vts_exporter
• configured prometheus to scrape metrics from nginx_vts_exporter

I'm able to see nginx_server_xxx and nginx_upstream_xxx metrics in prometheus graph console. However, I can't see nginx_filter_xxx metrics.

Also, filterZones property is present with full data in response to /status/format/json.

What I'm missing to get nginx_filter_xxx metrics in prometheus?

Below is the config of nginx_module_vts:

http {
  vhost_traffic_status_zone;
  vhost_traffic_status_filter_by_set_key $uri uri::$server_name;
  ...
}

location /status {
  vhost_traffic_status_display;
  vhost_traffic_status_display_format json;
}

Example of /status/format/json output:

{
	"hostName": "ctrl.localdomain",
	...
	"connections": {
		"active": 4,
		...
	},
	"sharedZones": {
		"name": "ngx_http_vhost_traffic_status",
		...
	},
	"serverZones": {
		"nginx-vts.localhost": {
			"requestCounter": 5,
			...
		},
		"php-fpm.localhost": {
			"requestCounter": 6,
			...
		},
		...
		"*": {
			"requestCounter": 18,
			...
		}
	},
	"filterZones": {
		"uri::nginx-vts.localhost": {
			"/status/format/json": {
				"requestCounter": 5,
				...
			}
		},
		"uri::php-fpm.localhost": {
			"/status": {
				"requestCounter": 6,
				...
			}
		},
		...
	},
	"upstreamZones": {
		"::nogroups": [{
			"server": "127.0.0.1:9000",
			"requestCounter": 6,
			...
			"inBytes": 624,
		}, {
			"server": "127.0.0.1:49090",
			"requestCounter": 7,
			...
		}]
	}
}

only seeing limited amount of metrics

Hi There,

Trying this out....

I can hit http://localhost:8080/status OK and loads a lot of json as per what I see on the status.html

When I hit the metrics end point - All I see is the following;

HELP nginx_server_connections nginx connections

TYPE nginx_server_connections gauge

nginx_server_connections{status="accepted"} 2.3859827e+07
nginx_server_connections{status="active"} 23
nginx_server_connections{status="handled"} 0
nginx_server_connections{status="reading"} 0
nginx_server_connections{status="requests"} 0
nginx_server_connections{status="waiting"} 0
nginx_server_connections{status="writing"} 0

HELP nginx_server_info nginx info

TYPE nginx_server_info gauge

nginx_server_info{hostName="",nginxVersion=""} 0

HELP nginx_vts_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which nginx_vts_exporter was built.

TYPE nginx_vts_exporter_build_info gauge

nginx_vts_exporter_build_info{branch="HEAD",goversion="go1.10",revision="8aa2881c7050d9b28f2312d7ce99d93458611d04

Running the latest exporter release and nginx plus 1.2.2

What am I missing here?

Cheers...

Pete

Cannot scrape URI on a different port

Hi there, thanks for a very useful project. I noticed an issue running this alongside NGINX with VTS.

We expose the VTS status reports on a virtual server running on port 8080, with IP whitelisting to ensure they can only be accessed from localhost, for security. This is using the following NGINX config (vhost_traffic_status_zone is specified elsewhere):

    server {
        listen      8080;
        server_name localhost;

        # Allow localhost (for health checks)
        allow 127.0.0.1;
        deny all;

        # Expose VTS metrics (only on localhost for security)

        location = /vts/format/html {
            # Exposes vhost traffic status as a HTML report
            vhost_traffic_status_display;
            vhost_traffic_status_display_format html;
        }
        location = /vts/format/json {
            # Exposes vhost traffic status as a JSON report (the HTML format
            # relies on this URL to retrieve data)
            # vts exporter also scrapes from this URL
            vhost_traffic_status_display;
            vhost_traffic_status_display_format json;
        }
    }

We then run the VTS exporter via the Docker Hub image (digest 8d14166508a5b19ccfd8d75b8933c52f327c7ba5fe728267cd6dc2eb869fd049), specifying NGINX_STATUS=http://localhost:8080/vts/format/json. This URL works fine with curl, but when trying to retrieve metrics, the exporter logs fetchHTTP failed parse "http://localhost:8080/vts/format/json": first path segment in URL cannot contain colon.

This is potentially to do with a change to URL parsing in Go 1.8, although I've not investigated fully: golang/go#19297

Our workaround for now is just to move the vts endpoints to a virtual server listening on port 80, but it'd be really useful to be able to use a different port. Thanks!

ngx_http_status_module was superseded by the ngx_http_api_module from nginx 1.13.0

Is there a plan to provide a sperated release for ngx_http_api_module?
http://nginx.org/en/docs/http/ngx_http_api_module.html

sorry. I see the exporter is base on nginx-module-vts, not stauts_module.

json.Unmarshal failed

Scraping information, i got log error: " json.Unmarshal failed invalid character '<' looking for beginning of value"

How to use with Graphana

1. I installed Prometheus and Graphana.
2. Configured Prometheus Data Source in Graphana and setup prometheus Dashboard. This works.
3. I ran the nginx-vts-exporter docker container with correct NGINX_STATUS url that returns json response. Following is the output.

NKS-AER:~ narendra$ docker run -ti --rm --env NGINX_STATUS="http://myhost.com/status/format/json" sophos/nginx-vts-exporter
2017/09/15 00:09:18 Starting nginx_vts_exporter (version=0.6, branch=HEAD, revision=ad91859ba2ee5b76887edb40c35f8991f0fa3df9)
2017/09/15 00:09:18 Build context (go=go1.8.3, user=travis@testing-gce-a196b404-c57f-47f5-b1bb-4cb5edab7d01, date=20170901-02:31:14)
2017/09/15 00:09:18 Starting Server at : :9913
2017/09/15 00:09:18 Metrics endpoint: /metrics
2017/09/15 00:09:18 Metrics namespace: nginx
2017/09/15 00:09:18 Scraping information from : http://myhost.com/status/format/json

4. I added this exporter in the Prometheus config as following:
   global:
   scrape_interval: 15s # By default, scrape targets every 15 seconds.

Attach these labels to any time series or alerts when communicating with

external systems (federation, remote storage, Alertmanager).

external_labels:
monitor: 'codelab-monitor'

A scrape configuration containing exactly one endpoint to scrape:

Here it's Prometheus itself.

scrape_configs:

The job name is added as a label job=<job_name> to any timeseries scraped from this config.

• job_name: 'prometheus'

  Override the global default and scrape targets from this job every 5 seconds.

  scrape_interval: 5s

  static_configs:

  • targets: ['localhost:9090']

• job_name: 'nginx'
  scrape_interval: 5s
  static_configs:

  • targets: ['localhost:9913']

5. I imported https://grafana.com/dashboards/2949 dashboard into my Graphana instance

I see the dashboard but no data on it. The output on the exporter console is not changing.

I believe the way this would work is as following:

1. Graphana UI will poll prometheus
2. Prometheus will keep polling every 5s the exporter
3. Exporter will fetch data from nginx whenever prometheus polls it.

Can someone guide me on this? I am new to both Graphana and Prometheus.

Problems runing exporter

hi,

I have installed a vm with a local prometheus install. I installed go and try to start the binary:

./nginx-vts-exporter -nginx.scrape_uri=http://ngnx-01.example.org/format/json -telemetry.endpoint=http://[::1]:9090/metrics

I get the output:

2017/09/13 20:19:44 Starting nginx_vts_exporter (version=0.6, branch=HEAD, revision=e48bd6899d8864b11b9f71f852b049ec3ad2ae26) 2017/09/13 20:19:44 Build context (go=go1.8.3, user=root@3b0fc1d0acb6, date=20170808-07:09:08) 2017/09/13 20:19:44 Starting Server at : :9913 2017/09/13 20:19:44 Metrics endpoint: http://[::1]:9090/metrics 2017/09/13 20:19:44 Metrics namespace: nginx 2017/09/13 20:19:44 Scraping information from : http://ngnx-01.example.org/format/json

an nothing happend. The ngn-01.example.org is reachable from the system, i have tested it with curl -I and a telnet ::1 9090 also works.

In the tcpdump on the system I don't see any traffic leaving the box. No error messages nothing...

Update README.md documentation references to nginx_upstream_requests --> nginx_upstream_requestsMsec

Documentation is not up to date. See this commit
03e03cf

Understanding the requests metric

I'm not sure if I get it or not so just wanted to ask to be sure.

Should the first prometheus query result in the same value as the second prometheus query?

sum(rate(nginx_server_requests{code="total", host="example.com"}[1h])) without (instance, kubernetes_pod_name)

sum(rate(nginx_server_requests{code!="total", host="example.com"}[1h])) without (instance, kubernetes_pod_name, code)

If they should be the same, I think there is an issue as I'm seeing a discrepency. I get the values below for these two queries.

1.2092047487499653

1.2089258466921395

nginx_upstream_response metric missing

Hello guys,

Am I missing something? Where is the main metric - nginx_upstream_response?
As grafana dashboard has it preconfigured
https://github.com/hnlq715/nginx-vts-exporter/blob/3b84c4ecab8ccb98dc59e6a8a00d76ab7d7b852f/dashboard/nginx-vts-exporter.json#L635
I may assume it was present earlier but now it is missing in the latest version (0.9.1), if this is expected can you advice the version where it is still present?

Resolver ignores /etc/hosts when running in Docker container

Hello!

Looks like Go resolver uses /etc/nsswitch.conf, but busybox base image doesn't include them:

# docker run  -ti --rm --env NGINX_STATUS="http://localhost/status/format/json" --entrypoint=sh sophos/nginx-vts-exporter
/ # cat /etc/nsswitch.conf
cat: can't open '/etc/nsswitch.conf': No such file or directory

In this case resolver uses DNS first and ever "localhost" resolves via external DNS.

It isn't expected behaviour because other busybox utilities uses /etc/host for resolving.

[Question] Using with multiple Virtual Hosts

hi!

Thanks for making such an awesome exporter available! I really want to utilize it, but I have a question first:

If I want to separate my metrics per virtual host/domain then would I need to run an instance/container per virtual host with differing NGINX_STATUS'?

For instance, if I have one nginx LB serving requests for a.blah.com and b.blah.com, do I have one exporter with NGINX_STATUS=http://a.blah.com/status/format/json and one with NGINX_STATUS=http://b.blah.com/status/format/json?

Thanks!

outBytes of type int

nginx_exporter 0.6 version

if nginx's outBytes > 2 147 483 648 ; than nginx_exporter error

errorlog:
2017/11/02 20:20:08 json.Unmarshal failed json: cannot unmarshal number 2251611814 into Go struct field Server.outBytes of type int
2017/11/02 20:20:23 json.Unmarshal failed json: cannot unmarshal number 2251765921 into Go struct field Server.outBytes of type int

[Question] Does this exporter support upstream in tcp stream?

Hi,

Nginx adds tcp stream support from 1.9, does this exporter support upstream in tcp stream? It would be wonderful if it is supported. Thanks.

Docker Container error

I'm starting Nginx up within kubernetes and using the container sophos/nginx-vts-exporter:latest, when kubernetes starts the pod and creates the exporter container it throws the following error:

tar: /bin/nginx-vts-exporter: not found in archive

Using sophos/nginx-vts-exporter:v0.4 has resulted in:

Connecting to github.com (192.30.253.112:443)
wget: can't execute 'ssl_helper': No such file or directory
wget: error getting response: Connection reset by peer

0.7.1 Packages

Any chance for 0.7.1 packages? I see 0.7.0 but not 0.7.1. Thanks

能不能支持单个http code 的请求量呀，

想区分 httpcode 400 404 499 500 502 等

Automated Docker Hub Build

Is there an automated Docker Hub build available? If not would it be possible to set one up.
I only saw this one https://hub.docker.com/r/albertdixon/nginx-vts-exporter/. But this doesn't seem to be an automated built image.
It would be awesome to have an automated build of this image.

---

One way would be to use a golang image and built it in it.

[ Question ]How to define the job in prometheus.yml?

Hi, I find there few doc about how to implement this in prometheus, below is my configuration
prometheus.yml

- job_name: 'nginx-exporter'
scrape_interval: 15s
static_configs:
- targets: ["http://172.17.15.101/status/format/json"]
relabel_configs:
- source_labels: [__address__]
regex: (^http://.)
target_label: __param_target
- source_labels: [__param_target]
target_label: host
- source_labels: []
regex: .
target_label: __address__
replacement: 172.17.15.101:9913

docker-compose.yml

nginx-exporter:
image: sophos/nginx-vts-exporter
ports:
- 9913:9913
environment:
NGINX_HOST: "http://172.17.15.101/status/format/json"

• Question 1. how to set monitor endpoint(s)?
• Question 2. how to implements this in prometheus.yml?

Thanks a ton....T T, wait for u help.

Getting "fetchHTTP failed HTTP status 401" error

Hi,

We are facing a strange issue with nginx-vts-exporter (v0.10.3). The exporter keeps flooding /var/log/messages with the following error:

#tail -n 10 /var/log/messages | grep nginx
nginx-vts-exporter: 2018/07/24 10:04:50 fetchHTTP failed HTTP status 401

In addition, it only exports the following metric:

#curl http://localhost:9913/metrics
# HELP nginx_vts_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which nginx_vts_exporter was built.
# TYPE nginx_vts_exporter_build_info gauge
nginx_vts_exporter_build_info{branch="HEAD",goversion="go1.10",revision="8aa2881c7050d9b28f2312d7ce99d93458611d04",version="0.10.3"} 1

We are using the default configuration (no environment variables or command line parameters), meaning it should be scraping http://localhost/status URL. The interesting part is that trying to CURL this URL from the server works perfectly:

# curl -sSL -D - http://localhost/status -o /dev/null
HTTP/1.1 200 OK
Server: nginx/1.14.0
Date: Tue, 24 Jul 2018 08:07:32 GMT
Content-Type: application/json
Content-Length: 50803
Connection: keep-alive

Some of the Nginx VHosts are indeed protected by basic auth, which could possibly result in 401, but localhost should not require the auth (which is confirmed by the CURL result above).

One more thing worth mentioning is that restarting nginx-vts-exporter solves the problem for some time (it starts getting the metrics properly). Anyhow, after some time it breaks again.

Any ideas what might be happening here?

Binaries for latest release

Hi, it seems that the binaries are not available for the latest release (0.7.1 and 0.8).
Is that intentional?