slok/go-http-metrics

Go modular http middleware to measure HTTP requests independent of metrics backend (with Prometheus and OpenCensus as backend implementations) and http framework/library

go-http-metrics

go-http-metrics knows how to measure http metrics in different metric formats and Go HTTP framework/libs. The metrics measured are based on RED and/or Four golden signals, follow standards and try to be measured in a efficient way.

Table of contents

• Metrics
• Metrics recorder implementations
• Framework compatibility middlewares
• Getting Started
• Prometheus query examples
• Options
  • Middleware Options
  • Prometheus recorder options
  • OpenCensus recorder options
• Benchmarks

Metrics

The metrics obtained with this middleware are the most important ones for a HTTP service.

• Records the duration of the requests(with: code, handler, method).
• Records the count of the requests(with: code, handler, method).
• Records the size of the responses(with: code, handler, method).
• Records the number requests being handled concurrently at a given time a.k.a inflight requests (with: handler).

Metrics recorder implementations

go-http-metrics is easy to extend to different metric backends by implementing metrics.Recorder interface.

• Prometheus
• OpenCensus

Framework compatibility middlewares

The middleware is mainly focused to be compatible with Go std library using http.Handler, but it comes with helpers to get middlewares for other frameworks or libraries.

When go-http-metrics is imported as a dependency, it will only import the libraries being used, this is safe because each lib/framework is in its own package. More information here and here

• Alice
• Chi
• Echo
• Fasthttp
• Gin
• Go http.Handler
• Go-restful
• Goji
• Gorilla
• Httprouter
• Iris
• Negroni

It supports any framework that supports http.Handler provider type middleware func(http.Handler) http.Handler (e.g Chi, Alice, Gorilla...). Use std.HandlerProvider

Getting Started

A simple example that uses Prometheus as the recorder with the standard Go handler.

package main

import (
    "log"
    "net/http"

    "github.com/prometheus/client_golang/prometheus/promhttp"
    metrics "github.com/slok/go-http-metrics/metrics/prometheus"
    "github.com/slok/go-http-metrics/middleware"
    middlewarestd "github.com/slok/go-http-metrics/middleware/std"
)

func main() {
    // Create our middleware.
    mdlw := middleware.New(middleware.Config{
        Recorder: metrics.NewRecorder(metrics.Config{}),
    })

    // Our handler.
    h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
        w.Write([]byte("hello world!"))
    })
    h = middlewarestd.Handler("", mdlw, h)

    // Serve metrics.
    log.Printf("serving metrics at: %s", ":9090")
    go http.ListenAndServe(":9090", promhttp.Handler())

    // Serve our handler.
    log.Printf("listening at: %s", ":8080")
    if err := http.ListenAndServe(":8080", h); err != nil {
        log.Panicf("error while serving: %s", err)
    }
}

For more examples check the examples. default and custom are the examples for Go net/http std library users.

Prometheus query examples

Get the request rate by handler:

sum(
    rate(http_request_duration_seconds_count[30s])
) by (handler)

Get the request error rate:

rate(http_request_duration_seconds_count{code=~"5.."}[30s])

Get percentile 99 of the whole service:

histogram_quantile(0.99,
    rate(http_request_duration_seconds_bucket[5m]))

Get percentile 90 of each handler:

histogram_quantile(0.9,
    sum(
        rate(http_request_duration_seconds_bucket[10m])
    ) by (handler, le)
)

Options

Middleware Options

The factory options are the ones that are passed in the moment of creating the middleware factory using the middleware.Config object.

Recorder

This is the implementation of the metrics backend, by default it's a dummy recorder.

Service

This is an optional argument that can be used to set a specific service on all the middleware metrics, this is helpful when the service uses multiple middlewares on the same app, for example for the HTTP api server and the metrics server. This also gives the ability to use the same recorder with different middlewares.

GroupedStatus

Storing all the status codes could increase the cardinality of the metrics, usually this is not a common case because the used status codes by a service are not too much and are finite, but some services use a lot of different status codes, grouping the status on the \dxx form could impact the performance (in a good way) of the queries on Prometheus (as they are already aggregated), on the other hand it losses detail. For example the metrics code code="401", code="404", code="403" with this enabled option would end being code="4xx" label. By default is disabled.

DisableMeasureSize

This setting will disable measuring the size of the responses. By default measuring the size is enabled.

DisableMeasureInflight

This settings will disable measuring the number of requests being handled concurrently by the handlers.

Custom handler ID

One of the options that you need to pass when wrapping the handler with the middleware is handlerID, this has 2 working ways.

• If an empty string is passed mdwr.Handler("", h) it will get the handler label from the url path. This will create very high cardnialty on the metrics because /p/123/dashboard/1, /p/123/dashboard/2 and /p/9821/dashboard/1 would have different handler labels. This method is only recomended when the URLs are fixed (not dynamic or don't have parameters on the path).

• If a predefined handler ID is passed, mdwr.Handler("/p/:userID/dashboard/:page", h) this will keep cardinality low because /p/123/dashboard/1, /p/123/dashboard/2 and /p/9821/dashboard/1 would have the same handler label on the metrics.

There are different parameters to set up your middleware factory, you can check everything on the docs and see the usage in the examples.

Prometheus recorder options

Prefix

This option will make exposed metrics have a {PREFIX}_ in fornt of the metric. For example if a regular exposed metric is http_request_duration_seconds_count and I use Prefix: batman my exposed metric will be batman_http_request_duration_seconds_count. By default this will be disabled or empty, but can be useful if all the metrics of the app are prefixed with the app name.

DurationBuckets

DurationBuckets are the buckets used for the request duration histogram metric, by default it will use Prometheus defaults, this is from 5ms to 10s, on a regular HTTP service this is very common and in most cases this default works perfect, but on some cases where the latency is very low or very high due the nature of the service, this could be changed to measure a different range of time. Example, from 500ms to 320s Buckets: []float64{.5, 1, 2.5, 5, 10, 20, 40, 80, 160, 320}. Is not adviced to use more than 10 buckets.

SizeBuckets

This works the same as the DurationBuckets but for the metric that measures the size of the responses. It's measured in bytes and by default goes from 1B to 1GB.

Registry

The Prometheus registry to use, by default it will use Prometheus global registry (the default one on Prometheus library).

Label names

The label names of the Prometheus metrics can be configured using HandlerIDLabel, StatusCodeLabel, MethodLabel...

OpenCensus recorder options

DurationBuckets

Same option as the Prometheus recorder.

SizeBuckets

Same option as the Prometheus recorder.

Label names

Same options as the Prometheus recorder.

UnregisterViewsBeforeRegister

This Option is used to unregister the Recorder views before are being registered, this is option is mainly due to the nature of OpenCensus implementation and the huge usage fo global state making impossible to run multiple tests. On regular usage of the library this setting is very rare that needs to be used.

go-http-metrics's Issues

Release a new version

@slok any chance we release a new version?

There have been many dependency bumps with security fixes that are not present in the latest release.

Possible to release new version?

There's been lots of great activity in the repository, but no new release for over a year. Would it be possible to release a new version? There have been security vulnerabilities in the echo, gin, and protobuf dependencies; it appears that the latest go.mod refers to the fixed versions, and it would be so helpful to close those security holes for users of the library

Support routes generated by grpc-gateway

Hello. Thank you for creating and maintaining this wonderful library. I've used it in a number of contexts and never encountered a situation I couldn't solve out-of-the-box using the available config options, or by implementing my own Reporter and handler if I needed non-standard behavior.

However, I recently wanted to collect metrics from the HTTP gateway of a gRPC service and realized that I had to fork Middleware in order to support the routes generated by grpc-gateway.
The discussion in this issue covers all of the relevant details if you feel like going into the weeds, but the gist of it is that the grpc-gateway muxer is a bit of a black box. It generates routes based on annotations to protobuf definitions and packages them all in a single HTTP handler that you can attach to a path prefix on your server (e.g. /api/*).

The result is that no middleware that sits before the grpc-gateway handler can know which route the incoming request is going to be dispatched to which, as far as metrics go, leaves us with two equally bad options:

1. Use the same handler ID for all the generated routes, which makes these metrics utterly useless. Or,
2. Use the full URL path, which results in an explosion of metrics in routes that include path parameters (e.g. /api/users/{user_id}/policies/{policy_id}) and again renders the results pretty much useless.

Fortunately, a solution was offered in this post. The idea is to provide hook function that is called by the grpc-gateway muxer after the route has been selected but before the request is actually served.

The problem is that the Measure function in Middleware, expects the handler ID to be known before calling the underlying handler, which doesn't work in this case.

The way I solved it locally is by forking Middleware and slightly modifying Measure to give reporters that didn't provide a handler ID before the call, a second chance to do it after.

The change is very small but that's all I needed to support this use case, which I believe many others may run into. Below is my modified implementation of Measure (the only change is in the deferred func). If it makes sense to you I'd be delighted to open a PR. I can also include my handler and hook function to help others collect grpc-gateway metrics without having to write their own Reporter.

Many thanks!

// Measure abstracts the HTTP handler implementation by only requesting a reporter, this
// reporter will return the required data to be measured.
// it accepts a next function that will be called as the wrapped logic before and after
// measurement actions.
func (m Middleware) Measure(handlerID string, reporter middleware.Reporter, next func()) {
	ctx := reporter.Context()

	// If there isn't predefined handler ID we
	// set that ID as the URL path.
	hid := handlerID
	if handlerID == "" {
		hid = reporter.URLPath()
	}

	// Measure inflights if required.
	if !m.cfg.DisableMeasureInflight && hid != "" {
		props := metrics.HTTPProperties{
			Service: m.cfg.Service,
			ID:      hid,
		}
		m.cfg.Recorder.AddInflightRequests(ctx, props, 1)
		defer m.cfg.Recorder.AddInflightRequests(ctx, props, -1)
	}

	// Start the timer and when finishing measure the duration.
	start := time.Now()
	defer func() {
		duration := time.Since(start)

		// If we need to group the status code, it uses the
		// first number of the status code because is the least
		// required identification way.
		var code string
		if m.cfg.GroupedStatus {
			code = fmt.Sprintf("%dxx", reporter.StatusCode()/100)
		} else {
			code = strconv.Itoa(reporter.StatusCode())
		}

                //************ Modification ***************
                // If reporter was unable to determine the path before calling the underlying handler, try again after.
                // This is the only departure from the original implementation.
		if hid == "" {
			hid = reporter.URLPath()
		}
               //************ End Modification ***************

		props := metrics.HTTPReqProperties{
			Service: m.cfg.Service,
			ID:      hid,
			Method:  reporter.Method(),
			Code:    code,
		}
		m.cfg.Recorder.ObserveHTTPRequestDuration(ctx, props, duration)

		// Measure size of response if required.
		if !m.cfg.DisableMeasureSize {
			m.cfg.Recorder.ObserveHTTPResponseSize(ctx, props, reporter.BytesWritten())
		}
	}()

	// Call the wrapped logic.
	next()
}

Additional labels

Just a request: would it be possible to add functionality that enables custom labels on the collected metrics?

I would like to classify requests by the customer id which is available in the header and/or context metadata so woud like to specify a function to get the value .

This is a newbie question as just trying to understand prom metrics collection...

I would be quite happy to submit a PR _ a modification to metrics,Config should do the trick I think.

no ServeHTTP method to conform to http.Handler chainability?

I'm trying to use this as an http.Handler wrapper for chi, a mux that accepts "standard" http.Handler wrapper go middlewares. It doesn't have a ServeHTTP method though, so it can't be chained in this way like my others, unless I'm misunderstanding something.

support the http.Hijacker interface in middleware.responseWriterInterceptor

If a another middleware try to hijack the connection, the wrapper won't forward the call to hijack. In my case the middleware tries to upgrade the connection to a websocket.

This is how it's implemented in Negroni :
https://github.com/urfave/negroni/blob/master/response_writer.go#L82

ResponseWriter optional interfaces

The current implementation of the standard lib implementation uses a wrapper of the response writer to capture status and other information. While it does conform to the interfaces required function it does not implement optional interfaces which are useful. As the this is run as a middleware before any handlers it means that all handlers will receive the wrapped response writer which as well.

This has the most noticeable impact when using io copy to write from a reader, as a zero copy system call will not be used. Instead Go will fallback to using an in memory buffer instead.

It is not the most logical side effect, and it is pretty easy to miss as the response writer is hidden and itself does not do a good job documenting the additional optional interfaces.

Here is a good blog post that explains the issue well.
https://avtok.com/2014/11/05/interface-upgrades.html

My suggestion is that we implement the ReadFrom function on the response writer so that io copy will work more efficiently for all users who may not be aware of the issue.

Enable scalable handlerID with GetPathTemplate support

Would it be possible to add an option to use GetPathTemplate in gorilla, or similar in other libraries instead of the exact url route when initialized with mdwr.Handler("", h). This will allow more scalable path tracking to generalize ID params and avoid high tag cardinality explosion.

If you want I can investigate making this change if you are willing to accept a PR.

CVE-2022-32149: Affecting golang.org/x/text v0.3.7

Indirect dependency, present in go.sum has a security issue:

https://ossindex.sonatype.org/vulnerability/CVE-2022-32149?component-type=golang&component-name=golang.org%2Fx%2Ftext&utm_source=nancy-client&utm_medium=integration&utm_content=0.0.0-dev

Wrapper for handlerID parsing

Hello there!
It'd be awesome to have extra param for new std.Handler method, so we could pass formatting function to get clear handlerID in `Measure.

Now it's painful, 'coz I need to extract request url via dirty function or wrap in one more time.

go http client example

Thanks for this great piece of work. I would like to know whether there is an example on how to use it with the http client.

Repo still maintained?

@slok is this repo still maintained? Seems like there has been no commit to master for 11 months and no release for 2 years. Really find this repo useful but we may have to abandon using it if the security fixes aren't patched

gorilla mux path compatibility

Paths are not supported with std.HandlerProvider

http_request_duration_seconds_count{code="500",handler="/hello/123",method="GET",service=""} 1
http_request_duration_seconds_count{code="500",handler="/hello/321",method="GET",service=""} 1

Should be:

http_request_duration_seconds_count{code="500",handler="/hello/{key}",method="GET",service=""} 2

Code example:

r := mux.NewRouter()

r.Use(std.HandlerProvider("", metrics_middleware.New(metrics_middleware.Config{
	Recorder: metrics.NewRecorder(metrics.Config{}),
})))

r.HandleFunc("/hello/{key}", hello).Methods("GET")

Error on "go get -u github.com/slok/go-http-metrics"

I get an error while trying to install the package.

go get -u github.com/slok/go-http-metrics

causes the following error:

go: finding github.com/golang/lint latest
go: github.com/golang/[email protected]: parsing go.mod: unexpected module path "golang.org/x/lint"
go: finding github.com/streadway/amqp latest
go get: error loading module requirements

It seems to be related to this issue

CWE 601: Affecting github.com/urfave/negroni package, versions >=0.0.0

Found in dependency

github.com/urfave/negroni v1.0.0

Issue: https://security.snyk.io/vuln/SNYK-GOLANG-GITHUBCOMURFAVENEGRONI-1658297

Middleware per route with the "chi" library

Hi @slok ,

When using http directly I do see how to set a specific handlerID for each route:
https://github.com/slok/go-http-metrics/blob/master/examples/custom/main.go#L54-L57

But in the chi example there is none:
https://github.com/slok/go-http-metrics/blob/master/examples/chi/main.go

Since your library seems not able take as entry a HandleFunc to return a HandleFunc, I guess I have to manage it with a more complex way with chi. Something like:

	r := chi.NewRouter()

	recorder := metrics.NewRecorder(metrics.Config{
		Registry:        reg,
		Prefix:          "exampleapp",
		DurationBuckets: []float64{1, 2.5, 5, 10, 20, 40, 80, 160, 320, 640},
	})
	mdlw := middleware.New(middleware.Config{
		Recorder:      recorder,
		GroupedStatus: true,
	})

	// Specific route
	r.Group(func(r chi.Router) {
		r.Use(std.HandlerProvider("my_specific_route", mdlw))

		r.Post("/", aaaaaaa)
	})

	return r

Is there a more simple way than embedding all my registered routes with Post / Get / ... inside individual groups?

Thank you,

Grafana Dashboard

Thanks for this great package.
I'm just wondering if someone has created a Grafana Dashboard in the community that they can share?
Many thanks.

Feature: Add request_total metric

Hello, we are missing a total request counter. As recommended in RED or golden signals, the error rate monitoring is essential.

In here we would add something like this:

r := &recorder{
	httpRequestTotal: prometheus.NewCounterVec(prometheus.CounterOpts{
		Namespace: cfg.Prefix,
		Subsystem: "http",
		Name:      "request_total",
		Help:      "The number of total HTTP requests.",
	}, []string{cfg.ServiceLabel, cfg.HandlerIDLabel, cfg.MethodLabel, cfg.StatusCodeLabel}),
        ...
}

A possible alert using this metric:

- alert: HttpHigh5xxErrorRate 
  expr: sum(rate(http_requests_total{status=~"5.*"}[5m])) / sum(rate(http_requests_total[5m])) * 100 > 5
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: Http high 5xx error rate
    description: "Too many HTTP requests with status 5xx (> 5%) on the server."

I'm happy to create a PR for this feature if you agree. Thank you!

CVE-2022-40083 affecting echo 4.7.2 version

URL Redirection to Untrusted
Site ('Open Redirect')
-->avd.aquasec.com/nvd/cve-2022-40083

Refactor to "plugin" model

This package looks super useful, but I wound up not using it because it has a bunch of deps on several different HTTP frameworks, which significantly increased the size of our go.sum.

It would be nicer to have a "plugin" model where you go get each framework-specific plugin separately, to avoid dependency bloat. (IMO the built-in "net/http" plugin could be included in this repo but the other 3rd-party frameworks should be separate repos).

Security Vulnerability: gopkg.in/yaml.v3

gopkg.in/yaml.v3 is a YAML support package for the Go language.

Affected versions of this package are vulnerable to NULL Pointer Dereference when parsing #\n-\n-\n0 via the parserc.go parser.

v0.10.0 is still on an outdated version: https://github.com/slok/go-http-metrics/blob/v0.10.0/go.mod#L59

Set ignore path

• I think there is no way to ignore specific paths.
  • Ex, health check and prometheus scrap endpoint
• What do you think that add configuration option for ignoring specific path.

panic: duplicate metrics collector registration attempted (in tests)

I'm running this middleware in my tests, that spawn multiple servers and it causes the following panic:

panic: duplicate metrics collector registration attempted [recovered]
panic: duplicate metrics collector registration attempted

goroutine 251 [running]:
testing.tRunner.func1.1(0x4b3ba20, 0xc0007ae8c0)
/usr/local/Cellar/go/1.15.2/libexec/src/testing/testing.go:1076 +0x30d
testing.tRunner.func1(0xc000daa300)
/usr/local/Cellar/go/1.15.2/libexec/src/testing/testing.go:1079 +0x41a
panic(0x4b3ba20, 0xc0007ae8c0)
/usr/local/Cellar/go/1.15.2/libexec/src/runtime/panic.go:969 +0x175
github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc000292e10, 0xc000a7d440, 0x3, 0x3)
/Users/achaplianka/Dvelop/Prometheus/k-api/vendor/github.com/prometheus/client_golang/prometheus/registry.go:400 +0xad
github.com/slok/go-http-metrics/metrics/prometheus.NewRecorder(0x0, 0x0, 0x521f440, 0xb, 0xb, 0xc000e35400, 0x8, 0x8, 0x4d2b3e0, 0xc000292e10, ...)

example with gin does not work properly

with the response type as json, it produces two items in output.

InstrumentHandlerResponseSize

Hello guys,

Is it possible to implement InstrumentHandlerResponseSize?
To capture the amount of bytes we will send in the response?