ttys3/axum-otel-metrics

axum OpenTelemetry Metrics middleware with prometheus exporter

axum-otel-metrics

axum OpenTelemetry metrics middleware with prometheus exporter

follow Semantic Conventions for HTTP Metrics

axum is an ergonomic and modular web framework built with Tokio, Tower, and Hyper

be default, the metrics will be exported at /metrics endpoint. and below metrics will be exported:

requests_total counter

requests_total

http_server_active_requests gauge

The number of active HTTP requests

http_server_request_duration_seconds histogram

http_server_request_duration_seconds_bucket
http_server_request_duration_seconds_sum
http_server_request_duration_seconds_count

http_server_request_size_bytes histogram

http_server_request_size_bytes_bucket
http_server_request_size_bytes_sum
http_server_request_size_bytes_count

http_server_response_size_bytes_ histogram

http_server_response_size_bytes__bucket
http_server_response_size_bytes__sum
http_server_response_size_bytes__count

labels for requests_total, http_server_request_duration_seconds, http_server_request_size_bytes, http_server_response_size_bytes :

http_request_method
http_route
http_response_status_code
server_address

labels for http_server_active_requests :

http_request_method
url_scheme

Usage

use axum_otel_metrics::HttpMetricsLayerBuilder;

let metrics = HttpMetricsLayerBuilder::new()
    .build();

let app = Router::new()
    // export metrics at `/metrics` endpoint
    .merge(metrics.routes())
    .route("/", get(handler))
    .route("/hello", get(handler))
    .route("/world", get(handler))
    // add the metrics middleware
    .layer(metrics);

Usage with State

use axum_otel_metrics::HttpMetricsLayerBuilder;

#[derive(Clone)]
pub struct SharedState {
}

let state = SharedState {
};

let metrics = HttpMetricsLayerBuilder::new()
    .build();

let app = Router::new()
    // export metrics at `/metrics` endpoint
    .merge(metrics.routes::<SharedState>())
    .route("/", get(handler))
    .route("/hello", get(handler))
    .route("/world", get(handler))
    // add the metrics middleware
    .layer(metrics)
    .with_state(state.clone());

OpenTelemetry Rust Instrumentation Status and Releases

https://opentelemetry.io/docs/instrumentation/rust/#status-and-releases

Traces	Metrics	Logs
Beta	Alpha	Alpha

OpenTelemetry Metrics Exporter

Push Metric Exporter https://opentelemetry.io/docs/reference/specification/metrics/sdk/#push-metric-exporter

Pull Metric Exporter https://opentelemetry.io/docs/reference/specification/metrics/sdk/#pull-metric-exporter

exporters

https://opentelemetry.io/docs/reference/specification/metrics/sdk_exporters/

In-memory https://opentelemetry.io/docs/reference/specification/metrics/sdk_exporters/in-memory/

Prometheus https://opentelemetry.io/docs/reference/specification/metrics/sdk_exporters/prometheus/

OTLP https://opentelemetry.io/docs/reference/specification/metrics/sdk_exporters/otlp/

Standard output https://opentelemetry.io/docs/reference/specification/metrics/sdk_exporters/stdout/

Metrics Data Model

https://opentelemetry.io/docs/reference/specification/metrics/data-model/

Related Projects

https://github.com/nlopes/actix-web-prom

Actix-web middleware to expose Prometheus metrics

https://github.com/sd2k/rocket_prometheus

Prometheus fairing and handler for Rocket

https://github.com/Ptrskay3/axum-prometheus

axum-prometheus relies on metrics.rs and its ecosystem to collect and export metrics - for instance for Prometheus, metrics_exporter_prometheus is used as a backend to interact with Prometheus.

axum-otel-metrics's Issues

with_skipper impossible to use in practice

In practice, anything that can be dynamic cannot be used in with_skipper on the HttpMetricsBuilderLayer, because the function that is supposed to be doing the skipping cannot be defined as an fn since it captures local state, and it cannot be defined as a closure either since the closure cannot be coerced to an fn since it captures local state.

How do we change the API so this works? Alternatively, is there something that I can do, as the caller of the code, to make this work?

How do I register custom metrics?

I've tried

lazy_static! {
    static ref CACHE_HIT_COUNTER: IntCounterVec = register_int_counter_vec!(
        "http_cache_hits_total",
        "Number of HTTP cache hits",
        &["http_response_status_code"]
    )
    .unwrap();
    static ref CACHE_MISS_COUNTER: IntCounterVec = register_int_counter_vec!(
        "http_cache_misses_total",
        "Number of HTTP cache misses",
        &["http_response_status_code"]
    )
    .unwrap();
}

but these metrics aren't getting registered in the registry that the HttpMetricsLayerBuilder creates and assigns to the HttpMetricsLayer my system uses.

How do I obtain a reference to the registry such that my code can use register_int_counter_vec_with_registry? Alternatively, how do I instruct the HttpMetricsLayerBuilder to use the default prometheus crate registry?

Thanks in advance.

prepare for axum 0.7

axum 0.7 will release soon. this will be a breaking change.

since we are at 0.x version, I think it is OK to bump minor version from v0.7.0 to v0.8.0

axum v0.6.x compatible middleware will dev under https://github.com/ttys3/axum-otel-metrics/tree/axum-0.6

the main branch goes for axum 0.7

• tokio-rs/axum#2354

add http_server_active_requests gauge meter

add http_server_active_requests gauge meter like

https://github.com/OutThereLabs/actix-web-opentelemetry/blob/main/src/middleware/metrics.rs#L28C22-L28C22

Decomposing the layer

Hey there :]
I'm seeing some great stuff in this library. Though I think it doesn't compose nicely at the moment.

I would like to add the metric tracking layer and the views, without the global setup and export route.

Currently the library does everything with one layer.

• Creating your global MeterProvider and Registry
• Adding views to the MeterProvider
• Set up a Meter and tracking axum metrics
• Adding a /metrics route to export on

Ideally we can pick-and-choose from this.

Rough outline of what that could be.

The views don't take any arguments, so could be a static method or it's own builder, say ViewBuilder::default().build()

The layer would exclusively measure, and it's builder could have an optional MeterProvider using the global one if None.

The export router would be a separate builder. Which could either take the current approach (make a new MeterProvider & Registry pair and set it as global) or be created from arguments passed in.

Is this project published under the Apache 2.0 license?

Hello! I would like to know what license this project is published under. Thanks in advance!

How can i add custom static (application) metrics?

I see there was a thread here #73 but no solution posted.
@DFINITYManu I know this was a long time ago, but what was your solution?

add actual tests

ref #76 (comment)

otel semconv v1.23.0 stable

https://opentelemetry.io/blog/2023/ecs-otel-semconv-convergence/

https://www.elastic.co/guide/en/ecs/master/ecs-server.html

https://opentelemetry.io/blog/2023/http-conventions-declared-stable/#http-server-duration-metric

v1.23.0 (stable).

Endpoint protection

I'm sorry if this isnt right format for this, but I was wondering there are plans to implement some endpoint protection? Exposing /metrics isn't going to be acceptable for my application, but I love this library.

Perhaps something similar to metrcs_exporter_prometheus?

This is a little verbose, but here's an excerpt from https://docs.rs/metrics-exporter-prometheus/latest/src/metrics_exporter_prometheus/builder.rs.html#203-214

/// Builder for creating and installing a Prometheus recorder/exporter.
pub struct PrometheusBuilder {
    #[cfg_attr(not(any(feature = "http-listener", feature = "push-gateway")), allow(dead_code))]
    exporter_config: ExporterConfig,
    #[cfg(feature = "http-listener")]
    allowed_addresses: Option<Vec<IpNet>>,
    quantiles: Vec<Quantile>,
    buckets: Option<Vec<f64>>,
    bucket_overrides: Option<HashMap<Matcher, Vec<f64>>>,
    idle_timeout: Option<Duration>,
    recency_mask: MetricKindMask,
    global_labels: Option<IndexMap<String, String>>,
}

    #[cfg(feature = "http-listener")]
    #[cfg_attr(docsrs, doc(cfg(feature = "http-listener")))]
    pub fn add_allowed_address<A>(mut self, address: A) -> Result<Self, BuildError>
    where
        A: AsRef<str>,
    {
        use std::str::FromStr;

        let address = IpNet::from_str(address.as_ref())
            .map_err(|e| BuildError::InvalidAllowlistAddress(e.to_string()))?;
        self.allowed_addresses.get_or_insert(vec![]).push(address);

        Ok(self)
    }

match exporter_config {
            ExporterConfig::Unconfigured => Err(BuildError::MissingExporterConfiguration),
            #[cfg(feature = "http-listener")]
            ExporterConfig::HttpListener { listen_address } => {
                let server = Server::try_bind(&listen_address)
                    .map_err(|e| BuildError::FailedToCreateHTTPListener(e.to_string()))?;
                let exporter = async move {
                    let make_svc = make_service_fn(move |socket: &AddrStream| {
                        let remote_addr = socket.remote_addr().ip();

                        // If the allowlist is empty, the request is allowed.  Otherwise, it must
                        // match one of the entries in the allowlist or it will be denied.
                        let is_allowed = allowed_addresses.as_ref().map_or(true, |addresses| {
                            addresses.iter().any(|address| address.contains(&remote_addr))
                        });

                        let handle = handle.clone();

                        async move {
                            Ok::<_, hyper::Error>(service_fn(move |_| {
                                let handle = handle.clone();

                                async move {
                                    if is_allowed {
                                        let output = handle.render();
                                        Ok::<_, hyper::Error>(Response::new(Body::from(output)))
                                    } else {
                                        Ok::<_, hyper::Error>(
                                            Response::builder()
                                                .status(StatusCode::FORBIDDEN)
                                                .body(Body::empty())
                                                .expect("static response is valid"),
                                        )
                                    }
                                }
                            }))
                        }
                    });
                    server.serve(make_svc).await
                };

                Ok((recorder, Box::pin(exporter)))
            }

doc: Which metrics are exposed?

Hi! I would like to see which metrics the crate does actually expose. Could you add this to the README?