micrometer-metrics/micrometer

An application observability facade for the most popular observability tools. Think SLF4J, but for observability.

micrometer's Issues

Add `@EnableMetrics`-like annotations and configuration for Boot 1.x and 2

Generalization of sanitizing tag values for different metrics backends

This currently affects URI template sanitization in DefaultWebMetricsTagProvider.

Provide Influx support

summary in PrometheusMeterRegistry.timer() does not respect tags

In PrometheusMeterRegistry.timer() the summary metric is looked up using only the metrics name.
around line 100 collectorMap.computeIfAbsent(name , ...).
This has the following effects:

• the labels for the summary are the labels of the first timer.
• timing of requests which cause an error (status >= 400) are conflated with timing of successful requests. Failing requests usually do not take as much time as working requests and therefore the statistic is skewed.
• timing of all requests, no matter how expensive they are are mashed together. They should be separated by the URL path (at least).

Is the name `spring-metrics` confusing?

I foresee two primary modules in this project. The Spring aspect that leverages actuator, etc makes sense being called spring-metrics, however the other aspect is similar to slf4j and really is just a metrics-facade.

I would love to be able to encourage all Java project to leverage the facade benefits, even if they aren't using Spring and I wouldn't want the name to drive them away.

Ideas:

• metrics-interface (maybe shortened to mi4j, but then it looks like leet speak 'miaj'), also is pretty generic,
• meterize - Memorable and unique, also leverages the 'meter' terminology well
• dendro - Out in left field, but 'dendrochronology' is the science of measuring tree growth via rings. (trees grows a lot during the 'Spring') ... yeah a stretch.

I realize that optional deps can keep the modules flexible, but it does feel like if the 2 concerns are made explicit it could save confusion in the long run.

I favor meterize, and spring-metrics, even though the names aren't parallel.

I don't think this is urgent, just wanted to make sure we had this conversation.

Support pass-through without aggregation?

Hey! I was wondering if you plan on supporting metrics that are not aggregated, but instead are sent directly to a daemon as they are recorded.

Background: In our setup, we use a dogstatsd like daemon that runs on every instance. It accepts metrics directly via UDP, aggregates them and then forwards to Datadog, as described here: http://docs.datadoghq.com/guides/dogstatsd/

In that case, the daemon already does aggregation. So if the library we use also does aggregation, we end up with metrics that are not as useful as they could be.

I see that the basic interfaces such as Timer would be generic enough to allow an implementation to forward as soon as the method is called. But there's also some parts of it that seem only useful for aggregation, such as totalTime.

As the library seems to not yet be in a released state, it might be a good point in time to consider this use case. Thanks!

(I'm currently looking at java-dogstatsd-client because it does what we want in that it directly sends to dogstatsd. But I'm also considering adding some bits on top such as support for @Timed.)

Provide statsd support

Any such implementation will have to be configured or somehow discover the particular statsd server it is interacting with so as to support the many specializations of the protocol (e.g. Datadog's).

Duplicate metrics

It seems that if many request came for the very first time that duplicate counters will be registered which leads to duplicate metrics in exporting and therefore overwriting metrics.

This will look like this.

http_server_requests{exception="None",instance="192.168.110.111",instanceType="local",job="MVG",method="DELETE",profile="dev",status="500",uri="v1/vehicles/{vin}",quantile="0.5"} 1.436569357
http_server_requests{exception="None",instance="192.168.110.111",instanceType="local",job="MVG",method="DELETE",profile="dev",status="500",uri="v1/vehicles/{vin}",quantile="0.95"} 2.810563785
http_server_requests{exception="None",instance="192.168.110.111",instanceType="local",job="MVG",method="DELETE",profile="dev",status="500",uri="v1/vehicles/{vin}",quantile="0.5"} NaN
http_server_requests{exception="None",instance="192.168.110.111",instanceType="local",job="MVG",method="DELETE",profile="dev",status="500",uri="v1/vehicles/{vin}",quantile="0.95"} NaN
http_server_requests{exception="None",instance="192.168.110.111",instanceType="local",job="MVG",method="DELETE",profile="dev",status="500",uri="v1/vehicles/{vin}",quantile="0.5"} NaN
http_server_requests{exception="None",instance="192.168.110.111",instanceType="local",job="MVG",method="DELETE",profile="dev",status="500",uri="v1/vehicles/{vin}",quantile="0.95"} NaN
http_server_requests{exception="None",instance="192.168.110.111",instanceType="local",job="MVG",method="DELETE",profile="dev",status="500",uri="v1/vehicles/{vin}",quantile="0.5"} NaN
http_server_requests{exception="None",instance="192.168.110.111",instanceType="local",job="MVG",method="DELETE",profile="dev",status="500",uri="v1/vehicles/{vin}",quantile="0.95"} NaN
http_server_requests_sum{exception="None",instance="192.168.110.111",instanceType="local",job="MVG",method="DELETE",profile="dev",status="500",uri="v1/vehicles/{vin}"} 0
http_server_requests_count{exception="None",instance="192.168.110.111",instanceType="local",job="MVG",method="DELETE",profile="dev",status="500",uri="v1/vehicles/{vin}"} 0

It seems that the dynamic creation of the endpoint metric should be synchronized over all threads?

Best regards,
Eloo

Blow up or warn when @Timed has no name on @Scheduled

Tiered metrics support

Support tiering metrics by priority to allow for different push intervals (the relatively fewer critical metrics are published at a shorter interval than less critical metrics).

Springboot App startup failed in integration tests

Hi,
i'm facing an issue when running multiple integration tests one after another.
It seems that some collectors are tried to be registers twice.
Collector already registered that provides name: jvm_buffer_count
Any idea how to get this fixed?

Thanks

LogBack metrics filter destroys logging configuration

Returning FilterReply.ACCEPT from Logback.Metrics.MetricTurboFilter.decide() makes
Logback to log everything (even TRACING) and does not obey logging confuiguration.

Support adding CustomMeters to MeterRegistry

This will parallel the CustomCollector concept in Prometheus.

Proposal:

• Add register(MeterReporter) to MetricRegistry
• MeterReporter's interface will return a list of metrics that will be converted to the underlying MeterRegistry's metrics. The important thing to keep in mind is that a MeterReporter will be able to report multiple meters.

Support interval counters

A counter that also keeps track of the time since last update (in the form of a gauge).

Add sample project for reference for Spring Metrics and Prometheus

I've created an initial sample project here https://github.com/checketts/spring-metrics-sample

Currently it doesn't do anything besides run.

Metrics I want to express:

• Requests
• Annotations on endpoints
• Caches
• Database
• Executors (with @Scheduled usage too)
• Logback
• Demonstrate manual adding counters, gauges, and timers.

Also I want to include a sample Prometheus and Grafana config with explanations for the various useful metrics.

`@Counter`, `@Gauge`, etc annotations on fields

Can we effectively annotate fields with a BeanPostProcessor. Why did Netflix move away from this model between Servo and Spectator?

Zabbix support

https://www.zabbix.com/

Is this something we want?

Finish support for Atlas

Dropwizard support

For 0.8.0, the plan is to extend support to Ganglia, Graphite, and JMX via a Micrometer Dropwizard registry.

Since this is the first hierarchical system supported, this implementation involves the introduction of a new HierarchicalNameMapper that allows you to generalize and fine tune the construction of hierarchical names from dimensional metrics ids.

Change `package` constructor to `public` on Simple* Meters

SimpleCounter, SimpleGauge, etc.. currently can't be reused in spring-metrics extensions, because their constructor is only package-visible.
I wrote simple extension on top of spring-metrics for use ClickHouse database as storage for metrics, and most of Meter classes is duplicates of Simple*Meter classes. I think some custom extensions may reuse code of Simple meters as is. Thanks!

Bridge from current Spring Boot Actuator metrics

Support for prometheus quantiles when using @Timed

@timed generates the _total and _sum metrics, (which can deduce the mean from)

Summaries also support setting quantiles so you can see the median (0.5) latency, and 99th percentile (0.99) It would be really useful to have quartile as an annotation option:

@Timed(value = "sample_bob_scheduled", quantiles = {0.5, 0.99})

Support global tags on the registry

These tags should be added to every Meter registered to the registry.

Currently s-c-stream's ApplicationMetricsExporter adds tags to metrics just before shipping the payload.

NullPointerException when no binders are present

In AbstractMetricRegistry the binders autowiring is not required so it can be null. The bind() method should protect against that.

StackTrace:

<removed Spring bean wiring stacks>
Caused by: java.lang.NullPointerException: null
	at org.springframework.metrics.instrument.internal.AbstractMeterRegistry.bind(AbstractMeterRegistry.java:51) ~[spring-metrics-0.2.0-dev.0.uncommitted+37fa988.jar:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_92]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_92]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_92]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_92]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
	... 32 common frames omitted

Add `RestTemplate` monitoring

Pull over support from s-c-netflix and generalize to the MeterRegistry abstraction.

Provide abstract spectator implementation for push backends taking step counts

Much of the implementation for Datadog, Influx, and Clickhouse is the same. The only real difference is the exposition format.

Timer record method throws 'Exception'

Please update record to throw an unchecked exception (subclass of RuntimeException)

I'm happy to send a PR.

Add timing support for the Webflux annotation-based programming model

Common tags not applied to every metric

Just tested the 0.7.0 Release and its new common tags.
But it seems that not all metrics are populated with these tags.

# HELP jvm_buffer_memory_used  
# TYPE jvm_buffer_memory_used gauge
jvm_buffer_memory_used{id="direct",} 8192.0
jvm_buffer_memory_used{id="mapped",} 0.0
# HELP jvm_buffer_count  
# TYPE jvm_buffer_count gauge
jvm_buffer_count{id="direct",} 1.0
jvm_buffer_count{id="mapped",} 0.0
# HELP jvm_memory_used  
# TYPE jvm_memory_used gauge
jvm_memory_used{id="Code Cache",} 2.2007552E7
jvm_memory_used{id="PS Eden Space",} 4.88267176E8
jvm_memory_used{id="PS Old Gen",} 3.4389504E7
jvm_memory_used{id="PS Survivor Space",} 0.0
jvm_memory_used{id="Compressed Class Space",} 8292496.0
jvm_memory_used{id="Metaspace",} 6.2853696E7
# HELP logback_events  
# TYPE logback_events counter
logback_events{level="trace",region="us-east-1",stack="prod",} 4965.0
logback_events{level="warn",region="us-east-1",stack="prod",} 109.0
logback_events{level="error",region="us-east-1",stack="prod",} 0.0
logback_events{level="debug",region="us-east-1",stack="prod",} 6065.0
logback_events{level="info",region="us-east-1",stack="prod",} 98.0
# HELP jvm_buffer_total_capacity  
# TYPE jvm_buffer_total_capacity gauge
jvm_buffer_total_capacity{id="direct",} 8192.0
jvm_buffer_total_capacity{id="mapped",} 0.0
# HELP jvm_memory_max  
# TYPE jvm_memory_max gauge
jvm_memory_max{id="Code Cache",} 2.5165824E8
jvm_memory_max{id="PS Eden Space",} 1.36839168E9
jvm_memory_max{id="PS Old Gen",} 2.863661056E9
jvm_memory_max{id="PS Survivor Space",} 3.145728E7
jvm_memory_max{id="Compressed Class Space",} 1.073741824E9
jvm_memory_max{id="Metaspace",} -1.0
# HELP jvm_memory_committed  
# TYPE jvm_memory_committed gauge
jvm_memory_committed{id="Code Cache",} 2.2544384E7
jvm_memory_committed{id="PS Eden Space",} 6.88390144E8
jvm_memory_committed{id="PS Old Gen",} 2.24395264E8
jvm_memory_committed{id="PS Survivor Space",} 3.145728E7
jvm_memory_committed{id="Compressed Class Space",} 8609792.0
jvm_memory_committed{id="Metaspace",} 6.418432E7

Tags were set like this

    @Autowired
    private PrometheusMeterRegistry prometheusMeterRegistry;

    @PostConstruct
    public void setCommonTags() {
        prometheusMeterRegistry.commonTags("stack", "prod", "region", "us-east-1");
    }

Thanks

Add ability for Prometheus default registry to be able to report

Micrometer instantiates its own instance of CollectorRegistry. This means that metrics that are wired in using the static/default one will not be reported via the micrometer wired endpoint.

PrometheusMetricRegistry doesn't actually register metrics with its internal CollectorRegistry

There are 2 options:

• PrometheusMetricRegistry could override Register from its parent class and add the metric to the internal registry. (Simplest, but could cause trouble with multiple registrations, so code will need to check/handle that)
• The PrometheusMetricRegistry could implement the Prometheus Collector interface itself and only register itself as the single collector that Prometheus call to fetch metrics from.

While the second option will be slightly more complex, I think it will give us the ability to shim/protect against any Prometheus/Spring Metrics differences.

Atlas push utility for short-lived batch operations

Create a mechanism that can synchronously push all recorded metrics to Atlas, such as might be called at the end of a streaming or batch operation.

Add `longTask` to `@Timed`

Use: for endpoints that are expected to take a long time. Applies to both @RequestMapping (I would think this would be rare) and @Scheduled.

Add support for instrumenting `ExecutorService`

Required by #1

Add support for timing `@Scheduled`

Required by #1. Dependent on enhancement to @Timed in #8.

Support Datadog

How to count exceptions with controlleradvice?

Hi,

i would like to keep track of thrown exception in a request but it seems that this is not possible if the request is catched by a controller advice.
Any idea how i can track thrown exception with controller advices?

Thanks

Prometheus: Min and Max request time

Hi,
is is currently possible to capture the minimum and maximum request time for an endpoint with @timed?
If not, does it make sense to implement something like this?
Thanks

Common tags are killing http_server_requests

When I apply common tags to a prometheus registry the http_server_requests metrics get "killed".
More precise the textwriter for format004 throws an index out of bound exception.
This issue occurs the first time after the very first request against a timed endpoint was made.

java.lang.IndexOutOfBoundsException: Index: 5, Size: 5
	at java.util.LinkedList.checkElementIndex(LinkedList.java:555) ~[na:1.8.0_131]
	at java.util.LinkedList.get(LinkedList.java:476) ~[na:1.8.0_131]
	at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:42) ~[simpleclient_common-0.0.25.jar:na]
	at io.micrometer.spring.export.prometheus.PrometheusActuatorEndpoint.invoke(PrometheusActuatorEndpoint.java:48) ~[micrometer-spring-legacy-0.7.0.RELEASE.jar:0.7.0.RELEASE]

The tags are set like this

    @Autowired
    private PrometheusMeterRegistry prometheusMeterRegistry;

    @PostConstruct
    public void setCommonTags() {
        prometheusMeterRegistry.commonTags("stack", "prod", "region", "us-east-1");
    }

thanks

Broaden the use of tag formatters

1. Ensure there is a tag formatter for every monitoring system
2. Expand the application of tag formatters to every tag (including tags found in application code), not just tags created by binders.

PRs don't show circle build status

Instrument `ThreadPoolTaskExecutor` and `ThreadPoolTaskScheduler`

In the case of ThreadPoolTaskExecutor, can monitor by adding a TaskDecorator.

ThreadPoolTaskScheduler will have to be wrapped at instantiation time.

Registering different meter types with the same name in PrometheusMeterRegistry

Currently, attempting to do this would throw a ClassCastException. Improve the error message.

Allow `Meter` to be typed

This way, a custom meter can mimick a core type like a Counter, Gauge, etc. but do so in a special way.

Any plans for Spring Cloud messaging support?

Hi there,

i would like to monitor my messaging channels as well as my http endpoints.
Currently we bridge the springboot actuator metrics to your prometheus metrics which works but isn't really the solution we want.
So are there plans to add support for spring cloud metrics like channel count or else?

Thanks for your work so far, really great library!

PrometheusLongTaskTimer does not allow tag variation

Need to add a Child concept to this similar to #27.

Unable to instrument dataSource

Hi,
i'm currently trying to instrument my DataSource in my SpringBoot 1.5.4 application but it seems that it isn't working properly.

Instrumentation is done according to the documentation (doc seems a bit outdated?)

    private void instrumentDataSource() {
        SpringMeters.monitor(
                registry,
                dataSource,
                metadataProviders,
                "data_source",
                Tag.of("stack", env.acceptsProfiles("cloud") ? "cloud" : "local")
        );
    }

But i'm only receiving NaN values :/
data_source_active_connections{stack="local",} NaN

Has someone successfully instrumented his dataSource? Any tips?

Thanks in advance

Add ability to emit tracing and logging when metering

Inspired by this talk: https://youtu.be/P8dc-rLnLr0?t=5h31m5s at Monitorama.

What if I were able to create a timer and when it runs it could check if there was an active slueth/zipkin trace and it could create a span. Also at the same time it could log out details. This way app code would be instrumented by a single line and we could emit the log/metric/trace info on their behalf.

Reduce thread contention in `Collector` lookup in `PrometheusMeterRegistry`

Employ the idea here.

Add Histogram support

The history of Dropwizard and Prometheus client-side quantile computation is too much to ignore.