tikv/rust-prometheus

Prometheus instrumentation library for Rust applications

Prometheus Rust client library

This is the Rust client library for Prometheus. The main data structures and APIs are ported from Go client.

Documentation

Find the latest documentation at https://docs.rs/prometheus.

Advanced

Crate features

This crate provides several optional components which can be enabled via Cargo [features]:

• gen: To generate protobuf client with the latest protobuf version instead of using the pre-generated client.

• nightly: Enable nightly only features.

• process: Enable process metrics support.

• push: Enable push metrics support.

Static Metric

When using a MetricVec with label values known at compile time prometheus-static-metric reduces the overhead of retrieving the concrete Metric from a MetricVec.

See static-metric directory for details.

Thanks

• brian-brazil
• ccmtaylor
• kamalmarhubi
• lucab
• koushiro

rust-prometheus's Issues

Provide a IntCounter and IntGauge

Mostly, we use the integer for counter and gauge, so it is more efficient to use atomic int64 directly than float64.

consider supporting local-core metrics

Now using metrics across multi threads has a low performance, so we have to use local metric and then flush to the global metric sometimes, it is not convenient, maybe we can use core-local metrics, But I don't know whether it is easy to do or not.

See http://rocksdb.org/blog/2017/05/14/core-local-stats.html

Minimize dependencies of the core instrumentation library

This is related to #84. Right now, there are 7 direct dependencies, and 36 transitive dependencies. Removing hyper will go a long way, dropping 21 of the dependencies. The other big improvement we can have is getting rid of regex, which will drop a further 10. With both removed, there would be just 5 dependencies, all of which are direct:

fnv (1.0.5)
lazy_static (0.2.1)
libc (0.2.17)
protobuf (1.0.24)
quick-error (0.2.2)

benchmark batch Metric Vector

If we want to inc some VecMetric many times, we may first cache in a map then inc in batch.

let m = HashMap::new();
m["a"] += 1
m["b"] += 1
// many times

for (key, value) in &m {
    metric.with_label_values(&[key]).add(value);
}

to compare with

 metric.with_label_values(&["a"]).inc(1);
 metric.with_label_values(&["b"]).inc(1);
// many times 

Use Generic Tuples to type labels

variable_labels can be expressed by generic tuples (variables in tuple length).

This brings several benifits:

1. Runtime cost of checking the length of variable_labels can be eliminated.
2. Label length can be checked in compile time.

more descriptive issues

I'd really like to contribute since I actually started working on a Prometheus client library some time ago but never got to finish it because I don't have enough time to do this right now. I found this project but the issues here have little to no descriptions or what actually is needed.

Add ConstMetric to be used in custom collectors

Double check that this is the recommended approach in client_golang, but it is what they do for the ProcessCollector: https://github.com/prometheus/client_golang/blob/master/prometheus/process_collector.go#L125-L130

check valid metric name and label name

add examples

Examples:

• work with hyper
• embed rust-prometheus
• global register and metrics usage

improve the performance for Metric vector

The MetricVec is at least 2 or 3 times slower than Metric, we should improve this.

Update the crates.io version?

Hi,

The last version published on crates.io is 0.2.8, which doesn't compile any more with current Cargo versions.

Could you publish an update to crates.io?

Thanks!

gather should returns a lexicographically sorted metric family slice

support push

port more vec metric test from golang client

Any objections to me merging this into rust metrics

I have my own very simple Prometheus implementation. Since yalls is already more baked i would like to switch to it.

Use spin lock instead of standard lock

I use crate spin 0.4.6 and find that the spin lock has a better performance than the standard lock.

In our case, we can't hold the lock for a long time, so using spin has a great benefit.

#[bench]
fn bench_lock_write_map(b: &mut Bencher) {
    let m = HashMap::with_capacity(100);
    let l = Arc::new(RwLock::new(m));

    b.iter(||{
        let mut a = l.write().unwrap();
        a.insert(1, 1);
    })
}

#[bench]
fn bench_spinlock_write_map(b: &mut Bencher) {
    let m = HashMap::with_capacity(100);
    let l = Arc::new(spin::RwLock::new(m));

    b.iter(||{
        let mut a = l.write();
        a.insert(1, 1);
    })
}


#[bench]
fn bench_lock_read_map(b: &mut Bencher) {
    let mut m = HashMap::with_capacity(100);
    m.insert(1, 1);
    let l = Arc::new(RwLock::new(m));

    b.iter(||{
        let a = l.read().unwrap();
        a.get(&1).unwrap();
    })
}

#[bench]
fn bench_spinlock_read_map(b: &mut Bencher) {
    let mut m = HashMap::with_capacity(100);
    m.insert(1, 1);
    let l = Arc::new(spin::RwLock::new(m));

    b.iter(||{
        let mut a = l.read();
        a.get(&1).unwrap();
    })
}

test tests::bench_lock_read_map      ... bench:          47 ns/iter (+/- 4)
test tests::bench_lock_write_map     ... bench:          45 ns/iter (+/- 4)
test tests::bench_spinlock_read_map  ... bench:          17 ns/iter (+/- 3)
test tests::bench_spinlock_write_map ... bench:          15 ns/iter (+/- 2)

It is not an error for a MetricFamily to have no metrics

Support additional label directly without introducing a new type

Like Prometheus java client, like

counter.inc();
counter.with_label_values(&["1", "2"]).inc();

Suggested by @brian-brazil

add tcp statistics

Maybe we can parse /proc/pid/net/sockstat

support summary

macro for registering vec metric

add example for how to use with hyper or iron

try to use Core-local Statistics mechanism

Now we find that the lock contention of Vector metrics is very serious. To avoid this, we usually implement a local metric mechanism then flush to the global metrics, this works well and has a high performance, but we can't always use this way conveniently in anywhere.

A better way is to use Core-local Statistics, like RocksDB does http://rocksdb.org/blog/2017/05/14/core-local-stats.html

add README

remove dead_code comment

data race with example_hyper

To reproduce, run example_hyper with RUSTFLAGS="-Z sanitizer=thread" cargo run on a recent nightly. Then run curl localhost:9898/metrics twice.

The sanitizer is a new cool feature. See [HowTo] Sanitize your Rust code!

listening addr "127.0.0.1:9898"
==================
WARNING: ThreadSanitizer: data race (pid=18085)
  Read of size 8 at 0x55e64c39ac08 by thread T3:
    #0 lazy_static::lazy::{{impl}}::get<prometheus::counter::Counter,fn() -> prometheus::counter::Counter> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/lazy_static-0.2.2/src/lazy.rs:19 (prometheus-race+0x00000010a744)
    #1 prometheus_race::{{impl}}::deref::__stability /home/bbigras/dev/rust/bugs/prometheus-race/<lazy_static macros>:21 (prometheus-race+0x00000010a744)
    #2 prometheus_race::{{impl}}::deref /home/bbigras/dev/rust/bugs/prometheus-race/<lazy_static macros>:22 (prometheus-race+0x00000010a744)
    #3 prometheus_race::main::{{closure}} /home/bbigras/dev/rust/bugs/prometheus-race/src/main.rs:44 (prometheus-race+0x00000010a212)
    #4 prometheus_race::main::{{closure}} /home/bbigras/dev/rust/bugs/prometheus-race/src/main.rs:44 (prometheus-race+0x00000010a212)
    #5 prometheus_race::main::{{closure}} /home/bbigras/dev/rust/bugs/prometheus-race/src/main.rs:44 (prometheus-race+0x00000010a212)
    #6 hyper::server::{{impl}}::handle<closure> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/mod.rs:434 (prometheus-race+0x0000000956a0)
    #7 hyper::server::{{impl}}::keep_alive_loop<closure,std::io::buffered::BufWriter<&mut hyper::net::HttpStream>> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/mod.rs:341 (prometheus-race+0x00000007c3a5)
    #8 hyper::server::{{impl}}::handle_connection<closure,hyper::net::HttpStream> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/mod.rs:287 (prometheus-race+0x00000007d25e)
    #9 hyper::server::handle::{{closure}}<closure,hyper::net::HttpListener> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/mod.rs:246 (prometheus-race+0x0000000d10cf)
    #10 hyper::server::listener::spawn_with::{{closure}}<hyper::net::HttpListener,closure> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/listener.rs:50 (prometheus-race+0x0000000d1814)
    #11 std::panic::{{impl}}::call_once<(),closure> /checkout/src/libstd/panic.rs:296 (prometheus-race+0x000000103fb8)
    #12 std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> /checkout/src/libstd/panicking.rs:450 (prometheus-race+0x00000008deb5)
    #13 panic_unwind::__rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:98 (prometheus-race+0x000000584c4a)
    #14 std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure>,()> /checkout/src/libstd/panic.rs:361 (prometheus-race+0x00000008b253)
    #15 std::thread::{{impl}}::spawn::{{closure}}<closure,()> /checkout/src/libstd/thread/mod.rs:357 (prometheus-race+0x00000008d2ca)
    #16 alloc::boxed::{{impl}}::call_box<(),closure> /checkout/src/liballoc/boxed.rs:638 (prometheus-race+0x0000000c16b6)
    #17 alloc::boxed::{{impl}}::call_once<(),()> /checkout/src/liballoc/boxed.rs:648 (prometheus-race+0x00000057cc14)
    #18 std::sys_common::thread::start_thread /checkout/src/libstd/sys_common/thread.rs:21 (prometheus-race+0x00000057cc14)
    #19 std::sys::imp::thread::{{impl}}::new::thread_start /checkout/src/libstd/sys/unix/thread.rs:84 (prometheus-race+0x00000057cc14)

  Previous write of size 8 at 0x55e64c39ac08 by thread T2:
    [failed to restore the stack]

  Location is global '<null>' at 0x000000000000 (prometheus-race+0x0000009b9c08)

  Thread T3 (tid=18279, running) created by thread T1 at:
    #0 pthread_create /checkout/src/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:902 (prometheus-race+0x0000004903c7)
    #1 std::sys::imp::thread::{{impl}}::new /checkout/src/libstd/sys/unix/thread.rs:72 (prometheus-race+0x00000057c915)
    #2 std::thread::spawn<closure,()> /checkout/src/libstd/thread/mod.rs:412 (prometheus-race+0x00000008b372)
    #3 hyper::server::listener::spawn_with<hyper::net::HttpListener,closure> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/listener.rs:45 (prometheus-race+0x0000000d15a8)
    #4 hyper::server::listener::{{impl}}::accept<hyper::net::HttpListener,closure> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/listener.rs:31 (prometheus-race+0x0000000c9258)
    #5 hyper::server::handle::{{closure}}<closure,hyper::net::HttpListener> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/mod.rs:248 (prometheus-race+0x0000000d1026)
    #6 std::panic::{{impl}}::call_once<(),closure> /checkout/src/libstd/panic.rs:296 (prometheus-race+0x0000001040ab)
    #7 std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> /checkout/src/libstd/panicking.rs:450 (prometheus-race+0x00000008dccb)
    #8 panic_unwind::__rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:98 (prometheus-race+0x000000584c4a)
    #9 std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure>,()> /checkout/src/libstd/panic.rs:361 (prometheus-race+0x00000008b153)
    #10 std::thread::{{impl}}::spawn::{{closure}}<closure,()> /checkout/src/libstd/thread/mod.rs:357 (prometheus-race+0x00000008cf4a)
    #11 alloc::boxed::{{impl}}::call_box<(),closure> /checkout/src/liballoc/boxed.rs:638 (prometheus-race+0x0000000c176c)
    #12 alloc::boxed::{{impl}}::call_once<(),()> /checkout/src/liballoc/boxed.rs:648 (prometheus-race+0x00000057cc14)
    #13 std::sys_common::thread::start_thread /checkout/src/libstd/sys_common/thread.rs:21 (prometheus-race+0x00000057cc14)
    #14 std::sys::imp::thread::{{impl}}::new::thread_start /checkout/src/libstd/sys/unix/thread.rs:84 (prometheus-race+0x00000057cc14)

  Thread T2 (tid=18278, running) created by thread T1 at:
    #0 pthread_create /checkout/src/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:902 (prometheus-race+0x0000004903c7)
    #1 std::sys::imp::thread::{{impl}}::new /checkout/src/libstd/sys/unix/thread.rs:72 (prometheus-race+0x00000057c915)
    #2 std::thread::spawn<closure,()> /checkout/src/libstd/thread/mod.rs:412 (prometheus-race+0x00000008b372)
    #3 hyper::server::listener::spawn_with<hyper::net::HttpListener,closure> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/listener.rs:45 (prometheus-race+0x0000000d15a8)
    #4 hyper::server::listener::{{impl}}::accept<hyper::net::HttpListener,closure> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/listener.rs:31 (prometheus-race+0x0000000c9258)
    #5 hyper::server::handle::{{closure}}<closure,hyper::net::HttpListener> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/mod.rs:248 (prometheus-race+0x0000000d1026)
    #6 std::panic::{{impl}}::call_once<(),closure> /checkout/src/libstd/panic.rs:296 (prometheus-race+0x0000001040ab)
    #7 std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> /checkout/src/libstd/panicking.rs:450 (prometheus-race+0x00000008dccb)
    #8 panic_unwind::__rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:98 (prometheus-race+0x000000584c4a)
    #9 std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure>,()> /checkout/src/libstd/panic.rs:361 (prometheus-race+0x00000008b153)
    #10 std::thread::{{impl}}::spawn::{{closure}}<closure,()> /checkout/src/libstd/thread/mod.rs:357 (prometheus-race+0x00000008cf4a)
    #11 alloc::boxed::{{impl}}::call_box<(),closure> /checkout/src/liballoc/boxed.rs:638 (prometheus-race+0x0000000c176c)
    #12 alloc::boxed::{{impl}}::call_once<(),()> /checkout/src/liballoc/boxed.rs:648 (prometheus-race+0x00000057cc14)
    #13 std::sys_common::thread::start_thread /checkout/src/libstd/sys_common/thread.rs:21 (prometheus-race+0x00000057cc14)
    #14 std::sys::imp::thread::{{impl}}::new::thread_start /checkout/src/libstd/sys/unix/thread.rs:84 (prometheus-race+0x00000057cc14)

SUMMARY: ThreadSanitizer: data race /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/lazy_static-0.2.2/src/lazy.rs:19 in lazy_static::lazy::{{impl}}::get<prometheus::counter::Counter,fn() -> prometheus::counter::Counter>
==================
[...]

port process collector

So we can monitor current running process CPU, Memory, etc.

support gauge

port more guage test from golang client.

support labels macro to create labels easily

we may refer https://github.com/pingcap/tikv/blob/master/src/util/macros.rs#L66 directly and change a little.

LocalHistogramTimer should have #[must_use]

If LocalHistogramTimer is not used, it is dropped immediately so that the timer cannot collect useful info. We need to explicitly state that it is must_use in case of programmer's mistakes.

support a global registry

Like

use prom;

prom.register(Box::new(c))
prom.gather()

add document

Use Mutable Reference in LocalHistogram Interfaces

As previously discussed in #141
Open an issue here in case of forgetting.

support metric vector and variable labels

register() in Registry would panic

Hi,

just saw that calling register() on a Registry could panic without any possibility for the user to let that error bubble up the chain to be handled gracefully outside the registry (see the call to unnwrap() in https://github.com/pingcap/rust-prometheus/blob/master/src/registry.rs#L203)

Is that intentional? Would it not be better to map over the result of write?

Jan

use macro to register global metics easily

Like

register_counter!(HTTP_COUNTER, counter_opts);

// then we can use 
HTTP_COUNTER.inc();

support histogram

Status of this library and comparison to other prometheus libs in Rust

I'm looking to use Prometheus in a Rust program. There are a few libraries on crates. Is this the one that people are coalescing around? It would be good if the README could have some small section stating its status and comparing it to alternatives. A sentence or two would be fine.

Histogram supports start_timer

Like

let timer = histogram.start_timer();
// do something
timer.observe_duration();

support output with text and protobuf format

• Text format
• Protocol buffer format

https://prometheus.io/docs/instrumenting/exposition_formats/

Counter and CounterVec don't allow set

Hi,

I'm currently implementing an exporter using this crate, so I'm mining metrics then exposing them.

This works fine - except I can't set a counter to a value. This is an absolute requirement for an exporter which monitors an external component. At each exporter run I have no idea what the last value of the counter was - so I can't increment by x.

Let me know if a PR would be appropriate.

Allow building without hyper

In the ideal use of Prometheus, libraries provide their own instrumentation:

The short answer is to instrument everything. Every library, subsystem and service should have at least a few metrics to give you a rough idea of how it is performing.
―https://prometheus.io/docs/practices/instrumentation/

Bringing in the push client by default makes that hard to do: any library instrumented with Prometheus suddenly brings in hyper and all its dependencies.

Here are a couple of ideas on how to fix this:

• make the push gateway client a separate crate
• add an optional feature to enable the push gateway client

Occasionally fails to push metrics to a pushgateway

The push client fails to push metrics after rebooting pushgateways, and halts.

support untyped

re-writer hyper example with cargo published hyper

cargo publish seems not support hyper with git dependence. we should use the published one.

simplify histogram register macro

• use Vec or &[f64] for buckets in register histogram macro
• add bucket arg for register histogram vec macro

use Atomic value to improve performance

Now we use RwLock<f64>, but we can use Atomic U64 like golang client to improve the performance.

But be careful that now the stable rust only supports AtomicUsize, so we must use this in 64 bit system only.

And we should do benchmark to see whether this can improve the performance or not.

port benchmark from golang client

Question on LocalHistogram and flush()

Hi,

at first: thanks for putting this crate together!!

I am wondering about the use of LocalHistgram: it seems I actively have to call flush() from the individual threads, meaning that I have to decide when to actually flush().

Do you think it makes sense to have any local histogram flushed implicitly when metrics are collected? That way one would not have to worry about periodically calling flush().

Does that thinking make sense and would it work to add it to the create at some point in time (mostly asking this to gain understanding not su much as a feature request yet)

UPDATE: I realise this would require registry to know all local histograms which sort of goes against their very reason. I keep the question open in case it is of interest. Please just close.

Thanks

jan