clj-commons/iapetos

A Clojure Prometheus Client

iapetos's Issues

Reload/refresh and using the default prometheus registry to instrument functions

Hi,

I am having a little trouble using the default registry and instrumenting functions with it. At first I thought it was solely due to the fact that I use component and refresh/reload of all namespaces a lot so i defonce:ed the definition of the registry and made sure to do fn/initialize only once.

Like so:

(defonce registry (let [registry iapetos.registry/default]
                    (try
                      (-> registry
                          (fn/initialize))
                      (catch IllegalArgumentException e
                        registry))))

Still, this is what happens when I try to call an instrumented function

user> (require '[iapetos.core :as prometheus]
               '[iapetos.collector.fn :as fn])
nil
user> (defn my-fun-1 [] "EHLO")
#'user/my-fun-1
user> (fn/instrument! prometheus/default-registry #'my-fun-1)
#object[iapetos.registry.IapetosRegistry 0x78aa7886 "iapetos.registry.IapetosRegistry@78aa7886"]
user> (my-fun-1)
IllegalArgumentException No implementation of method: :increment* of protocol: #'iapetos.operations/IncrementableCollector found for class: nil  clojure.core/-cache-protocol-fn (core_deftype.clj:583)
user> 

I have checked that the fn_... metrics are registered in the export from the default registry, but I cannot seem to get instrumentation to work. Somehow it seems that I am caught between some protocol redefinition or cached reference in iapetos and what the Prometheus default registry thinks is already registered.

The above code works fine in a fresh project, void of any dependencies or code except for iapetos itself.

How would you suggest I go about using the default registry?
I would prefer not having to use it, but most other integrations (like, for example the logback appender) insist on using Prometheus CollectorRegistry/defaultRegistry.

Recreate changelog

we have a CHANGELOG.md file which is not complete

The release notes are sparse among the GitHub release. we should have a complete release changelog, cpy and paste the release note from github and copy them to changelog.

Task to do after release 0.1.9

update dependencies

Hi @xsc thx first of all for this wonderfull project. I really like it.

My question is, that I saw the dependency are outdated. Do you think that updating would cause some breaking changes? I would not mind also to help a little .
Bye

Missing with-activity-counter for percentage; variable `inc`/`dec`

My use-case is rather straight forward, but I think it is generalizable, so I keep it that way.
In a request-response scenario, I have a max amount of tokens. So I have a range of e.g. 0-10. For my dashboard I have the requirement to show the saturation of this piece of software, which is best represented using a percentage. I would like to use iapetos for that. However, iapetos currently only provides the with-activity-counter to achieve saturation representation. While observe had to be used customly for percentage calculation

As my suggestion would also add value to the library, I decided to put up an issue.

Alternatives I considered:

• Implement myself - I could do that, but I think this is something other people would also benefit from.
• While I could measure the activity using the with-activity-counter, I would still need to cross-reference that by the number of threads I provide. This is variable, so I would need to publish another metric. Infeasible.
• Change the requirement - nah... Percentage is an intuitive metric, which doesn't change the meaning when scaling horizontally. On startup, I can acquire the (0, max) directly, while to get it from another source would be overly tedious.

I have 2 possible solutions:

• Extend inc/dec to allow variable
• Add only the with-saturation-percentage wrapper

Suggestion for a discussible implementation:

;; with inc/dec modifications
(defmacro with-saturation-percentage
  [collector max-count & body]
  `(let [c#       ~collector
           steps# (/ max-count 100)]
     (inc c# steps#)
     (try
       (do ~@body)
       (finally
         (dec c# steps#)))))

;; in-situ implementation
(defmacro with-saturation-percentage
  [collector max-count & body]
  `(let [c# ~collector
         cnt# (atom 0)]
     (observe c# (/ (swap! cnt# inc) max-count))
     (try
       (do ~@body)
       (finally
         (observe c# (/ (swap! cnt# dec) max-count))))))

I find the in-situ implementation not that great, as it would introduce possibly a lot of single-use atoms.
I also thought about calculation on (value c#), but I think that would not be thread safe.

get a release version `0.1.10`

Can we get a release version 0.1.10 for this change please?

Only able to find [clj-commons/iapetos "0.1.9"], in repositories and that version has this problem when using push-gateway. I had to manually include io.prometheus version 0.8.0 in my project

Originally posted by @WarFox in #38 (comment)

Add optional exception label to with-duration

Currently, when using with-duration it does not differentiate failed executions from successful ones. Since with-duration is probably around IO a failure is definitely interesting and IMO should be separated by a label (e.g. status with value success/failed - but I didn't give it much thought).

The current solution is to either write something specific (macro/manual code) or wrap twice (using with-duration and exceptions.with-exception). IMO the common use case of with-duration should consider exceptions since it is probably mostly around IO.

WDYT?

Composite collectors

I like the ideas behind with-timestamps and with-counters macros. However, their usage is harmed by having to manually assemble the composite collector maps of e.g.

(with-timestamps 
  {:last-run (registry :app/last-run), 
   :last-success (registry :app/last-success), 
   :last-failure (registry :app/last-failure)}
  ...)

Right now I'm using the following hack:

(require '[medley.core :as m])
(require '[iapetos.core :as ia])

(defn- composite-collector [collector-fn state-name opts ks]
  (let [nm #(keyword (namespace state-name)
                     (str (name state-name) "." (name %)))
        g #(collector-fn (nm %) opts)]
    (zipmap ks (map g ks))))

(defn timestamps-gauge [state-name opts]
  (composite-collector ia/gauge state-name opts
    [:last-run :last-success :last-failure]))

(defn- composite? [collector-or-composite-map]
  (not (satisfies? iapetos.collector/Collector collector-or-composite-map)))

(defn register [registry collector]
  (if (composite? collector)
    (apply ia/register registry (vals collector))
    (ia/register registry collector)))

(defn at
  ([registry collector]
   (at registry collector {}))
  ([registry collector labels]
   (if (composite? collector)
     (m/map-vals #(registry % labels) collector)
     (registry collector labels))))

(defmacro with-timestamps
  ;; rewritten to not expect the literal map as input
  )

;; Usage
(def collector (timestamps-gauge :app/function {:labels [:part]}))
(def registry (-> (ia/collector-registry) (register collector))

(with-timestamps (at registry collector {:part "x"}) ...)

I can imagine people coming up with various Collectors assembled from the basic counters/gauges. It would be nice if we could treat them as atomic units.
What do you think about making the composite collectors a first-class citizen of Iapetos?

Expose ring internal functions

A number of ring internal methods are useful beyond their usage in purely ring apps (e.g. for writing a pedestal/reitit interceptor), such as record-metrics!. Since these are private, it's unreliable to depend on them. Thoughts on exposing these?

Next planned release?

I apologize if creating an issue isn't the best way to ask this question, but I was wondering if you were planning on releasing 1.8.0 anytime soon? I would love to use your unregister functionality that you have added.

Thanks for the library!

index documentatoin of readme

Right now the readme.md of iapetos is a big file

it would be nice if we can provide a schema like this

iapetos

Table of contents

• bo
• foo

with links on titles

jvm collector broke

user=> (require '[iapetos.collector.jvm :as jvm])
Syntax error (ClassNotFoundException) compiling at (iapetos/collector/jvm.clj:1:1).
io.prometheus.client.hotspot.StandardExports

user=> (str (System/getProperty "java.vm.vendor") " " (System/getProperty "java.vm.version"))
"AdoptOpenJDK 25.232-b09"

;; or

user=> (str (System/getProperty "java.vm.vendor") " " (System/getProperty "java.vm.version"))
"Amazon.com Inc. 25.232-b09"

;; or

user=> (str (System/getProperty "java.vm.vendor") " " (System/getProperty "java.vm.version"))
"Eclipse OpenJ9 openj9-0.17.0"

;; or

user=> (str (System/getProperty "java.vm.vendor") " " (System/getProperty "java.vm.version"))
"Amazon.com Inc. 11.0.3+7-LTS"

java -version

openjdk version "1.8.0_232"
OpenJDK Runtime Environment Corretto-8.232.09.1 (build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM Corretto-8.232.09.1 (build 25.232-b09, mixed mode)

java -version

openjdk version "1.8.0_232"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.232-b09, mixed mode)

java -version

openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-b09)
Eclipse OpenJ9 VM (build openj9-0.17.0, JRE 1.8.0 Mac OS X amd64-64-Bit Compressed References 20191017_369 (JIT enabled, AOT enabled)
OpenJ9   - 77c1cf708
OMR      - 20db4fbc
JCL      - 97b5ec8f383 based on jdk8u232-b09)

openjdk version "11.0.3" 2019-04-16 LTS
OpenJDK Runtime Environment Corretto-11.0.3.7.1 (build 11.0.3+7-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.3.7.1 (build 11.0.3+7-LTS, mixed mode)

Support "Info" metrics

Info metrics were added with first class support in 0.10.0 of the Prometheus Java client, see prometheus/client_java#615 for the exact change.

See https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#info in the OpenMetrics specification for more details.

Upstream doc

prometheus/docs#1405

@danielcompton cc

Jetty Metrics

Is there a way to collect embedded jetty metrics ? I am interested in these metrics https://github.com/prometheus/client_java/blob/master/simpleclient_jetty/src/main/java/io/prometheus/client/jetty/JettyStatisticsCollector.java#L44

• Thanks

Trying to add the labels for JVM metrics

I was trying to add the labels for JVM metrics but not sure how to add it.

The original metrics output look like this by using code.

# HELP jvm_memory_bytes_max Max (bytes) of a given JVM memory area.
# TYPE jvm_memory_bytes_max gauge
jvm_memory_bytes_max{area="heap",} 2.863661056E9
jvm_memory_bytes_max{area="nonheap",} -1.0

But I would like to see something like this (i.e: service="XXXX",area="XXXX")

# HELP jvm_memory_bytes_max Max (bytes) of a given JVM memory area.
# TYPE jvm_memory_bytes_max gauge
jvm_memory_bytes_max{area="heap",service="XXXX",area="XXXX"} 2.863661056E9
jvm_memory_bytes_max{area="nonheap",service="XXXX",area="XXXX"} -1.0

fix clidoc badge

https://github.com/clj-commons/iapetos in readme the clidoc is broken and also the link to page

Make io.prometheus.client.CollectorRegistry.defaultRegistry usable together with iapetos

G'day! This is an awesome library. Thank you so much for making it.

I'm enjoying the ring and jvm wrappers very much. However, I've recently added to my project the InstrumentedAppender but it only has a mechanism for registering its collector with the Prometheus Java client's default registry.

This means that the ring wrapper exposing metrics at /metrics doesn't pick up any non-iapetos collectors because a different default registry is created by iapetos.

I'm wondering if https://github.com/xsc/iapetos/blob/master/src/iapetos/registry.clj#L79 should be more like:

(defn create
  ([]
   (IapetosRegistry. "iapetos_registry" (CollectorRegistry/defaultRegistry) {} {})))
  ([registry-name]
   (IapetosRegistry. registry-name (CollectorRegistry.) {} {})))

This keeps the "iapetos_registry" name for the default registry for any existing code that might use it, but uses the underlying Prometheus Java client's default registry.

Allow fn instrument and wrap-instrumentation to take labels

The wrap-instrumentation function (and associated code), only add a fn-name and result label to their metrics:

iapetos/src/iapetos/collector/fn.clj

Lines 24 to 49 in 55ae550

	(defn wrap-instrumentation
	"Wrap the given function to write a series of execution metrics to the given
	registry. See [[initialize]]."
	[f registry fn-name
	{:keys [duration?
	exceptions?
	last-failure?
	run-count?]
	:or {duration? true
	exceptions? true
	last-failure? true
	run-count? true}}]
	(let [labels {:fn fn-name, :result "success"}
	failure-labels (assoc labels :result "failure")]
	(wrap->>
	f
	duration? (prometheus/with-duration
	(registry :fn/duration-seconds labels))
	exceptions? (ex/with-exceptions
	(registry :fn/exceptions-total labels))
	last-failure? (prometheus/with-failure-timestamp
	(registry :fn/last-failure-unixtime labels))
	run-count? (prometheus/with-failure-counter
	(registry :fn/runs-total failure-labels))
	run-count? (prometheus/with-success-counter
	(registry :fn/runs-total labels)))))

It would be really nice to be able to pass optional additional labels.

I'm still a little new to clojure, but I could try to submit a PR if you think that would be welcome.

Migrate to 1.0.0 library

The Java client library has released a 1.0.0 version: https://github.com/prometheus/client_java/releases/tag/v1.0.0

The main draws of using this are:

• Opentelemetry compatibility
• Native histograms

This new major version has some fairly major API changes which break the current iapetos model: https://prometheus.github.io/client_java/migration/simpleclient/. Particularly, Collectors no longer exist.

thrown exceptions are not reported to the HTTP counter on ring collector

I'm using ring/compojure to run my JSON api server
I use the wrap-instrumentation ring collector as a middleware to report prometheus metrics

My problem is that the current collector does not report thrown exceptions as 5XX status class to the http_requests_total metric
I know it increments http_exceptions_total but I still find it strange that if I those exceptions are not counted anywhere in the requests total or the latency histogram
I guess my questions are:

1. Have I missed anything or is this the behavior the ring collector works with?
2. If this is the current behavior I would like to suggest a new parameter to the options map, let's call it :exception-response which is a ring map - for example {:status 500 :body "something bad happened"}
   we can then use the record-metrics! on the ring map in case we have exception here to have both HTTP response code and latency monitored in case of a thrown exception

would be happy to get your opinion on this

add codeowner file

@MalloZup @xsc

Can't use `get` or `unregister` successfully

Vaguely related to #11, I'm using Integrant and getting "Collector already registered that provides name: default_logic_process_duration_count"

Should be an easy fix, I just need to see if the collector is already registered and if it is, don't re-register it. My problem is that I can't seem to make get or unregister work. I guess I'm supplying the wrong arguments? Doc isn't helping.

  (:require [messenger.metrics :as metrics]
            [iapetos.core :as iap]
            [iapetos.registry :as iapreg]
            [iapetos.collector :as iapcoll]))

(def process-latency-histogram
  (iap/histogram
   :logic-process-duration
   {:description "Time in seconds that worker.streams.logic-process took"
    :labels [:action :step :state]
    :buckets [0.001 0.005 0.01 0.02 0.05 0.1 0.2 0.5 1.0 2.0]}))

(defonce metrics-registry
  (let [mr metrics/metrics-registry]
    (when (iapreg/get mr (iapcoll/metric process-latency-histogram) (:labels process-latency-histogram))
      (iap/unregister mr (iapcoll/metric process-latency-histogram)))
    (iap/register mr process-latency-histogram)))

I've also tried "logic-process-duration", "logic_process_duration", "default_logic_process_duration_count", :logic-process-duraion, :logic_process_duration, :default/logic_process_duration, :default/logic-process-duration...

I also tried iapreg/clear but that also didn't seem to work, I tried it both with doto and -> :(

Help, please?

Update Java Simple Client Dependencies

Hi,

I added the [io.prometheus/simpleclient_hotspot "0.0.19"] as specified in the README, but i was greeted by a java.lang.NoClassDefFoundError: io/prometheus/client/SummaryMetricFamily when trying to access the /metrics endpoint.

Adding [io.prometheus/simpleclient "0.0.19"] to the deps as well fixed the problem for me.

Am I doing something wrong? Is it just a case of updating the readme?

investigate about codecoverage migration

the codecoverage in readme point to the old repo.

We should investigate if we can enable that service here, and ask maybe to clj-common about this, if it is possible.

Currenlty I never setted up such service

openjdk 11 doesn't work on travis.

investigate why openjdk 11 is failing on travis and fix issues. Enable it on travis when possible

Sporadic Test Failures on TravisCI (with OpenJDK?)

There seem to be recurring test failures on TravisCI, akin to:
https://travis-ci.org/xsc/iapetos/jobs/150695791

I've only seen them for OpenJDK builds and mostly before/during iapetos.standalone-test, so it might be connected to the HTTP server implementation in combination with OpenJDK.

Trying to modify buckets for existing http_request_latency_seconds_bucket

Currently, i have been leveraging prometheus's http_request_latency_seconds_bucket to see my web app's latency. I'm seeing latency assigned to these buckets: https://github.com/soundcloud/prometheus-clj/blob/f56c78c6dbc3feb7c4974327d93ed78173faa7d1/src/prometheus/core.clj#L13

However, these buckets don't work for my case, as many requests often exceed 5 seconds. I would need to add additional buckets greater than 5. Is there an easy way to do this?

choose where to publish jars

need discussion with org

Remove `dec` function

Prometheus counters should be monotonically increasing

The prometheus documentation states that:

    Counters should not be used to expose current counts of items whose number can also go down, e.g. the number of currently running goroutines.

Currently counters can be decremented by decrement or passing a negative value into increment.

See prometheus/client_ruby#17 for a similar change on the ruby client with further explanation in the linked issue.

There might be some other use case for this..? Brian suggests there is in the Ruby client PR but I can't think of it.

Please release the next version. :-)

Hi,

I'd like to be able to use the custom bucketing for my services.

Best regards,
pnathan

Mutable collector registry, using collector objects directly

Hi, a great library you have there!

I wanted to discuss the API a little bit if you don't mind - maybe we'll be able to find a common ground. Couple things that bit me right off the bat when I started using Iapetos were:

• having to go through (registry :collector) in order to use the collector
• registry being immutable

my initial attempt at using the library looked something like this:

(require '[mount.core :refer [defstate]])
(require '[iapetos.core :as ia])

;; this goes into Metrics ns
(defstate registry
  :start (ia/collector-registry))

;; this goes into Clients ns
(defstate number-of-clients
  :start (with-let [g (ia/gauge :app/number-of-clients)]
             (ia/register registry g))

(defn go []
  (ia/inc number-of-clients))

This obviously fails because the registry is immutable and the collector returned from ia/gauge cannot be used directly with ia/inc. It would be great if we could somehow allow for collectors to be declared in separate namespaces and allow registering with a Registry dynamically, e.g.:

;; Metrics ns
(defstate registry
  :start (ia/collector-registry))

(defmacro defgauge [nm k & opts]
  `(defstate ~nm
      :start (let [g# (apply ia/gauge ~k ~@opts)]
                 (ia/register! registry g#)
                 g#)

;; A ns
(defgauge clients :app/number-of-clients {:labels [:part]})

(defn go [part]
  (ia/inc (clients {:part part})))

This makes use of a ia/register! which modifies the registry to include the new collector. The returned collector also captures the registry, allowing the collector itself to be used as a function.

I think this offers two advantages:

• Collectors do not have to all be declared in the same place - they're declared in the namespaces that actually use them
• There's no need to refer to the registry, the collectors can be used as ordinary vars

Does this make any sense? What do you think?