deadtrickster/prometheus-plugs

Prometheus.erl Elixir Plugs

Prometheus.io Plugs Instrumenter/Exporter

Elixir Plug integration for prometheus.ex

Quick introduction by @skosch: Monitoring Elixir apps in 2016: Prometheus/Grafana Step-by-Step Guide

• IRC: #elixir-lang on Freenode;
• Slack: #prometheus channel - Browser or App(slack://elixir-lang.slack.com/messages/prometheus).

Instrumentation

To instrument whole plug pipeline use Prometheus.PlugPipelineInstrumenter:

defmodule MyApp.Endpoint.PipelineInstrumenter do
  use Prometheus.PlugPipelineInstrumenter
end

To instrument just a single plug use Prometheus.PlugInstrumenter:

defmodule MyApp.CoolPlugInstrumenter do
  use Prometheus.PlugInstrumenter, [plug: Guardian.Plug.EnsureAuthenticated,
                                    counter: :guardian_ensure_authenticated_total,
                                    histogram: :guardian_ensure_authenticated_duration_microseconds,
                                    labels: [:authenticated]]
end

Both modules implement plug interface and Prometheus.PlugInstrumenter generates proxy for specified plug so you'll need to replace instrumented plug with your instrumenter in pipeline.

Instrumenters configured via :prometheus app environment. Please consult respective modules documentation on what options are available.

Exporting

To export metric we first have to create a plug that will serve scraping requests.

defmodule MyApp.MetricsExporter do
  use Prometheus.PlugExporter
end

Call the MyApp.MetricsExporter.setup/0 function when the application starts.

# e.g. in `application.ex`
MyApp.MetricsExporter.setup()

Then we add exporter to MyApp pipeline:

plug MyApp.MetricsExporter

You can configure path, export format and Prometheus registry via :prometheus app environment. For more information please see Prometheus.PlugExporter module documentation.

Export endpoint can be secured using HTTP Basic Authentication:

  auth: {:basic, "username", "password"}

Integrations / Collectors / Instrumenters

• Ecto collector
• Plugs Instrumenter/Exporter
• Elli middleware
• Fuse plugin
• Phoenix instrumenter
• Process Info Collector
• RabbitMQ Exporter

Installation

The package can be installed as:

1. Add :prometheus_plug to your list of dependencies in mix.exs:

   def deps do
     [{:prometheus_plugs, "~> 1.1.1"}]
   end

2. Ensure prometheus is started before your application:

   def application do
     [applications: [:prometheus_plugs]]
   end

License

This project is licensed under the MIT license. Copyright (c) 2016-present, Ilya Khaprov.

prometheus-plugs's Issues

Initialize counter metrics upon declaration

Declaring a counter does not mean it exists, as it will only have a value after the first inc. So, the fist time a counter is bumped it goes from not existing to 1, which makes it hard to create alert expressions. I propose that we manually assure the metric exists, (It can be done by doing a inc by 0) so we have stable counters.

Maybe this should be adressed by prometheus_ex? What you think?

Unknown metric {registry: default, name: telemetry_scrape_duration_seconds} with phoenix 1.4 rc1

I run MshopWeb.Endpoint.Instrumenter.setup() in start function at application.ex

deps
{:prometheus_phoenix, "> 1.2.0"},
{:prometheus_plugs, "> 1.1.5"},

[info] GET /metrics
[error] Ranch listener MshopWeb.Endpoint.HTTP had connection process started with :cowboy_protocol:start_link/4 at #PID<0.452.0> exit with reason: {{%Prometheus.UnknownMetricError{name: :telemetry_scrape_duration_seconds, registry: :default}, [{:prometheus_metric, :check_mf_exists, 4, [file: 'src/prometheus_metric.erl', line: 142]}, {:prometheus_summary, :insert_metric, 5, [file: 'src/metrics/prometheus_summary.erl', line: 355]}, {:prometheus_summary, :observe, 4, [file: 'src/metrics/prometheus_summary.erl', line: 161]}, {MshopWeb.Plugs.MetricsExporter, :"-scrape_data/1-after$^1/0-0-", 2, [file: 'lib/mshop_web/plugs/metrics.ex', line: 2]}, {MshopWeb.Plugs.MetricsExporter, :scrape_data, 1, [file: 'lib/mshop_web/plugs/metrics.ex', line: 2]}, {MshopWeb.Plugs.MetricsExporter, :call, 2, [file: 'lib/mshop_web/plugs/metrics.ex', line: 2]}, {MshopWeb.Endpoint, :plug_builder_call, 2, [file: 'lib/mshop_web/endpoint.ex', line: 1]}, {MshopWeb.Endpoint, :"call (overridable 3)", 2, [file: 'lib/plug/debugger.ex', line: 122]}, {MshopWeb.Endpoint, :call, 2, [file: 'lib/mshop_web/endpoint.ex', line: 1]}, {Plug.Adapters.Cowboy.Handler, :upgrade, 4, [file: 'lib/plug/cowboy/handler.ex', line: 18]}, {:cowboy_protocol, :execute, 4, [file: 'c:/projects/elixir/mshop_umbrella/deps/cowboy/src/cowboy_protocol.erl', line: 442]}]}, {MshopWeb.Endpoint, :call, [%Plug.Conn{adapter: {Plug.Adapters.Cowboy.Conn, :...}, assigns: %{}, before_send: [], body_params: %Plug.Conn.Unfetched{aspect: :body_params}, cookies: %Plug.Conn.Unfetched{aspect: :cookies}, halted: false, host: "127.0.0.1", method: "GET", owner: #PID<0.452.0>, params: %Plug.Conn.Unfetched{aspect: :params}, path_info: ["metrics"], path_params: %{}, port: 4000, private: %{}, query_params: %Plug.Conn.Unfetched{aspect: :query_params}, query_string: "", remote_ip: {127, 0, 0, 1}, req_cookies: %Plug.Conn.Unfetched{aspect: :cookies}, req_headers: [{"host", "127.0.0.1:4000"}, {"connection", "keep-alive"}, {"cache-control", "max-age=0"}, {"upgrade-insecure-requests", "1"}, {"user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36"}, {"dnt", "1"}, {"accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8"}, {"accept-encoding", "gzip, deflate, br"}, {"accept-language", "en,zh-CN;q=0.9,zh;q=0.8,zh-TW;q=0.7,en-US;q=0.6"}, {"cookie", "pga4_session=1f9b79bc-3a0f-489c-a5a2-3ffdb7a532f6!I4Q4CRnZ6JV8Qs4fEXolO6kiTlM="}], request_path: "/metrics", resp_body: nil, resp_cookies: %{}, resp_headers: [{"cache-control", "max-age=0, private, must-revalidate"}], scheme: :http, script_name: [], secret_key_base: nil, state: :unset, status: nil}, []]}}

PlugsPipelineInstrumenter and PlugInstrumenter

Rename Prometheus.PlugsInstrumenter to Prometheus.PlugPipelineInstrumenter and instroduce
Prometheus.PlugInstrumenter.

PlugInstrumenter will act as a wrapper for single existing Plug.

Documentation for integration with phoenix

Trying to configure exporter
Documentation desperately needed :( Currently it's not clear what should be done.

Current code, compiles, pretends all ok - but doesn't actually work (i.e. /metrics is 404)
Am i missing something important?

# lib/d2.ex
defmodule D2 do
  use Application
  def start(_type, _args) do
    import Supervisor.Spec, warn: false
    D2.MetricsExporter.setup()
...

defmodule D2.MetricsExporter do
  use Prometheus.PlugExporter
end

# config/config.exs
config :prometheus, D2.MetricsExporter,
  path: "/metrics",
  format: :text,
  registry: :default

# web/router.ex
defmodule D2.Router do
  use D2.Web, :router
  pipeline :browser do
    plug D2.MetricsExporter
    plug :accepts, ["html"]
    plug :fetch_session
    plug :fetch_flash
    plug :put_secure_browser_headers
  end

scope "/", D2 do
    pipe_through :browser
    ...
end
...
end

# mix.exs
  def application do
    [mod: {D2, []},
     applications: [:phoenix, ...
                    :prometheus, :prometheus_ex, :prometheus_phoenix, :prometheus_ecto]]
  end

  defp deps do
    [{:phoenix, "~> 1.2.0"},
      ....
     {:prometheus_ex, "~> 1.0.0-alpha8"},
     {:prometheus_phoenix, "~> 1.0.0-alpha8"},
     {:prometheus_ecto, "~> 1.0.0-alpha8"},
     {:prometheus_plugs, "~> 1.0.0-alpha8"},

     {:cowboy, "~> 1.0"}]
  end

Implement telemetry_scrape_ metrics

https://docs.google.com/document/d/1Q0MXWdwp1mdXCzNRak6bW5LLVylVRXhdi7_21Sg15xQ/edit?pref=2&pli=1

Investigate request/response size measurement

This should be really fast!

Tests & CI

Content type negotiation for exporter

wait for deadtrickster/prometheus.erl#29

Configuring basic auth in Distillery

In my rel/config/config.exs
I set

config :prometheus, ValiotApp.PrometheusExporter,
  auth: {:basic, System.get_env("METRICS_USER"), System.get_env("METRICS_PASS")}

But PrometheusExporter isn't using this configuration. If I instead set it on config/prod.exs it uses the configuration but the env variables aren't set at build time so it has an empty user and password.

I believe it might by related to #25

Performance impact

Hey Ilya, thanks for the work you put into this :)

There is a slight (ok.. huge) performance hit in the exporter plug. Due to calculating the data first, then checking for a route match, the data is calculated on every request, but during high application usage almost never used.

:plug warning when compiling

Hi! Thanks for the lib!

I'm getting this when compiling:

warning: Plug.Builder.compile/3 defined in application :plug is used by the current application but the current application does not depend on :plug. To fix this, you must do one of:

  1. If :plug is part of Erlang/Elixir, you must include it under :extra_applications inside "def application" in your mix.exs

  2. If :plug is a dependency, make sure it is listed under "def deps" in your mix.exs

  3. In case you don't want to add a requirement to :plug, you may optionally skip this warning by adding [xref: [exclude: [Plug.Builder]]] to your "def project" in mix.exs

  lib/prometheus/plug_instrumenter.ex:227: Prometheus.PlugInstrumenter.__before_compile__/1

warning: Plug.Router.Utils.split/1 defined in application :plug is used by the current application but the current application does not depend on :plug. To fix this, you must do one of:

  1. If :plug is part of Erlang/Elixir, you must include it under :extra_applications inside "def application" in your mix.exs

  2. If :plug is a dependency, make sure it is listed under "def deps" in your mix.exs

  3. In case you don't want to add a requirement to :plug, you may optionally skip this warning by adding [xref: [exclude: [Plug.Router.Utils]]] to your "def project" in mix.exs

  lib/prometheus/plug_exporter.ex:73: Prometheus.PlugExporter.__using__/1

Here are my versions:

Erlang/OTP 24 [erts-12.3.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Mix 1.13.4 (compiled with Erlang/OTP 22)

To fix this you need to add the code below to the mix.exs

  def application do
    [
      applications: [:logger, :prometheus_ex, :accept],
      extra_applications: [:plug]  # added this line
    ]
  end

I can open a PR if you don't mind.

1.1.5 - warning: System.stacktrace/0 is deprecated, use __STACKTRACE__ instead

use Prometheus.PlugPipelineInstrumenter

The code above triggers the following warning:

warning: System.stacktrace/0 is deprecated, use __STACKTRACE__ instead

on Elixir 1.11[.2].

This is a problem because we use mix compile --warnings-as-errors in our CI to prevent developer mistakes/warnings seeping into master, which means our CI builds fail because of this warning.

Looks like you can also mark the macro code as generated to supress warnings https://github.com/elixir-lang/elixir/blob/b889975c5574d9edd679ecf55c6707e3eb1f51d1/lib/ex_unit/lib/ex_unit/assertions.ex#L710

[Suggestion] Telemetry integration

Now than Phoenix 1.4 has been released and it's defaulting to use Telemetry (https://github.com/beam-telemetry/telemetry) may be a good idea to integrate it so every metric managed by telemetry shows up in prometheus.

Generate modules from configuration

Prometheus.PlugPipelineInstrumenter generates dialyzer warnings

I've defined a module in our code that uses the PlugPipelineInstrumenter:

  use Prometheus.PlugPipelineInstrumenter

But when I run dialyzer I get the following output for it:

[omitted]/pipeline_instrumenter.ex:4: The inferred type for the 1st argument of call/2 (#{'__struct__':='Elixir.Plug.Conn', 'adapter':={atom(),_}, 'assigns':=#{atom()=>_}, 'before_send':=[fun((_) -> any())], 'body_params':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary() | [any()] | map()}, 'cookies':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary()}, 'halted':=_, 'host':=binary(), 'method':=binary(), 'owner':=pid(), 'params':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary() | [any()] | map()}, 'path_info':=[binary()], 'path_params':=#{binary()=>binary() | [any()] | map()}, 'port':=char(), 'private':=#{atom()=>_}, 'query_params':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary() | [any()] | map()}, 'query_string':=binary(), 'remote_ip':={byte(),byte(),byte(),byte()} | {char(),char(),char(),char(),char(),char(),char(),char()}, 'req_cookies':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary()}, 'req_headers':=[{_,_}], 'request_path':=binary(), 'resp_body':='nil' | binary() | maybe_improper_list(binary() | maybe_improper_list(any(),binary() | []) | byte(),binary() | []), 'resp_cookies':=#{binary()=>map()}, 'resp_headers':=[{_,_}], 'scheme':='http' | 'https', 'script_name':=[binary()], 'secret_key_base':='nil' | binary(), 'state':='chunked' | 'file' | 'sent' | 'set' | 'set_chunked' | 'set_file' | 'unset', 'status':='nil' | non_neg_integer()}) is not a supertype of #{'__struct__':='Elixir.Plug.Conn', 'adapter':={atom(),_}, 'assigns':=#{atom()=>_}, 'before_send':=[fun((#{'__struct__':='Elixir.Plug.Conn', 'adapter':={_,_}, 'assigns':=map(), 'before_send':=[fun((_) -> any())], _=>_}) -> #{'__struct__':='Elixir.Plug.Conn', 'adapter':={_,_}, 'assigns':=map(), 'before_send':=[fun((_) -> any())], _=>_})], 'body_params':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary() | [binary() | [any()] | #{binary()=>_}] | #{binary()=>binary() | [any()] | #{binary()=>_}}}, 'cookies':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary()}, 'halted':=_, 'host':=binary(), 'method':=binary(), 'owner':=pid(), 'params':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary() | [binary() | [any()] | #{binary()=>_}] | #{binary()=>binary() | [any()] | #{binary()=>_}}}, 'path_info':=[binary()], 'path_params':=#{binary()=>binary() | [binary() | [any()] | #{binary()=>_}] | #{binary()=>binary() | [any()] | #{binary()=>_}}}, 'peer':={{byte(),byte(),byte(),byte()} | {char(),char(),char(),char(),char(),char(),char(),char()},char()}, 'port':=char(), 'private':=#{atom()=>_}, 'query_params':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary() | [binary() | [any()] | #{binary()=>_}] | #{binary()=>binary() | [any()] | #{binary()=>_}}}, 'query_string':=binary(), 'remote_ip':={byte(),byte(),byte(),byte()} | {char(),char(),char(),char(),char(),char(),char(),char()}, 'req_cookies':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary()}, 'req_headers':=[{binary(),binary()}], 'request_path':=binary(), 'resp_body':='nil' | binary() | maybe_improper_list(binary() | maybe_improper_list(any(),binary() | []) | byte(),binary() | []), 'resp_cookies':=#{binary()=>#{}}, 'resp_headers':=[{binary(),binary()}], 'scheme':='http' | 'https', 'script_name':=[binary()], 'secret_key_base':='nil' | binary(), 'state':='chunked' | 'file' | 'sent' | 'set' | 'set_chunked' | 'set_file' | 'unset', 'status':='nil' | non_neg_integer()}, which is expected type for this argument in the callback of the 'Elixir.Plug' behaviour
[omitted]/pipeline_instrumenter.ex:4: The inferred return type of call/2 (#{'__struct__':='Elixir.Plug.Conn', 'adapter':={atom(),_}, 'assigns':=#{atom()=>_}, 'before_send':=[fun((_) -> any()),...], 'body_params':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary() | [any()] | map()}, 'cookies':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary()}, 'halted':=_, 'host':=binary(), 'method':=binary(), 'owner':=pid(), 'params':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary() | [any()] | map()}, 'path_info':=[binary()], 'path_params':=#{binary()=>binary() | [any()] | map()}, 'port':=char(), 'private':=#{atom()=>_}, 'query_params':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary() | [any()] | map()}, 'query_string':=binary(), 'remote_ip':={byte(),byte(),byte(),byte()} | {char(),char(),char(),char(),char(),char(),char(),char()}, 'req_cookies':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary()}, 'req_headers':=[{_,_}], 'request_path':=binary(), 'resp_body':='nil' | binary() | maybe_improper_list(binary() | maybe_improper_list(any(),binary() | []) | byte(),binary() | []), 'resp_cookies':=#{binary()=>map()}, 'resp_headers':=[{_,_}], 'scheme':='http' | 'https', 'script_name':=[binary()], 'secret_key_base':='nil' | binary(), 'state':='chunked' | 'file' | 'sent' | 'set' | 'set_chunked' | 'set_file' | 'unset', 'status':='nil' | non_neg_integer()}) has nothing in common with #{'__struct__':='Elixir.Plug.Conn', 'adapter':={atom(),_}, 'assigns':=#{atom()=>_}, 'before_send':=[fun((#{'__struct__':='Elixir.Plug.Conn', 'adapter':={_,_}, 'assigns':=map(), 'before_send':=[fun((_) -> any())], _=>_}) -> #{'__struct__':='Elixir.Plug.Conn', 'adapter':={_,_}, 'assigns':=map(), 'before_send':=[fun((_) -> any())], _=>_})], 'body_params':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary() | [binary() | [any()] | #{binary()=>_}] | #{binary()=>binary() | [any()] | #{binary()=>_}}}, 'cookies':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary()}, 'halted':=_, 'host':=binary(), 'method':=binary(), 'owner':=pid(), 'params':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary() | [binary() | [any()] | #{binary()=>_}] | #{binary()=>binary() | [any()] | #{binary()=>_}}}, 'path_info':=[binary()], 'path_params':=#{binary()=>binary() | [binary() | [any()] | #{binary()=>_}] | #{binary()=>binary() | [any()] | #{binary()=>_}}}, 'peer':={{byte(),byte(),byte(),byte()} | {char(),char(),char(),char(),char(),char(),char(),char()},char()}, 'port':=char(), 'private':=#{atom()=>_}, 'query_params':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary() | [binary() | [any()] | #{binary()=>_}] | #{binary()=>binary() | [any()] | #{binary()=>_}}}, 'query_string':=binary(), 'remote_ip':={byte(),byte(),byte(),byte()} | {char(),char(),char(),char(),char(),char(),char(),char()}, 'req_cookies':=#{'__struct__'=>'Elixir.Plug.Conn.Unfetched', 'aspect'=>atom(), binary()=>binary()}, 'req_headers':=[{binary(),binary()}], 'request_path':=binary(), 'resp_body':='nil' | binary() | maybe_improper_list(binary() | maybe_improper_list(any(),binary() | []) | byte(),binary() | []), 'resp_cookies':=#{binary()=>#{}}, 'resp_headers':=[{binary(),binary()}], 'scheme':='http' | 'https', 'script_name':=[binary()], 'secret_key_base':='nil' | binary(), 'state':='chunked' | 'file' | 'sent' | 'set' | 'set_chunked' | 'set_file' | 'unset', 'status':='nil' | non_neg_integer()}, which is the expected return type for the callback of the 'Elixir.Plug' behaviour

Any idea what the cause of this is? And is there any way I can fix this by making changes to my own source code?

Thanks!

Resolving Hex dependencies... hung/stuck/no-response

Hello, im experiencing an issue when adding prometheus-plugs as a dependency to my project, and calling mix deps.get from the terminal here is how my dependencies look.

defp deps do
    [
      {:yaml_elixir, "~> 2.1.0"},
      {:tesla, "~> 1.2.0"},
      {:poison, "~> 4.0.0"},
      {:event_bus, "~> 1.6.0"},
      {:elixir_uuid, "~> 1.2"},
      {:csv, "~> 2.0.0"},
      {:decimal, "~> 1.0"},
      {:prometheus, "~> 4.0", override: true},
      {:prometheus_ex, "~> 3.0"},
      {:prometheus_plugs, "~> 1.1"},
      {:prometheus_process_collector, "~> 1.0"}
    ]
  end

I highly suspect it has to do with a circular dependency, but i dont know how to fix it.
It also spikes CPU usage to 100%.

Forward-friendly exporter plug

The current exporter plug is serving metrics on given path (/metrics by default), or passing the conn. This is for plug pipeline... and it doesn't work for phoenix router!

  # phoenix example
  # in endpoint

  plug My.PrometheusExporter.

This works... but it sounds strange. For example, to change the path, I need to set it at application config of the exporter module, not when using it.

One option is to add plug option :path.. but that is wrong approach, since plug can be composable!

Instead of we make this as "app" like plug having lots of configs... we can just have very simple exporter plug use forward mechanism. (Plug - forward/2 / Phoenix - forward/4).

  # phoenix example

  pipeline :default do
    plug :accepts, ["json"]
  end

  scope "/" do
    pipe_through :default
    forward "/metrics", My.PrometheusExporter
  end

In this case, path for metrics is configured at router, not on application config of the exporter module. Also the exporter plug runs only on that path (not for all requests as in pipeline)

Change http_request_<duration> for PlugPipelineInstrumenter

Hey,

I tried switching from :microseconds to :seconds since Prometheus suggests to use second as base unit. The problem is, the config setting only changes the metric name but not the buckets and value. While defining custom buckets is not a big deal, the reported value is still in microseconds.

Or do I miss something here?

Use Erlang Prometheus library instead of Elixir wrapper

Related to deadtrickster/prometheus.ex#26

Latency calculation doesn't account for native time unit

The Erlang docs are pretty clear that the units on :erlang.monotonic_time() are determined by the runtime and not necessarily standard across operating systems. The Prometheus.PlugPipelineInstrumenter module doesn't account for the difference between the unit of the result and the unit of the histogram.

You can determine the relative precision difference with :erlang.convert_time_unit(1, duration_unit, :native) (which on my system results in 1000 for the default :microseconds unit in the instrumentor). Once the factor is known, the observed timing can be scaled before recording. Note that :erlang.convert_time_unit/3 will truncate values smaller than the target unit to 0, and so division by the factor is probably preferable.

Phoenix Plug thinks metrics aren't defined when using PlugExporter

Here's the crash dump when attempting to hit the /metrics route via a configured plugExporter:

[error] #PID<0.1324.0> running AppWeb.Endpoint (cowboy_protocol) terminated
Server: app.myserver.com:80 (http)
Request: GET /metrics
** (exit) an exception was raised:
    ** (Prometheus.UnknownMetricError) Unknown metric {registry: default, name: telemetry_scrape_duration_seconds}.
        (prometheus) src/prometheus_metric.erl:142: :prometheus_metric.check_mf_exists/4
        (prometheus) src/metrics/prometheus_summary.erl:355: :prometheus_summary.insert_metric/5
        (prometheus) src/metrics/prometheus_summary.erl:161: :prometheus_summary.observe/4
        (prometheus_ex) lib/prometheus/metric/summary.ex:81: Prometheus.Metric.Summary.observe/2
        (App) lib/app/metrics/exporter.ex:2: App.Metrics.PlugExporter.scrape_data/1
        (App) lib/app/metrics/exporter.ex:2: App.Metrics.PlugExporter.call/2
        (App) lib/app_web/endpoint.ex:1: AppWeb.Endpoint.plug_builder_call/2
        (App) lib/plug/debugger.ex:122: AppWeb.Endpoint."call (overridable 3)"/2
        (App) lib/app_web/endpoint.ex:1: AppWeb.Endpoint.call/2
        (plug) lib/plug/adapters/cowboy/handler.ex:16: Plug.Adapters.Cowboy.Handler.upgrade/4
        (cowboy) /var/nodes/app/deps/cowboy/src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4

This seems to crash when looking at the two Summary metrics present within the plug as part of reporting.

my mix.ex:

{:prometheus, "~> 4.3"},
{:prometheus_ex, "~> 3.0.5" },
{:prometheus_plugs, "~> 1.1"}

Not really sure what I'm doing wrong at this point. The plug dependency looks like it's hitting an ETS table but can't find the metric, but it's not defined anywhere within my app outside of the plug dependency. What to do?

[Question] How to reduce the cardinality of request_path?

Hi,

Thanks for providing this great library. I am using it now to put monitoring instrumentation in our Elixir code. I have question, for service that use REST api approach, where it is normal to put the id as url path like this: GET /parent/child/<id>, if I put request_path as label, the cardinality of that label would be huge.

Is there any way we can filter/remove the <id> path?
Would be helpful if there is any pointer how to do it?

Currently I put the custom label by this code

defmodule AppWeb.Prometheus.PipelineInstrumenter do

  use Prometheus.PlugPipelineInstrumenter

  def label_value(:request_path, conn) do
    conn.request_path
  end
end

then the config is

config :prometheus, AppWeb.Prometheus.PipelineInstrumenter,
  labels: [:status_class, :status_code, :method, :host, :scheme, :request_path],
  duration_buckets: [10, 100, 1_000, 10_000, 100_000,
                      300_000, 500_000, 750_000, 1_000_000,
                      1_500_000, 2_000_000, 3_000_000],
  registry: :default,
  duration_unit: :microseconds

Sorry to ask this noob question, I am still new in Elixir world

Adding `request_path` Causes Preformance Issue

👋 Recently I forked this library to add request_path as one of the items to watch : benfalk@c78bd3b

After putting it through a stress test I notice that the performance gets slower and slower over time. Bombarding my phoenix application with requests from locust starts with an average response time of ~20ms; however, as the test continues the average continues to grow well into a full second per request. As soon as I remove the path label I've added in this fork the performance of a load test remains fairly constant through the entire stress test.

I've just started looking into this, if you have any ideas they would be greatly appreciated.

Documentation

Allow "dynamic" configuration of endpoint that exporter listen on

Currently this can be set only during compilation, but then it is still handled on runtime. This prevents use cases like creating micro service but allowing end user to configure metrics endpoint without recompilation of whole project.