beam-telemetry/telemetry_metrics_prometheus

Complete Telemetry.Metrics Reporter solution for Prometheus

TelemetryMetricsPrometheus

TelemetryMetricsPrometheus is a Telemetry.Metrics Reporter for aggregating and exposing Prometheus metrics based on Telemetry.Metrics definitions. TelemetryMetricsPrometheus provides a server out of the box exposing a /metrics endpoint, making setup a breeze.

Is this the right Prometheus package for me?

If you want to take advantage of consuming :telemetry events with the ease of defining and managing metrics Telemetry.Metrics brings for Prometheus, then yes! This package provides a simple and straightforward way to aggregate and report Prometheus metrics. Whether you're using Prometheus servers, Datadog, or any other monitoring solution which supports scraping, you're in luck!

If you're not interested in taking advantage of Telemetry.Metrics but still want to implement Prometheus or use :telemetry in your project, have a look at something like the OpenCensus project and see if it better meets your needs.

What if I don't want the bundled server?

TelemetryMetricsPrometheus.Core is the core implementation. You can run Core directly, however you will need to do all of the plumbing to expose a /metrics route.

Installation

The package can be installed by adding telemetry_metrics_prometheus to your list of dependencies in mix.exs:

def deps do
  [
    {:telemetry_metrics_prometheus, "~> 1.1.0"}
  ]
end

See the documentation on Hexdocs for more information.

Contributing

Contributors are highly welcome!

Additional documentation and tests are welcome.

Please open an issue for discussion before undertaking anything non-trivial before jumping in and submitting a PR.

telemetry_metrics_prometheus's Issues

Proposal: split server from project

Thanks for this library. I'd like to preface by saying that I have read https://github.com/bryannaegele/telemetry_metrics_prometheus/blob/master/docs/rationale.md#why-include-a-server

I'd like to propose splitting this library out into 2 parts, one dealing with the conversion of telemetry events to a prometheus friendly format, the other dealing with the web server side of things.

I agree with the inclusion of a server in terms of getting up and running quickly, however I think there are some downsides to this approach that make using the library difficult for certain use cases:

• Using a different version of cowboy (or gun which depends on cowlib) causing dependency issues.
• Using a different authentication method such as an IP whitelist or HTTP basic auth.
• Environments where exposing a different port is not possible, such as Heroku.

This library already does a lot of heavy lifting in handling the telemetry events. I think it could be split in 2 ways.

Extraction of web:

1. Remove the Router and server code from telemetry_metrics_prometheus into a separate project (such as telemetry_metrics_prometheus_web or telemetry_metrics_prometheus_plug)
2. Depend on telemetry_metrics_prometheus from telemetry_metrics_prometheus_web
3. Using telemetry_metrics_prometheus_web behaves as it does today, running its own webserver.

Extraction of core:

1. Leave the Router and server code in telemetry_metrics_prometheus and extract everything else into telemetry_metrics_prometheus_core
2. Depend on telemetry_metrics_prometheus_core from telemetry_metrics_prometheus
3. Using telemetry_metrics_prometheus behaves as it does today, users with different needs can use telemetry_metrics_prometheus_core.

If you agree with this proposal then I am happy to help in whatever way I can.

Ability to register multiple metrics with same name

I have a use case where I need to capture different telemetry events as a single prometheus metric. Part of the key in the telemetry event would be converted to prometheus label. I expected the below approach to work, but the second one overwrites the first one because we use the metric name as the id in the attach call. Any idea how I can handle this case?

[
  last_value([:job, :pending, :total],
    event_name: [:queue, :foo, :pending],
    tags: [:queue],
    tag_values: fn _ -> %{queue: :foo} end
  ),
  last_value([:job, :pending, :total],
    event_name: [:queue, :bar, :pending],
    tags: [:queue],
    tag_values: fn _ -> %{queue: :bar} end
  )
]

Support for summary metric type

Is there a technical reason summary is unsupported? It is the default summary type generated in phoenix projects. This is something I would be interesting in working on, but I thought I would ask before I spent time looking at it.

Add HTTPS support

TLS support can be accomplished by passing the appropriate options down to plug_cowboy and surfacing the option to users.

Change the handler_id to use event_name instead of metric_name

Hi @bryannaegele,

We have the following scenario, we want to publish metrics from Ecto, with the same metric_name even though the event_name is different.

[
 %{metric_name: [:ecto, :query], event_name: [:my_app, :repo, :query]},
 %{metric_name: [:ecto, :query], event_name: [:my_app, :another_repo, :query]},
 %{metric_name: [:ecto, :query], event_name: [:another_app, :repo, :query]}
]

That way we can build only one dashboard with the same query for each panel at Grafana with a selector for each repo, retrieving the repo from labels.

And I think we can achieve that, changing the event handler_id to use event_name instead of metric_name. What do you think?

Reset after scrape

Hey Bryan, I am working on implementing more observability in my team's systems, and I stumbled across your project.

As I understand it, there is no documented way to reset metrics.

Would you accept a PR for a configuration option to reset on scrape?

Document running multiple instances such as in an umbrella application

Document the patterns for running multiple instances, such as in an umbrella app. Include the child spec as demonstrated in the test suite and a sample prometheus scrape configuration.

Link to repo that use TelemetryMetricsPrometheus

Greetings,

I have read Hex documentation for TelemetryMetricsPrometheus, Telemetry.Metrics, OpentelemetryPhoenix
It probably should be enough, but I fail to get content from localhost:9568/metrics. I get a 200 Response, but 0 content-length.

A link to any repo using TelemetryMetricsPrometheus (and Phoenix, if possible), would be much appreciated.

From my application.ex

  def start(_type, _args) do
    :opentelemetry_cowboy.setup()
    OpentelemetryPhoenix.setup(adapter: :cowboy2)
    OpentelemetryEcto.setup([:excdf, :repo])

    metrics =  [
    Telemetry.Metrics.last_value("vm.memory.total", unit: :byte),
    Telemetry.Metrics.counter("http.request.count", unit: :byte),
    Telemetry.Metrics.sum("http.request.payload_size", unit: :byte)
    ]

    children = [
      # Start the Telemetry supervisor
      ExcdfWeb.Telemetry,
      {Phoenix.PubSub, name: ExcdfWeb.PubSub},
      {TelemetryMetricsPrometheus, [metrics: metrics]},
      # Start the Endpoint (http/https)
      ExcdfWeb.Endpoint
    ]

    # See https://hexdocs.pm/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: ExcdfWeb.Supervisor]
    Supervisor.start_link(children, opts)
  end

Include a reference table of supported metrics

I believe all reporters should include a table with the name of the Telemetry.Metric and what is their equivalent on the report terminology. Something like this:

Telemetry Metric	XYZ Metric
counter	...
last_value	...
sum	...
summary	...
histogram	...

If a metric is not supported, that's a good place to document it too.

WDYT?

Unable to start multiple instances in a single VM

From a Plug.Cowboy perspective, should it be possible to run multiple instances of the Telemetry.Metrics.Prometheus in the same VM as long as they're configured to use different ports?

Our umbrella app normally runs with individual components in separate containers in production but all together in a single Erlang VM for end-to-end testing and I'm getting failed to connect: eaddrinuse error messages when more than one instance of the exporter plug web server is turned on at once. Each component runs an instance of the exporter via a module that pulls in the exporter via a using macro and runs its own instance under a different registered name. Each using macro feeds the different exporter instances a different port on startup but something is still registering a conflict.

Am I missing something about what Plug/Cowboy considers a requirement for a unique address?

Incompatible with telemetry_metrics 0.5.0

After the changes to telemetry_metrics in 0.5.0, this project needs to be updated.

Divergence from Prometheus Python Client

Hey all, I was looking into some work between elixir and python, and I saw that this package and the prometheus python client might be diverging in the implementations. This file/line in particular are an example I found: https://github.com/prometheus/client_python/blob/master/prometheus_client/metrics.py#L610