prometheus/docs

Prometheus documentation: content and static site generator

Prometheus

Visit prometheus.io for the full documentation, examples and guides.

Prometheus, a Cloud Native Computing Foundation project, is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts when specified conditions are observed.

The features that distinguish Prometheus from other metrics and monitoring systems are:

• A multi-dimensional data model (time series defined by metric name and set of key/value dimensions)
• PromQL, a powerful and flexible query language to leverage this dimensionality
• No dependency on distributed storage; single server nodes are autonomous
• An HTTP pull model for time series collection
• Pushing time series is supported via an intermediary gateway for batch jobs
• Targets are discovered via service discovery or static configuration
• Multiple modes of graphing and dashboarding support
• Support for hierarchical and horizontal federation

Architecture overview

Install

There are various ways of installing Prometheus.

Precompiled binaries

Precompiled binaries for released versions are available in the download section on prometheus.io. Using the latest production release binary is the recommended way of installing Prometheus. See the Installing chapter in the documentation for all the details.

Docker images

Docker images are available on Quay.io or Docker Hub.

You can launch a Prometheus container for trying it out with

docker run --name prometheus -d -p 127.0.0.1:9090:9090 prom/prometheus

Prometheus will now be reachable at http://localhost:9090/.

Building from source

To build Prometheus from source code, You need:

• Go version 1.17 or greater.
• NodeJS version 16 or greater.
• npm version 7 or greater.

Start by cloning the repository:

git clone https://github.com/prometheus/prometheus.git
cd prometheus

You can use the go tool to build and install the prometheus and promtool binaries into your GOPATH:

GO111MODULE=on go install github.com/prometheus/prometheus/cmd/...
prometheus --config.file=your_config.yml

However, when using go install to build Prometheus, Prometheus will expect to be able to read its web assets from local filesystem directories under web/ui/static and web/ui/templates. In order for these assets to be found, you will have to run Prometheus from the root of the cloned repository. Note also that these directories do not include the React UI unless it has been built explicitly using make assets or make build.

An example of the above configuration file can be found here.

You can also build using make build, which will compile in the web assets so that Prometheus can be run from anywhere:

make build
./prometheus --config.file=your_config.yml

The Makefile provides several targets:

• build: build the prometheus and promtool binaries (includes building and compiling in web assets)
• test: run the tests
• test-short: run the short tests
• format: format the source code
• vet: check the source code for common errors
• assets: build the React UI

Service discovery plugins

Prometheus is bundled with many service discovery plugins. When building Prometheus from source, you can edit the plugins.yml file to disable some service discoveries. The file is a yaml-formated list of go import path that will be built into the Prometheus binary.

After you have changed the file, you need to run make build again.

If you are using another method to compile Prometheus, make plugins will generate the plugins file accordingly.

If you add out-of-tree plugins, which we do not endorse at the moment, additional steps might be needed to adjust the go.mod and go.sum files. As always, be extra careful when loading third party code.

Building the Docker image

The make docker target is designed for use in our CI system. You can build a docker image locally with the following commands:

make promu
promu crossbuild -p linux/amd64
make npm_licenses
make common-docker-amd64

Using Prometheus as a Go Library

Remote Write

We are publishing our Remote Write protobuf independently at buf.build.

You can use that as a library:

go get buf.build/gen/go/prometheus/prometheus/protocolbuffers/go@latest

This is experimental.

Prometheus code base

In order to comply with go mod rules, Prometheus release number do not exactly match Go module releases. For the Prometheus v2.y.z releases, we are publishing equivalent v0.y.z tags.

Therefore, a user that would want to use Prometheus v2.35.0 as a library could do:

go get github.com/prometheus/[email protected]

This solution makes it clear that we might break our internal Go APIs between minor user-facing releases, as breaking changes are allowed in major version zero.

React UI Development

For more information on building, running, and developing on the React-based UI, see the React app's README.md.

More information

• Godoc documentation is available via pkg.go.dev. Due to peculiarities of Go Modules, v2.x.y will be displayed as v0.x.y.
• See the Community page for how to reach the Prometheus developers and users on various communication channels.

Contributing

Refer to CONTRIBUTING.md

License

Apache License 2.0, see LICENSE.

docs's Issues

Document new authentication features

Introduced by prometheus/prometheus#921

Document changes to global label handling

See prometheus/prometheus#1051

Document clean shutdown beyond the FAQ

Currently we only mention in the FAQ that a clean requires sigterm or , that should be in the general docs too and also the http endpoint.

Hmm, should we mention that in the crash recovery error message?

Document clamp functions

prometheus/prometheus#1231

Docs: Getting Started: GOPATH and mercurial

Following the getting started tutorial, where one runs the example processes.
You could add people need to set up their GOPATH for go get -d and go build to work.

# Fetch the client library code and compile example.
git clone https://github.com/prometheus/client_golang.git
cd client_golang
export GOPATH=$(pwd)
cd examples/random
go get -d
go build

Also, when doing go get -d I got an error for Mercurial not being installed:

$ go get -d
go: missing Mercurial command. See https://golang.org/s/gogetcmd
package bitbucket.org/ww/goautoneg: exec: "hg": executable file not found in $PATH

Might be worth mentioning.

Document proxy_url

List 3rd party software exporting prometheus metrics or supporting prometheus in other ways

So far, we only list exporters, but there is so much more...

https://github.com/google/cadvisor
https://github.com/GoogleCloudPlatform/kubernetes
https://github.com/armon/go-metrics
https://github.com/coreos/etcd
https://github.com/mesosphere/kubernetes-mesos
https://github.com/peterbourgon/gokit

Anything else?

Document optional time parameter for API

prometheus/prometheus#1207

Document changes to DNS SD config

The possibility to query A records was added.

Document 'bool' modifier

See prometheus/prometheus#1049

Add tutorials/blog posts about securing a Prometheus setup

Those only have to cover the most basic scenarios to secure a Prometheus setup so people have a base for further customization.

Document selector anchoring behaviour

Document prometheus/prometheus#1201 (again)

Prometheus won't read file_sd_config

When I try to load a file_sd_config file, I get no error, but prometheus doesn't show any of those hosts on the status page as candidate for scraping, no matter how long I wait. Nothing is logged, even with -log.level=debug, and promtool doesn't check that file.

included file:

targets:
  - host1.example.com:9103
  - host2.example.com:9103
  - host3.example.com:9103
  - host4.example.com:9103
  - host5.example.com:9103
labels:
  - job: zookeeper

I'm using this prometheus:

prometheus, version 0.15.0rc3 (branch: master, revision: d437fce)
  build user:       [email protected]
  build date:       20150720-16:25:16
  go version:       1.4.2

Document irate

prometheus/prometheus#1155

Add a glossary

We've lots of terms, and I expect some of them (like bridge vs. collector vs exporter) are only in a few people's heads. We should have a glossary.

Rule-Checker documentation outdated

In "Recording rules" the rule_checker tool is referenced but according to CHANGELOG the tool has been replaced and is not longer go get-able.

Update alertmanager docs

http://prometheus.io/docs/alerting/alertmanager/ still documents the old/deprecated alertmanager.

Add documentation for new EC2 meta labels

prometheus/prometheus@9e05d82

Docs: Getting Started: Expression Browser

Go to the expression browser and verify that Prometheus now has information about time series that these example endpoints expose, such as the rpc_durations_microseconds metric.

If the "expression browser" is located under the "Graph" tab, adding the 3 random servers only provides

• scrape_duration_seconds
• up

rpc_durations_microseconds is available on the respective metrics endpoints of the random services. It does not show up in the dropdown next to the blue "Execute" button.

Document promdash label filters

I couldn't find the documentation about legend format filters. Maybe there are more promdash features which should be documented?

@stuartnelson3 @juliusv

Update comparison section

From jaredwiltshire via prometheus-developers:

"The InfluxDB comparison highlights that Prometheus uses much less disk space than Prometheus due to its data storage model. However this seems to no longer be the case - https://influxdb.com/blog/2015/10/07/the_new_influxdb_storage_engine_a_time_structured_merge_tree.html
It also references a Github issue for InfluxDB regarding indexing which has since been closed."

I also believe we might want to include a few other monitoring and event-logging systems.

Document Consul ServiceID

...once prometheus/prometheus#1019 is merged.

Docs: Getting Started: No "Tabular" view

Following http://prometheus.io/docs/introduction/getting_started/.
Under ## Using the expression browser It says to navigate te the "Tabular" view under the "Graph" tab.
No such view is apparent.

Running prometheus 0.15.1 via the latest prom/prometheus docker image (ecebe0acff82)

Document new API status codes

Document nerve SD

prometheus/prometheus#1299

Add comparison to Netflix's Atlas to documentation

From prometheus/prometheus#745

OFFSET documentation missing

Title says it all. It should probably be added to the basics section.

Reword to let newcomers work with a release, not with HEAD.

If doing everything from 'getting started' page, one will find out that config is wrong:

marko@marko-desktop:~/tmp/prometheus (master) $ ./prometheus -config.file=prometheus.conf
prometheus, version 0.13.3 (branch: master, revision: 572b280)
  build user:       marko@marko-desktop
  build date:       20150520-18:45:13
  go version:       1.4.2
E0520 18:56:16.629990   18370 main.go:224] Couldn't load configuration (-config.file=prometheus.conf): yaml: line 3: did not find expected ',' or '}'
E0520 18:56:16.630068   18370 main.go:225] Note: The configuration format has changed with version 0.14, please check the documentation.

Document Kubernetes SD changes

See prometheus/prometheus#1050

Move Prometheus Client Data Exposition Format documentation

I might have missed it, but I couldn't find the client format documentation. The format is currently documented in a google doc, but I'd vote to move it here.

https://docs.google.com/document/d/1ZjyKiKxZV83VI9ZKAXRGKaUKK2BIWCT7oiGBKDBpjEY/edit

Document label_replace() function

See prometheus/prometheus#998

Document Kubernetes SD configuration.

Once prometheus/prometheus#905 is merged.

Document `honor_labels`

honor_labels still needs to be documented (PR into next-release branch please).

Check for broken links in CI

Set html page title

The HTML page title currently always says "Prometheus". This isn't helpful for SEO and search reasons. Add the title of each page to the html title attribute, something like $page | Prometheus e.g. Getting started | Prometheus.

Document labelmap relabeling action

Provide RSS feed for blog posts

It's awesome that we have a blog now, but it's not possible to subscribe to updates. While RSS/atom is not super popular anymore, it would be nice to provide a feed so that people can use tools to get notified immediately about new blog posts.

Document scheme relabeling

See prometheus/prometheus#967

Favicon is no longer being served.

curl -sSI http://prometheus.io/favicon.ico | grep HTTP
HTTP/1.1 404 Not Found

Table of contents on pages

For some pages like the FAQ and SD it'd be good to have a ToC

Document Marathon SD

See prometheus/prometheus#902

Document AM HipChat receiver

Add non-goals

Perhaps next to the road map, add a "non-road-map" with the non-goals, i.e. things that sound like nice features but which we do not intend to implement for reasons to be stated in the new section.

Ideas so far:

• Rule templating language.
• Strings as sample values (and neither ints...).
• Support event logging / become a log db.

More?

Can't click left side menu when browser too small

If I resize the browser until the content gets pushed down below the side menu, I can't click the links on the left side menu (#col-md-3) anymore. Setting z-index:100 fixes that but not sure where to put that and if there is a cleaner way.

What is a Prometheus Domain?

On the metrics naming docs it references a "Prometheus Domain". However, I can't find any more docs on this. I just want some clarity that the naming of the etcd metrics are alright; for example should these be prefixed with etcd_?

• rafthttp_message_sent_latency_microseconds
• rafthttp_message_sent_failed_total

Document `__tmp_` prefix.

We should document that if anyone needs temporary labels, they should use the __tmp_ prefix to ensure that their temporary label name will never conflict with any existing labels.

Document branches and publishing for the docs site properly.

We need to clearly document at various places that prometheus.io is based on HEAD and represents the latest releases of the various other repos, while unreleased features (in the HEAD of the various repos) are in the next-release branch of docs.

Currently, we document something completely wrong on http://prometheus.io/docs/introduction/install/ :
"Note that this documentation (as published on prometheus.io) refers to the latest production release. The head of the prometheus/docs GitHub repository refers to the (possibly not yet released) head of the prometheus/prometheus (and other) repositories."

Document new string syntax

We should document the backticks for the next release.

Add section about crash recovery.

• that it is required after a crash
• how long it might take
• what can go wrong (esp. mention the 'orphaned' directory)

Document EC2 service discovery