autopilotpattern/prometheus

Implementation of the Autopilot Pattern for a Prometheus.io server

prometheus's Issues

Metrics forwarder

ContainerMonitor supports the Prometheus API but its clear, particularly given the poor story around long-term persistence (ref #12), that we want to have a blueprint for metrics forwarding using the Prometheus API. This metrics forwarder would scrape Prometheus endpoints and then forward to an end-user's existing metrics collectors (ex. graphite, ES, nagios, etc.).

Open questions for this issue:

• what metrics collector(s) should we support?
• what existing tooling is there for scraping Prometheus and forwarding the metrics? telegraf seems promising.
• should this be a separate blueprint or keep it in this repo as an alternative option?

Edit see also https://github.com/misterbisson/telegraf/tree/wip

Config template uses wrong service name

telemetry is the wrong service name for the ContainerPilot telemetry service

{{ if service "telemetry" }} - targets: [{{range $i, $e := service "telemetry"}}{{if ne $i 0}},{{end}}'{{$e.Address}}:{{$e.Port}}'{{end}},'localhost:9090']

This can be fixed immediately, but service discovery can also be moved internal with #5

Automatically discover endpoints via discovery service

In our prometheus config we're using a list of service targets (and, incidentally, the wrong service!). But Prometheus supports using the service catalog directly via the <consul_sd_config>. This effectively makes it "autopilot pattern native", and we can just use that and eliminate the need to have backend handlers entirely.

Update to expect triton-docker

...Which sets TRITON_CNS_SEARCH_DOMAIN_PRIVATE

Account for Differing CloudAPI/Docker Endpoints

Hi,

I'm going to throw a PR together for this, but wanted to log it. The Triton setup script assumes a certain API endpoint format (with datacenter at the beginning of the URL), which may not be the case for many non-Joyent datacenters.

For example:

$ ./setup.sh

Error! The Triton CLI configuration does not match the Docker CLI configuration.

Docker user: jay.schmidt
Triton user: jay.schmidt
Docker data center: docker
Triton data center: cloudapi

With a config of:

$ env | grep DOCKER
DOCKER_CLIENT_TIMEOUT=300
DOCKER_CERT_PATH=/Users/jschmidt/.sdc/docker/cak1-jay.schmidt
DOCKER_HOST=tcp://docker.cak-1.virington.com:2376
DOCKER_TLS_VERIFY=1
$ env | egrep "SDC|TRITON"
SDC_KEY_ID=6b:95:03:3d:d3:6e:52:69:01:96:1a:46:4a:8d:c1:7e
SDC_ACCOUNT=jay.schmidt
SDC_URL=https://cloudapi.cak-1.virington.com

In my case, the DC is the secondary item in the URL (due to using CMON). One possible fix is to just have a flag to ignore this check; a second fix is to actually query cloudapi/docker to check the datacenter. I'll work on this later this month.

Jay

Update ContainerPilot and Prometheus versions, implement new Consul co-process

And probably roll in #5 as well

Update to Prometheus 1.5.0

1.5.0 was released on 2017-01-23, per https://prometheus.io/download/

That release has Triton RFD27/Container Monitor support via prometheus/prometheus#2307

Update Consul

This now uses autopilotpattern/consul:0.7.2-r0.8 , but that was released in January 2017. Newer releases are available: https://github.com/autopilotpattern/consul/releases

Data center confusion

The config in #18 takes a data center, but it's possible to run it in a different data center. The result can be very confusing, because the data center being used for discovery is not shown anywhere within Prometheus itself.

The embarrassment this causes was discovered in CMON-66.

Add Grafana

Either in the same container or in a second container.

See Grafa+Prometheus docs in https://prometheus.io/docs/visualization/grafana/

Persistent storage

Prometheus mostly punts on the question of long-term storage of metrics. Each Prometheus node stores time series in local files with a LevelDB index. As of this writing it appears the design for remote storage will be to have some kind of generic interface for remote storage over gRPC.

At some point we'll want to have an answer to this question and I want to leave this issue here. In the meantime we'll also be following up on the notion of an example metrics forwarder (ref #13).

Update ContainerPilot 3.x

Pull Pacakge Definitions at Runtime Rather than Hardcode

Hi,

For non-public cloud use, this package will most likely not be available:

- com.joyent.package=g4-highcpu-1G

Suggestion would be to move this to generic memory/cpu target level and/or have it pull package definitions as part of the setup process to fill this line out.

Jay

docker-compose.yml environment missing

Looks like,

47 environment:
48 - CONSUL=pc.svc.${TRITON_CNS_SEARCH_DOMAIN_PRIVATE}

Is missing from the consul definition. When I ran triton-compose scale consul=3 Consul couldn't find its cluster members.

Support non public cloud CNS domains

DNS/CNS information is hard-coded for Triton Public cloud:

        dns_suffix: 'cmon.{{env "TRITON_DC"}}.triton.zone'
        endpoint: 'cmon.{{env "TRITON_DC"}}.triton.zone'

Those should have an env var that users can change to make it work in other clouds with Container Monitor.