Propagate static_labels from target collectors config to metrics to enable re-use of query config across fleet of identical but unique server DSN's

[free]

Propagate static_labels from target collectors config to metrics to enable re-use of query config across fleet of identical but unique server DSN's

[free]

Why not apply the "static label per DSN" in the Prometheus config? Something like this:

scrape_configs:
  - job_name: 'somedb'
    static_configs:
      - targets: [ 'instance1.cluster13.foo.org:9399', 'instance2.cluster13.foo.org:9399' ]
        labels:
            cluster_identifier: cluster13
      - targets: [ 'instance1.cluster14.foo.org:9399', 'instance2.cluster14.foo.org:9399' ]
        labels:
            cluster_identifier: cluster14

This also works with file_sd_config, but you have to place the labels in the targets file, next to the targets themselves (i.e. the part after static_configs above all goes into the targets file).

Or, if you're using some other form of service discovery (e.g. kubernetes_sd_config), you can use metric_relabel_configs to create a cluster_identifier label out of e.g. the instance name. It's all documented here: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config

[michaeljoy]

True, dns_sd_config just gets messy with the regex, and we don't necessarily name the scraper running the same as the cluster_identifier as they aren't running on the same hosts.

That is certainly a valid idea, however we also have other labels we want to add that don't belong in the DNS like customer_name, customer_id, etc... so it still gets super messy trying to do it in the prometheus config (also we want this to be dynamic).

Long term we'll have to replace the service discovery with Consul to be able to do this OOB tagging without having to muck with automating hundreds of duplicative file templates that have to be pulled down at container start to bootstrap the config.

I guess, the min-maxer in me just wants to see the labels created once with the DSN instead of dozens of times per file alongside the DSN in the query target config since you can't have more than one server per config anyways.