webdevops/azure-devops-exporter

Prometheus exporter for Azure DevOps (VSTS) including agent pools, builds, releases, deployments, pullrequests and repo stats

Azure DevOps Exporter (VSTS)

Prometheus exporter for Azure DevOps (VSTS) for projects, builds, build times (elapsed and queue wait time), agent pool utilization and active pull requests.

Configuration

Usage:
  azure-devops-exporter [OPTIONS]

Application Options:
      --log.debug                             debug mode [$LOG_DEBUG]
      --log.devel                             development mode [$LOG_DEVEL]
      --log.json                              Switch log output to json format [$LOG_JSON]
      --scrape.time=                          Default scrape time (time.duration) (default: 30m) [$SCRAPE_TIME]
      --scrape.time.projects=                 Scrape time for project metrics (time.duration) [$SCRAPE_TIME_PROJECTS]
      --scrape.time.repository=               Scrape time for repository metrics (time.duration) [$SCRAPE_TIME_REPOSITORY]
      --scrape.time.build=                    Scrape time for build metrics (time.duration) [$SCRAPE_TIME_BUILD]
      --scrape.time.release=                  Scrape time for release metrics (time.duration) [$SCRAPE_TIME_RELEASE]
      --scrape.time.deployment=               Scrape time for deployment metrics (time.duration) [$SCRAPE_TIME_DEPLOYMENT]
      --scrape.time.pullrequest=              Scrape time for pullrequest metrics  (time.duration) [$SCRAPE_TIME_PULLREQUEST]
      --scrape.time.stats=                    Scrape time for stats metrics  (time.duration) [$SCRAPE_TIME_STATS]
      --scrape.time.resourceusage=            Scrape time for resourceusage metrics  (time.duration) [$SCRAPE_TIME_RESOURCEUSAGE]
      --scrape.time.query=                    Scrape time for query results  (time.duration) [$SCRAPE_TIME_QUERY]
      --scrape.time.live=                     Scrape time for live metrics (time.duration) (default: 30s) [$SCRAPE_TIME_LIVE]
      --stats.summary.maxage=                 Stats Summary metrics max age (time.duration) [$STATS_SUMMARY_MAX_AGE]
      --azure.tenant-id=                      Azure tenant ID for Service Principal authentication [$AZURE_TENANT_ID]
      --azure.client-id=                      Client ID for Service Principal authentication [$AZURE_CLIENT_ID]
      --azure.client-secret=                  Client secret for Service Principal authentication [$AZURE_CLIENT_SECRET]
      --azuredevops.url=                      Azure DevOps URL (empty if hosted by Microsoft) [$AZURE_DEVOPS_URL]
      --azuredevops.access-token=             Azure DevOps access token [$AZURE_DEVOPS_ACCESS_TOKEN]
      --azuredevops.access-token-file=        Azure DevOps access token (from file) [$AZURE_DEVOPS_ACCESS_TOKEN_FILE]
      --azuredevops.organisation=             Azure DevOps organization [$AZURE_DEVOPS_ORGANISATION]
      --azuredevops.apiversion=               Azure DevOps API version (default: 5.1) [$AZURE_DEVOPS_APIVERSION]
      --azuredevops.agentpool=                Enable scrape metrics for agent pool (IDs) [$AZURE_DEVOPS_AGENTPOOL]
      --whitelist.project=                    Filter projects (UUIDs) [$AZURE_DEVOPS_FILTER_PROJECT]
      --blacklist.project=                    Filter projects (UUIDs) [$AZURE_DEVOPS_BLACKLIST_PROJECT]
      --list.query=                           Pairs of query and project UUIDs in the form: '<queryId>@<projectId>' [$AZURE_DEVOPS_QUERIES]
      --tags.schema=                          Tags to be extracted from builds in the format 'tagName:type' with following types: number,
                                              info, bool [$AZURE_DEVOPS_TAG_SCHEMA]
      --tags.build.definition=                Build definition ids to query tags (IDs) [$AZURE_DEVOPS_TAG_BUILD_DEFINITION]
      --cache.path=                           Cache path (to folder, file://path... or
                                              azblob://storageaccount.blob.core.windows.net/containername or
                                              k8scm://{namespace}/{configmap}}) [$CACHE_PATH]
      --request.concurrency=                  Number of concurrent requests against dev.azure.com (default: 10) [$REQUEST_CONCURRENCY]
      --request.retries=                      Number of retried requests against dev.azure.com (default: 3) [$REQUEST_RETRIES]
      --servicediscovery.refresh=             Refresh duration for servicediscovery (time.duration) (default: 30m)
                                              [$SERVICEDISCOVERY_REFRESH]
      --limit.project=                        Limit number of projects (default: 100) [$LIMIT_PROJECT]
      --limit.builds-per-project=             Limit builds per project (default: 100) [$LIMIT_BUILDS_PER_PROJECT]
      --limit.builds-per-definition=          Limit builds per definition (default: 10) [$LIMIT_BUILDS_PER_DEFINITION]
      --limit.releases-per-project=           Limit releases per project (default: 100) [$LIMIT_RELEASES_PER_PROJECT]
      --limit.releases-per-definition=        Limit releases per definition (default: 100) [$LIMIT_RELEASES_PER_DEFINITION]
      --limit.deployments-per-definition=     Limit deployments per definition (default: 100) [$LIMIT_DEPLOYMENTS_PER_DEFINITION]
      --limit.releasedefinitions-per-project= Limit builds per definition (default: 100) [$LIMIT_RELEASEDEFINITION_PER_PROJECT]
      --limit.build-history-duration=         Time (time.Duration) how long the exporter should look back for builds (default: 48h)
                                              [$LIMIT_BUILD_HISTORY_DURATION]
      --limit.release-history-duration=       Time (time.Duration) how long the exporter should look back for releases (default: 48h)
                                              [$LIMIT_RELEASE_HISTORY_DURATION]
      --server.bind=                          Server address (default: :8080) [$SERVER_BIND]
      --server.timeout.read=                  Server read timeout (default: 5s) [$SERVER_TIMEOUT_READ]
      --server.timeout.write=                 Server write timeout (default: 10s) [$SERVER_TIMEOUT_WRITE]

Help Options:
  -h, --help                                  Show this help message

Authentication

This exporter supports Azure DevOps PAT tokens and ServicePrincipal authentication with Client Secret and (AKS) Workload Identity.

Metrics

Metric	Scraper	Description
azure_devops_stats	live	General scraper stats
azure_devops_agentpool_info	live	Agent Pool informations
azure_devops_agentpool_size	live	Number of agents per agent pool
azure_devops_agentpool_usage	live	Usage of agent pool (used agents; percent 0-1)
azure_devops_agentpool_queue_length	live	Queue length per agent pool
azure_devops_agentpool_agent_info	live	Agent information per agent pool
azure_devops_agentpool_agent_status	live	Status informations (eg. created date) for each agent in a agent pool
azure_devops_agentpool_agent_job	live	Currently running jobs on each agent
azure_devops_project_info	live/projects	Project informations
azure_devops_build_latest_info	live	Latest build information
azure_devops_build_latest_status	live	Latest build status informations
azure_devops_pullrequest_info	pullrequest	Active PullRequests
azure_devops_pullrequest_status	pullrequest	Status informations (eg. created date) for active PullRequests
azure_devops_pullrequest_label	pullrequest	Labels set on active PullRequests
azure_devops_build_info	build	Build informations
azure_devops_build_status	build	Build status infos (queued, started, finished time)
azure_devops_build_stage	build	Build stage infos (duration, errors, warnings, started, finished time)
azure_devops_build_phase	build	Build phase infos (duration, errors, warnings, started, finished time)
azure_devops_build_job	build	Build job infos (duration, errors, warnings, started, finished time)
azure_devops_build_task	build	Build task infos (duration, errors, warnings, started, finished time)
azure_devops_build_definition_info	build	Build definition info
azure_devops_release_info	release	Release informations
azure_devops_release_artifact	release	Release artifcact informations
azure_devops_release_environment	release	Release environment list
azure_devops_release_environment_status	release	Release environment status informations
azure_devops_release_approval	release	Release environment approval list
azure_devops_release_definition_info	release	Release definition info
azure_devops_release_definition_environment	release	Release definition environment list
azure_devops_repository_info	repository	Repository informations
azure_devops_repository_stats	repository	Repository stats
azure_devops_repository_commits	repository	Repository commit counter
azure_devops_repository_pushes	repository	Repository push counter
azure_devops_query_result	live	Latest results of given queries
azure_devops_deployment_info	deployment	Release deployment informations
azure_devops_deployment_status	deployment	Release deployment status informations
azure_devops_stats_agentpool_builds	stats	Number of buildsper agentpool, project and result (counter)
azure_devops_stats_agentpool_builds_wait	stats	Build wait time per agentpool, project and result (summary)
azure_devops_stats_agentpool_builds_duration	stats	Build duration per agentpool, project and result (summary)
azure_devops_stats_project_builds	stats	Number of builds per project, definition and result (counter)
azure_devops_stats_project_builds_wait	stats	Build wait time per project, definition and result (summary)
azure_devops_stats_project_builds_success	stats	Success rating of build per project and definition (summary)
azure_devops_stats_project_builds_duration	stats	Build duration per project, definition and result (summary)
azure_devops_stats_project_release_duration	stats	Release environment duration per project, definition, environment and result (summary)
azure_devops_stats_project_release_success	stats	Success rating of release environment per project, definition and environment (summary)
azure_devops_resourceusage_build	resourceusage	Usage of limited and paid Azure DevOps resources (build)
azure_devops_resourceusage_license	resourceusage	Usage of limited and paid Azure DevOps resources (license)
azure_devops_api_request_*		REST api request histogram (count, latency, statuscCodes)

Prometheus queries

Last 3 failed releases per definition for one project

topk by(projectID,releaseDefinitionName,path) (3,
  azure_devops_release_environment{projectID="XXXXXXXXXXXXXXXX", status!="succeeded", status!="inProgress"}
  * on (projectID,releaseID,environmentID) group_left() (azure_devops_release_environment_status{type="created"})
  * on (projectID,releaseID) group_left(releaseName, releaseDefinitionID) (azure_devops_release_info)
  * on (projectID,releaseDefinitionID) group_left(path, releaseDefinitionName) (azure_devops_release_definition_info)
)

Agent pool usage (without PoolMaintenance)

count by(agentPoolID) (
  azure_devops_agentpool_agent_job{planType!="PoolMaintenance"}
  * on(agentPoolAgentID) group_left(agentPoolID) (azure_devops_agentpool_agent_info)
)
/ on (agentPoolID) group_left() (azure_devops_agentpool_size)
* on (agentPoolID) group_left(agentPoolName) (azure_devops_agentpool_info)

Current running jobs

label_replace(
    azure_devops_agentpool_agent_job{planType!="PoolMaintenance"}
    * on (agentPoolAgentID) group_left(agentPoolID,agentPoolAgentName) azure_devops_agentpool_agent_info
    * on (agentPoolID) group_left(agentPoolName) (azure_devops_agentpool_info)
  , "projectID", "$1", "scopeID", "^(.+)$"
)
* on (projectID) group_left(projectName) (azure_devops_project_info)

Agent pool size

azure_devops_agentpool_info
* on (agentPoolID) group_left() (azure_devops_agentpool_size)

Agent pool size (enabled and online)

azure_devops_agentpool_info
* on (agentPoolID) group_left() (
  count by(agentPoolID) (azure_devops_agentpool_agent_info{status="online",enabled="true"})
)

azure-devops-exporter's Issues

unsupported protocol scheme collect releases

I have a problem with the scrapping only with collect releases because another collects works fine

this is the error.

Thanks for your help.

Build Pipeline Environment details

Hi Team

I've been testing the exporter and its working great.

I'm wondering if it's possible to surface the details for the Build Pipeline environments? I can see all the info for the Release Pipeline environments but since those are considered classic/deprecated we only use the Environments that are part of the build pipelines.

The part of the API I can see that collect that info is:

_apis/distributedtask/environments/{environmentId}/environmentdeploymentrecords?api-version=7.0

issue setting up exporter

I am trying to run this image as a kuberenetes deployment on my local K8. We have a azure server running on other machine . I provided azure environment variables like AZURE_DEVOPS_URL, AZURE_DEVOPS_ORGANISATION and AZURE_DEVOPS_ACCESS_TOKEN, but still getting the following error :

the required flags --azuredevops.access-token' and --azuredevops.organisation' were not specified

Can someone provide us info on how we can setup this up .
Note: Prometheus is also running on my local k8.

minimum required access in readme

Hi thank you for the great work!
Can you include a part in the docs, what is the minimum accesses we need to give to the exporter to see every metrics?
If that's not trivial task, how can we help?

Not an issue : Need to know how to use this

I have setup my self hosted agents

I also have prometheus running in my nomad cluster

How do I use azure-devops-exporter to get my build agents to my prom

Number of Jobs in queue (waiting for free agents) in Agent Pool

Hi

We started to use this Azure DevOps exporter and it is working really well.

We need to have the information on how many jobs are in queue in real time (waiting for a free Agent) per Agent Pool.

We looked to get it but we did not find a way also combining different others metrics

Is it possible to get this metric?

Could you help us on that?

Thanks

Alberto

About Duplicate Data

Hi,

I am using your exporter and it is very powerful. It can grab a lot of information. But I have a couple of issues that are still unresolved. The first is that every time I use exporter. It always crawls a lot of duplicate data. The duplicate data appears once every 15 seconds. I don't really understand why this is happening. Would you please explain it ?

Thanks

Question Metrics labels

Hi,

Thanks for the initiative, nicely written golang, only unittests missing 😉

I was trying out the exporter and checking the metrics, while question araised regarding labels for the metrics and before changing the behaviour wanted just to doublecheck, whenever I am missing anything

for example I have a metric azure_devops_repository_info. Metric example smth like:

azure_devops_repository_info{instance="host.docker.internal:8080",job="prometheus",projectID="fb540d14-e477-487a-8af2-a152aa0a328f",repositoryID="03124f65-c354-4fb6-aeba-956e3f43ccc4",repositoryName="HubDetect"}

I assume, that this ☝️ metric is for keeping the linkage between repositoryID and repositoryName and carries data about to which project repo belongs to

then I have azure_devops_project_info metric, as example

azure_devops_project_info{instance="host.docker.internal:8080",job="prometheus",projectID="fb540d14-e477-487a-8af2-a152aa0a328f",projectName="DIGMOWEB_POC"}     1

☝️ this metric seems like only to keep the linkage between projectID and projectName

then I have azure_devops_repository_commits metric, where I have a real metric about commits made to repository in ADO
example of the metric:

azure_devops_repository_commits{instance="host.docker.internal:8080",job="prometheus",projectID="fb540d14-e477-487a-8af2-a152aa0a328f",repositoryID="03124f65-c354-4fb6-aeba-956e3f43ccc4"}	0

azure_devops_repository_commits metric labels projectID and repositoryID for out of the box, doesnt say a lot, so seems, I need to leftjoin azure_devops_repository_commits metric with azure_devops_repository_commits and azure_devops_project_info

The question - why you are not placing the repoName together with ProjectName to the azure_devops_repository_commits metric in the first place and removing the need of left_joins in prometheus?

Same applies not only for azure_devops_repository_commits, but also for azure_devops_repository_stats, azure_devops_repository_pushes etc.

Support for project names with spaces

When running this service against an Azure DevOps organisation with access to a project with a space in it's name, it will not scrape any information except for the project name, for that project.

Logs:

INFO : main.go:75: Init Azure DevOps exporter v0.7.0 (written by webdevops.io)
INFO : main.go:77: Init Azure connection
INFO : main.go:80: Starting metrics collection
INFO : main.go:81: set scape interval[Default]: 30m0s
INFO : main.go:82: set scape interval[Live]: 30s
INFO : main.go:83: set scape interval[Project]: 30m0s
INFO : main.go:84: set scape interval[Repository]: 30m0s
INFO : main.go:85: set scape interval[PullRequest]: 30m0s
INFO : main.go:86: set scape interval[Build]: 30m0s
INFO : main.go:87: set scape interval[Release]: 30m0s
INFO : main.go:88: set scape interval[Deployment]: 30m0s
INFO : collector_base.go:58: collector[General]: sleeping 30s
INFO : collector_base.go:47: collector[General]: starting metrics collection
INFO : collector_base.go:54: collector[General]: finished metrics collection (duration: 88.3µs)
INFO : main.go:91: Starting http server on :8080
INFO : collector_base.go:47: collector[Repository]: starting metrics collection
INFO : collector_base.go:58: collector[Project]: sleeping 30s
INFO : collector_base.go:47: collector[Project]: starting metrics collection
INFO : collector_base.go:58: collector[AgentPool]: sleeping 30s
INFO : collector_base.go:47: collector[AgentPool]: starting metrics collection
INFO : collector_base.go:58: collector[LatestBuild]: sleeping 30s
INFO : collector_base.go:47: collector[LatestBuild]: starting metrics collection
INFO : collector_base.go:58: collector[PullRequest]: sleeping 30m0s
INFO : collector_base.go:47: collector[PullRequest]: starting metrics collection
INFO : collector_base.go:58: collector[Build]: sleeping 30m0s
INFO : collector_base.go:47: collector[Build]: starting metrics collection
INFO : collector_base.go:58: collector[Release]: sleeping 30m0s
INFO : collector_base.go:47: collector[Release]: starting metrics collection
INFO : collector_base.go:58: collector[Deployment]: sleeping 30m0s
INFO : collector_base.go:47: collector[Deployment]: starting metrics collection
INFO : collector_base.go:58: collector[Repository]: sleeping 30m0s
INFO : collector_base.go:54: collector[Project]: finished metrics collection (duration: 476.7µs)
ERROR: metrics_latest_build.go:68: project[Mikael Lindemann]call[ListLatestBuilds]: Response status code is 404 (expected 200)
ERROR: metrics_build.go:93: project[Mikael Lindemann]call[ListBuildDefinitions]: Response status code is 404 (expected 200)
INFO : collector_base.go:54: collector[LatestBuild]: finished metrics collection (duration: 190.9261ms)
ERROR: metrics_release.go:166: project[Mikael Lindemann]call[ListReleaseDefinitions]: Response status code is 404 (expected 200)
INFO : collector_base.go:54: collector[PullRequest]: finished metrics collection (duration: 308.7521ms)
ERROR: metrics_deployment.go:68: project[Mikael Lindemann]call[ListReleaseDefinitions]: Response status code is 404 (expected 200)
ERROR: metrics_build.go:118: project[Mikael Lindemann]call[ListBuilds]: Response status code is 404 (expected 200)
ERROR: metrics_agentpool.go:150: agentpool[Mikael Lindemann]call[ListAgentQueues]: %!v(MISSING)
INFO : collector_base.go:54: collector[Repository]: finished metrics collection (duration: 374.7094ms)
INFO : collector_base.go:54: collector[Deployment]: finished metrics collection (duration: 449.8707ms)
INFO : collector_base.go:54: collector[Build]: finished metrics collection (duration: 452.6225ms)
ERROR: metrics_agentpool.go:234: project[Mikael Lindemann]call[ListBuildHistory]: Response status code is 404 (expected 200)
INFO : collector_base.go:54: collector[Release]: finished metrics collection (duration: 566.8395ms)
INFO : collector_base.go:54: collector[AgentPool]: finished metrics collection (duration: 620.8595ms)
INFO : collector_base.go:58: collector[General]: sleeping 30s
INFO : collector_base.go:47: collector[General]: starting metrics collection
INFO : collector_base.go:54: collector[General]: finished metrics collection (duration: 96.3µs)
INFO : collector_base.go:58: collector[Project]: sleeping 30s
INFO : collector_base.go:47: collector[LatestBuild]: starting metrics collection
INFO : collector_base.go:58: collector[LatestBuild]: sleeping 30s
INFO : collector_base.go:47: collector[Project]: starting metrics collection
INFO : collector_base.go:58: collector[AgentPool]: sleeping 30s
INFO : collector_base.go:47: collector[AgentPool]: starting metrics collection
INFO : collector_base.go:54: collector[Project]: finished metrics collection (duration: 608µs)
ERROR: metrics_latest_build.go:68: project[Mikael Lindemann]call[ListLatestBuilds]: Response status code is 404 (expected 200)
INFO : collector_base.go:54: collector[LatestBuild]: finished metrics collection (duration: 195.5343ms)
ERROR: metrics_agentpool.go:150: agentpool[Mikael Lindemann]call[ListAgentQueues]: %!v(MISSING)
ERROR: metrics_agentpool.go:234: project[Mikael Lindemann]call[ListBuildHistory]: Response status code is 404 (expected 200)
INFO : collector_base.go:54: collector[AgentPool]: finished metrics collection (duration: 486.1339ms)

GET /metrics:

# HELP azure_devops_agentpool_info Azure DevOps agentpool
# TYPE azure_devops_agentpool_info gauge
azure_devops_agentpool_info{agentPoolID="1",agentPoolName="Default",agentPoolType="automation",isHosted="false"} 1
azure_devops_agentpool_info{agentPoolID="2",agentPoolName="Hosted",agentPoolType="automation",isHosted="true"} 1
azure_devops_agentpool_info{agentPoolID="3",agentPoolName="Hosted VS2017",agentPoolType="automation",isHosted="true"} 1
azure_devops_agentpool_info{agentPoolID="4",agentPoolName="Hosted Windows Container",agentPoolType="automation",isHosted="true"} 1
azure_devops_agentpool_info{agentPoolID="5",agentPoolName="Hosted macOS",agentPoolType="automation",isHosted="true"} 1
azure_devops_agentpool_info{agentPoolID="6",agentPoolName="Hosted Ubuntu 1604",agentPoolType="automation",isHosted="true"} 1
azure_devops_agentpool_info{agentPoolID="7",agentPoolName="Hosted Windows 2019 with VS2019",agentPoolType="automation",isHosted="true"} 1
# HELP azure_devops_agentpool_size Azure DevOps agentpool
# TYPE azure_devops_agentpool_size gauge
azure_devops_agentpool_size{agentPoolID="1"} 0
azure_devops_agentpool_size{agentPoolID="2"} 1
azure_devops_agentpool_size{agentPoolID="3"} 1
azure_devops_agentpool_size{agentPoolID="4"} 1
azure_devops_agentpool_size{agentPoolID="5"} 1
azure_devops_agentpool_size{agentPoolID="6"} 1
azure_devops_agentpool_size{agentPoolID="7"} 1
# HELP azure_devops_build_definition_info Azure DevOps build definition
# TYPE azure_devops_build_definition_info gauge
azure_devops_build_definition_info{buildDefinitionID="1",buildDefinitionName="lindemann-CI",buildNameFormat="",path="\\",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",url="https://dev.azure.com/mikaellindemann/fd87d7c7-4920-45da-8ddb-10023926482b/_build/definition?definitionId=1"} 1
azure_devops_build_definition_info{buildDefinitionID="2",buildDefinitionName="other - clone lindemann",buildNameFormat="",path="\\",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",url="https://dev.azure.com/mikaellindemann/fd87d7c7-4920-45da-8ddb-10023926482b/_build/definition?definitionId=2"} 1
# HELP azure_devops_build_info Azure DevOps build
# TYPE azure_devops_build_info gauge
azure_devops_build_info{agentPoolID="6",buildDefinitionID="1",buildID="5",buildName="lindemann-CI",buildNumber="5",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",reason="manual",requestedBy="Mikael Lindemann",result="succeeded",sourceBranch="refs/heads/master",sourceVersion="474b08c147fbe869a379f8e8dcf16f6fa9d12980",status="completed",url="https://dev.azure.com/mikaellindemann/fd87d7c7-4920-45da-8ddb-10023926482b/_build/results?buildId=5"} 1
azure_devops_build_info{agentPoolID="6",buildDefinitionID="2",buildID="6",buildName="other - clone lindemann",buildNumber="6",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",reason="manual",requestedBy="Mikael Lindemann",result="canceled",sourceBranch="refs/heads/master",sourceVersion="",status="completed",url="https://dev.azure.com/mikaellindemann/fd87d7c7-4920-45da-8ddb-10023926482b/_build/results?buildId=6"} 1
azure_devops_build_info{agentPoolID="6",buildDefinitionID="2",buildID="7",buildName="other - clone lindemann",buildNumber="7",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",reason="manual",requestedBy="Mikael Lindemann",result="failed",sourceBranch="refs/heads/master",sourceVersion="763d4d7310e7cf8fa0d6abb81e021c465d4cb405",status="completed",url="https://dev.azure.com/mikaellindemann/fd87d7c7-4920-45da-8ddb-10023926482b/_build/results?buildId=7"} 1
azure_devops_build_info{agentPoolID="6",buildDefinitionID="2",buildID="8",buildName="other - clone lindemann",buildNumber="8",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",reason="manual",requestedBy="Mikael Lindemann",result="failed",sourceBranch="refs/heads/master",sourceVersion="763d4d7310e7cf8fa0d6abb81e021c465d4cb405",status="completed",url="https://dev.azure.com/mikaellindemann/fd87d7c7-4920-45da-8ddb-10023926482b/_build/results?buildId=8"} 1
# HELP azure_devops_build_latest_info Azure DevOps build (latest)
# TYPE azure_devops_build_latest_info gauge
azure_devops_build_latest_info{agentPoolID="6",buildDefinitionID="1",buildID="5",buildName="lindemann-CI",buildNumber="5",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",reason="manual",requestedBy="Mikael Lindemann",result="succeeded",sourceBranch="refs/heads/master",sourceVersion="474b08c147fbe869a379f8e8dcf16f6fa9d12980",status="completed",url="https://dev.azure.com/mikaellindemann/fd87d7c7-4920-45da-8ddb-10023926482b/_build/results?buildId=5"} 1
azure_devops_build_latest_info{agentPoolID="6",buildDefinitionID="2",buildID="8",buildName="other - clone lindemann",buildNumber="8",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",reason="manual",requestedBy="Mikael Lindemann",result="failed",sourceBranch="refs/heads/master",sourceVersion="763d4d7310e7cf8fa0d6abb81e021c465d4cb405",status="completed",url="https://dev.azure.com/mikaellindemann/fd87d7c7-4920-45da-8ddb-10023926482b/_build/results?buildId=8"} 1
# HELP azure_devops_build_latest_status Azure DevOps build (latest)
# TYPE azure_devops_build_latest_status gauge
azure_devops_build_latest_status{buildID="5",buildNumber="5",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="finished"} 1.548048423e+09
azure_devops_build_latest_status{buildID="5",buildNumber="5",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="queued"} 1.548048399e+09
azure_devops_build_latest_status{buildID="5",buildNumber="5",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="started"} 1.548048405e+09
azure_devops_build_latest_status{buildID="8",buildNumber="8",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="finished"} 1.553153677e+09
azure_devops_build_latest_status{buildID="8",buildNumber="8",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="queued"} 1.553153624e+09
azure_devops_build_latest_status{buildID="8",buildNumber="8",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="started"} 1.553153628e+09
# HELP azure_devops_build_status Azure DevOps build
# TYPE azure_devops_build_status gauge
azure_devops_build_status{buildID="5",buildNumber="5",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="finished"} 1.548048423e+09
azure_devops_build_status{buildID="5",buildNumber="5",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="jobDuration"} 18.7761264
azure_devops_build_status{buildID="5",buildNumber="5",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="queued"} 1.548048399e+09
azure_devops_build_status{buildID="5",buildNumber="5",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="started"} 1.548048405e+09
azure_devops_build_status{buildID="6",buildNumber="6",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="finished"} 1.553153521e+09
azure_devops_build_status{buildID="6",buildNumber="6",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="jobDuration"} 0
azure_devops_build_status{buildID="6",buildNumber="6",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="queued"} 1.553153521e+09
azure_devops_build_status{buildID="6",buildNumber="6",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="started"} 1.553153521e+09
azure_devops_build_status{buildID="7",buildNumber="7",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="finished"} 1.553153571e+09
azure_devops_build_status{buildID="7",buildNumber="7",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="jobDuration"} 19.6865878
azure_devops_build_status{buildID="7",buildNumber="7",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="queued"} 1.553153546e+09
azure_devops_build_status{buildID="7",buildNumber="7",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="started"} 1.553153551e+09
azure_devops_build_status{buildID="8",buildNumber="8",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="finished"} 1.553153677e+09
azure_devops_build_status{buildID="8",buildNumber="8",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="jobDuration"} 49.5133065
azure_devops_build_status{buildID="8",buildNumber="8",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="queued"} 1.553153624e+09
azure_devops_build_status{buildID="8",buildNumber="8",projectID="fd87d7c7-4920-45da-8ddb-10023926482b",type="started"} 1.553153628e+09
# HELP azure_devops_project_info Azure DevOps project
# TYPE azure_devops_project_info gauge
azure_devops_project_info{projectID="76541ec8-e5fe-4900-ac6d-1da10fa68675",projectName="Mikael Lindemann"} 1
azure_devops_project_info{projectID="d2abdbf7-4ffb-4b8f-9e3a-19ba915ea5a4",projectName="mikael"} 1
azure_devops_project_info{projectID="fd87d7c7-4920-45da-8ddb-10023926482b",projectName="lindemann"} 1
# HELP azure_devops_repository_commits Azure DevOps repository commits
# TYPE azure_devops_repository_commits counter
azure_devops_repository_commits{projectID="d2abdbf7-4ffb-4b8f-9e3a-19ba915ea5a4",repositoryID="4b074828-a086-43f9-98c7-6a359230520c"} 0
azure_devops_repository_commits{projectID="fd87d7c7-4920-45da-8ddb-10023926482b",repositoryID="11aabd63-4c2c-42a0-94ee-93cdc45039c2"} 0
azure_devops_repository_commits{projectID="fd87d7c7-4920-45da-8ddb-10023926482b",repositoryID="c3b3ce7b-ac9c-4dff-bd71-e099d506f945"} 0
# HELP azure_devops_repository_info Azure DevOps repository
# TYPE azure_devops_repository_info gauge
azure_devops_repository_info{projectID="d2abdbf7-4ffb-4b8f-9e3a-19ba915ea5a4",repositoryID="4b074828-a086-43f9-98c7-6a359230520c",repositoryName="mikael"} 1
azure_devops_repository_info{projectID="fd87d7c7-4920-45da-8ddb-10023926482b",repositoryID="11aabd63-4c2c-42a0-94ee-93cdc45039c2",repositoryName="other"} 1
azure_devops_repository_info{projectID="fd87d7c7-4920-45da-8ddb-10023926482b",repositoryID="c3b3ce7b-ac9c-4dff-bd71-e099d506f945",repositoryName="lindemann"} 1
# HELP azure_devops_repository_pushes Azure DevOps repository pushes
# TYPE azure_devops_repository_pushes counter
azure_devops_repository_pushes{projectID="d2abdbf7-4ffb-4b8f-9e3a-19ba915ea5a4",repositoryID="4b074828-a086-43f9-98c7-6a359230520c"} 0
azure_devops_repository_pushes{projectID="fd87d7c7-4920-45da-8ddb-10023926482b",repositoryID="11aabd63-4c2c-42a0-94ee-93cdc45039c2"} 0
azure_devops_repository_pushes{projectID="fd87d7c7-4920-45da-8ddb-10023926482b",repositoryID="c3b3ce7b-ac9c-4dff-bd71-e099d506f945"} 0
# HELP azure_devops_repository_stats Azure DevOps repository
# TYPE azure_devops_repository_stats gauge
azure_devops_repository_stats{projectID="fd87d7c7-4920-45da-8ddb-10023926482b",repositoryID="11aabd63-4c2c-42a0-94ee-93cdc45039c2",type="size"} 713
azure_devops_repository_stats{projectID="fd87d7c7-4920-45da-8ddb-10023926482b",repositoryID="c3b3ce7b-ac9c-4dff-bd71-e099d506f945",type="size"} 710
# HELP azure_devops_stats Azure DevOps statistics
# TYPE azure_devops_stats gauge
azure_devops_stats{name="dev.azure.com",type="concurrency"} 0
azure_devops_stats{name="dev.azure.com",type="requests"} 4
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.58e-05
go_gc_duration_seconds{quantile="0.25"} 1.61e-05
go_gc_duration_seconds{quantile="0.5"} 3.67e-05
go_gc_duration_seconds{quantile="0.75"} 6.22e-05
go_gc_duration_seconds{quantile="1"} 7.04e-05
go_gc_duration_seconds_sum 0.0002012
go_gc_duration_seconds_count 5
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 19
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.12"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.251384e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.4173736e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.448403e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 237983
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.0009346921986869198
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 2.38592e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 2.251384e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 6.2062592e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 4.489216e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 17915
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 0
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.6551808e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.5532615574351516e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 255898
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 3472
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 71136
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 81920
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 720677
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 557056
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 557056
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.1762168e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 9
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.16
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 10
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.4311424e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.55326155468e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.16502528e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 0
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

Note that there is almost no data for projectID=76541ec8-e5fe-4900-ac6d-1da10fa68675.

How do I run the exporter on my Azure Devops?

Hi, I'm running Prometheus on Docker, I've already put the settings in my prometheus.yml file below:

global:
scrape_interval: 15s
evaluation_interval: 15s

alerting:
alertmanagers:
- static_configs:
- targets:

'evaluation_interval'.
rule_files:

- "first_rules.yml"

- "second_rules.yml"

• job_name: "prometheus"

  static_configs:

  • targets: ["localhost:9090"]

• job_name: azure-devops-exporter
  honor_timestamps: true
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:

  • targets:
    • prometheus-azure-devops-exporter.default.svc:8080

Variables by examples
AZURE_DEVOPS_URL= https://dev.azure.com/company

Should I add these variables to the Prometheus container settings before running?

Retrieve assigned request for agent

I am trying to retrieve all agents and see if they have a currently assigned request, however if I join the two metrics:

azure_devops_agentpool_agent_info
+ on(agentPoolAgentID) group_left() (azure_devops_agentpool_agent_job)

it only returns agents that have an assigned request but not the rest. I am trying to graph agent pool utilisation and need a way to find used agents / total agents.
Thanks

Grafana dashboard

Few people have raised issue for grafana dashboard but it doesn't addressed clearly. Would you please clarify on this point. Do you have any grafana json for this project? It'll be really helpful to entire community on top of fetching data into prometheus. Or direct us to right link/doc.

Builds metrics collection time

Not sure that understand code logic here:

azure-devops-exporter/metrics_stats.go

Line 210 in f84fc36

minTime := *m.CollectorReference.collectionLastTime

Based on this code we get last builds for project stats based on --scrape.time interval which is 30 mins by default.
Is it intended? Why we are not using --limit.build-history-duration like here in example?

azure-devops-exporter/metrics_build.go

Line 121 in f84fc36

minTime := time.Now().Add(-opts.Limit.BuildHistoryDuration)

Is there an way to fetch all projects from one organization by default

Is there an way to fetch all projects from one organization by default and same org name can be passed as runtime args . Currently i'm not able to achieve this

Question: How to properly setup so it scrapes all metrics on the README

I think I have properly configured my environment. I am running GKE and I have a Deployment with the docker image and the variables:

      - name: AZURE_DEVOPS_ACCESS_TOKEN
        value: "myTokenHere"
      - name: AZURE_DEVOPS_ORGANISATION
        value: "myOrgHere"
      - name: AZURE_DEVOPS_FILTER_PROJECT
        value: "myProjectHere"

I have properly configured prometheus to scrape metrics from the Pod, here is a screenshot of my Prometheus.

As you can see it only scrapes the: "azure_devops_stats" metric. How do I get the other metrics?

401 on _apis/Contribution/dataProviders/query?

Hi!

Thank you for this great tool!

In the logs I systematically get an error at startup:

{"level":"error","caller":"azure-devops-exporter/metrics_resourceusage.go:59","msg":"response status code is 401 (expected 200), url: https://dev.azure.com/***/_apis/Contribution/dataProviders/query?api-version=5.1-preview.1","collector":"ResourceUsage","stacktrace":"
main.(*MetricsCollectorResourceUsage).collectResourceUsageAgent
/go/src/github.com/webdevops/azure-devops-exporter/metrics_resourceusage.go:59
main.(*MetricsCollectorResourceUsage).Collect
/go/src/github.com/webdevops/azure-devops-exporter/metrics_resourceusage.go:53
github.com/webdevops/go-common/prometheus/collector.(*Collector).collectRun.func1
/go/pkg/mod/github.com/webdevops/[email protected]/prometheus/collector/collector.go:330
"}

Do you know which permission is needed to access this API?
I found this documentation, but it didn't help much..

I'm actually not sure what impact this error has on the tool.

Skip disabled repositories

We have a couple of disabled repositories and they yield errors in the log:

{
   "collector":"Repository",
   "file":"metrics_repository.go:127",
   "func":"collectRepository",
   "level":"error",
   "msg":"response status code is 404 (expected 200), url: https://dev.azure.com/gropyus/_apis/git/repositories/0ea2ea25-0d9d-4b4f-b3df-fbef5da3063d/commits?searchCriteria.fromDate=2021-10-07T06%3A03%3A51Z\u0026api-version=5.1",
   "project":"architecture",
   "repository":"base-image"
}

Even though it's not a big deal, it would be nice if disabled repositories were skipped by default.

[PromQL] How to use these metrics?

I've managed to find some useful visualizations of our devops data, but it has been hard due to the nature of the format in which this exporter scrapes.

I managed to list average build times per definition

avg(
  azure_devops_build_status{type="jobDuration"}
    + on(buildID) group_left(buildDefinitionID) (0 *azure_devops_build_info)
) by (buildDefinitionID)

And Ive had a hard time finding consistent data for current build count. for exemaple, these two metrics give off different data (probably due to the scrape timing being different)

azure_devops_stats{name="Build"}

rate(azure_devops_agentpool_builds[2m])

Would you consider adding some query samples or even a simple grafana dashboard to get us started exploring your metrics exporter?

[feauture request] Provide latest release status metric

I think, a metric similar to azure_devops_build_latest_status would be also useful for releases.

In this way someone would be able to see the status of the last release for every Release definition and figure out if a release is failing or not.

Number of failed builds

Wouldn't it be a good idea to add something like the number of consecutive build results (successful/failed/cancelled) to the latest status for a build definition?

Try https://github.com/microsoft/azure-devops-go-api

Gauge azure_devops_release_environment_status jobDuration is 0

When looking at jobDuration of azure_devops_release_environment_status it appears to be 0.

azure_devops_release_environment_status{type="jobDuration"}

jobDuration is populated for azure_devops_build_status

azure_devops_build_status{type="jobDuration"}

Is it possible to get this property populated for releases?

rate limit issue

Hi There,
When I run with default settings I got rate limit exceeded issue in my production environment.
How specific time values we need to change to avoid this issue?
If I want to poll every day only once what parameter should I use and value?

Build Test Results or Count

Would it be possible to get the number of tests ran during a build?

Pool queue length is not calculated correctly and always returns 0

In https://github.com/webdevops/azure-devops-exporter/blob/8b8b1a2cd592097b47b26786602e3d180a38216d/metrics_agentpool.go#L241C4-L241C4, the agentPoolQueueLength is calculated by checking for the presence of an assignTime field in the result of the jobrequests query.

This always yields a queue length of 0 for us. I believe that the way it is calculated is incorrect.

The screenshot above shows the result of the jobrequests query for us, and as you can see, the assignTime is always present, even for jobs that have not yet been executed (the one with ID 5).

We could check for the presence of the result field, although that would also count jobs that are currently running, I believe (which would be fine for me).

unsupported protocol scheme

v0.7.2 via docker

docker run --rm -e SERVER_BIND=":8180" -e AZURE_DEVOPS_ACCESS_TOKEN="<token>" -e AZURE_DEVOPS_URL="http://<tfs2018_host>:8080/tfs" -e AZURE_DEVOPS_ORGANISATION="somedomain.com" webdevops/azure-devops-exporter:latest

Seeing the following error. We do not have https turned on, this is an internal only box. Maybe someday we will... but for now, can we support this?

RESTY 2019/04/14 23:03:22 WARNING - Using Basic Auth in HTTP mode is not secure.
RESTY 2019/04/14 23:03:22 ERROR Get //%2Fcf50689a-3dca-47b7-9e2c-69371f767be6%2F_apis%2Fbuild%2Fbuilds%3Fapi-version=4.1&minTime=2019-04-14T23%253A02%253A52Z/cf50689a-3dca-47b7-9e2c-69371f767be6/_apis/build/builds?api-version=4.1&minTime=2019-04-14T23%3A02%3A52Z: unsupported protocol scheme "", Attempt 1
RESTY 2019/04/14 23:03:22 WARNING - Using Basic Auth in HTTP mode is not secure.
RESTY 2019/04/14 23:03:22 ERROR Get //%2Fcf50689a-3dca-47b7-9e2c-69371f767be6%2F_apis%2Fbuild%2Fbuilds%3Fapi-version=4.1&minTime=2019-04-14T23%253A02%253A52Z/cf50689a-3dca-47b7-9e2c-69371f767be6/_apis/build/builds?api-version=4.1&minTime=2019-04-14T23%3A02%3A52Z: unsupported protocol scheme "", Attempt 2
RESTY 2019/04/14 23:03:22 WARNING - Using Basic Auth in HTTP mode is not secure.
RESTY 2019/04/14 23:03:22 ERROR Get //%2Fcf50689a-3dca-47b7-9e2c-69371f767be6%2F_apis%2Fbuild%2Fbuilds%3Fapi-version=4.1&minTime=2019-04-14T23%253A02%253A52Z/cf50689a-3dca-47b7-9e2c-69371f767be6/_apis/build/builds?api-version=4.1&minTime=2019-04-14T23%3A02%3A52Z: unsupported protocol scheme "", Attempt 3

This spams the logs over and over again. Turning on verbose logging doesn't seem to yield any additional details.

Thanks,
Drew

Question: Not sure how or if we can pull DORA 4 (leadtime, change failure rate, recovery)

Looking for solution to help solve obtaining the DORA 4 metrics (lead time, deployment frequency, change failure rate, time to restore).

It looks like deployment frequency is doable here. But I can seem to make the right joins or correlate the right fields to get the lead time metric. Change failure would be the same. Which fields, based on commits through to release would correlate. Not sure how this would be accomplished in the current state.

Secret Access Token

Maybe I'm missing a way to do this, but I'm trying to keep my Azure DevOps Access Token out of the docker-compose file. In other instances, I've stored it as a "secret", then create a custom entrypoint that set's the env variable based on that secret. Here's an example:

docker-compose.yml

version: '3.5'
services:
  azure-devops-exporter:
    image: webdevops/azure-devops-exporter:21.5.0
    container_name: azure-devops-exporter
    entrypoint: /entrypoint
    environment:
      - AZURE_DEVOPS_URL=https://dev.azure.com/org
      - AZURE_DEVOPS_ORGANISATION=org
      - SCRAPE_TIME_PULLREQUEST=30s
    restart: unless-stopped
    volumes:
      - ./entrypoint.sh:/entrypoint
    ports:
      - 8080:8080
    secrets:
      - devops_token
secrets:
  devops_token:
    file: ./secrets/devops_token

entrypoint.sh

#!/bin/bash -e

export AZURE_DEVOPS_ACCESS_TOKEN=$(cat /run/secrets/devops_token)

/azure-devops-exporter

The problem is, because this is a "distroless" container there is no shell to execute the custom entrypoint. Is there another way to pass in my access token without putting it in the docker-compose file? I know some other images support putting "__FILE" at the end of the env variable to pass in a file for the variable like this:

environment:
  - AZURE_DEVOPS_ACCESS_TOKEN__FILE=/run/secrets/devops_token

But that doesn't look like it's supported in this image.

Thanks for the help!

Unable to see lot of metrics after adding the exporter

Hi ,

Recently I'm trying to build a dashboard in grafana for each project we have in one organization and building release pipeline deployment status and current release definition details into it . I deployed the exporter as docker instance and integrated with prometheus . Able to see other metrics such as agent build , build info etc but the below metrics are not present when i try to query it . please help me in fixing this

azure_devops_release_info	release	Release informations
azure_devops_release_artifact	release	Release artifcact informations
azure_devops_release_environment	release	Release environment list
azure_devops_release_environment_status	release	Release environment status informations
azure_devops_release_approval	release	Release environment approval list
azure_devops_release_definition_info	release	Release definition info
azure_devops_release_definition_environment	release	Release definition environment list

None of the release metrics are available in UI and in metric log as well

how to store metrics to external prometheus database

Hi There,
I was able to run the docker image and able to see the collected metrics information but I dont see any prometheus url to query ?
is it possible to push these metrics to my own prometheus databse ?

visualstudio.com

We have an on-prem VSTS solution, you seem to be hard-setting the domain here:

azure-devops-exporter/src/azure-devops-client/main.go

Line 82 in 52720db

c.restClient.SetHostURL(fmt.Sprintf("https://%v.visualstudio.com/", *c.organization))

Can we stop doing that and allow us to specify the full URL?

Thanks,
Drew

Azure job wait-time per pool

We have a use case where we need to get the wait time of the jobs in a pool
We also are looking for a way to get wait time for a specific build in a pool
like azure_devops_stats_agentpool_builds_wait_sum

The documentation is rather vague, how exactly are we supposed to use it? I am new to promethous as a whole

This is the intended result

Vulnerabilities

When scanning the latest docker image, I'm getting these vulnerabilities reported.
Both have a fixed version, i.e. a simple update should suffice:

azure-devops-exporter (gobinary)
================================
Total: 2 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 1, CRITICAL: 0)

┌──────────────────┬────────────────┬──────────┬────────────────────────────────────┬─────────────────────────────────────┬──────────────────────────────────────────────────────────┐
│     Library      │ Vulnerability  │ Severity │         Installed Version          │            Fixed Version            │                          Title                           │
├──────────────────┼────────────────┼──────────┼────────────────────────────────────┼─────────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ golang.org/x/net │ CVE-2022-41721 │ HIGH     │ v0.0.0-20220909164309-bea034e7d591 │ 0.1.1-0.20221104162952-702349b0e862 │ A request smuggling attack is possible when using        │
│                  │                │          │                                    │                                     │ MaxBytesHandler. Whe ...                                 │
│                  │                │          │                                    │                                     │ https://avd.aquasec.com/nvd/cve-2022-41721               │
│                  ├────────────────┼──────────┤                                    ├─────────────────────────────────────┼──────────────────────────────────────────────────────────┤
│                  │ CVE-2022-41717 │ MEDIUM   │                                    │ 0.4.0                               │ golang: net/http: An attacker can cause excessive memory │
│                  │                │          │                                    │                                     │ growth in a Go...                                        │
│                  │                │          │                                    │                                     │ https://avd.aquasec.com/nvd/cve-2022-41717               │
└──────────────────┴────────────────┴──────────┴────────────────────────────────────┴─────────────────────────────────────┴──────────────────────────────────────────────────────────┘

Prometheus exporter for Azure DevOps Server

Any plan to support it for Azure DevOps Server?

We are interested in getting statistics of build times (elapsed and queue wait time), agent pool utilization,... but in Azure DevOps Server 2019.

If it is not possible, is there any other alternative?

Thank you in advance

ADO Build metrics granularity

Hi,
What do you think about increasing granularity level for build metrics?
I mean add additional metrics for stage/job/tasks objects in ADO.
From this api:
https://docs.microsoft.com/en-us/rest/api/azure/devops/build/timeline/get?view=azure-devops-rest-6.1

Access tokens expiration metrics

Hello,

Because this exporter will probably run on a "technical account" it will be not always obvious to keep attention on azure DevOps email or being in a mailing list.
So a metrics to expose at least the current user (the technical account used for metrics collect) access tokens expiration date would be super useful.

Create a new environment to support the initial date to get the history

It is very good, if you have an environment that we can set how long the exporter should look back.

Example: I need to get the history of commits from all repos, but today I can't do that.

Could you add that?

azure_devops_stats_project_builds_duration has a value of NaN

It appears that the value for azure_devops_stats_project_builds_duration is NaN.

Had a quick look through the stats exporter and couldn't really see what was going on.

Any ideas?

Missing azure_devops_agentpool_agent metrics

Hi,

I seem to be missing the following metrics,

azure_devops_agentpool_agent_status
azure_devops_agentpool_agent_job
azure_devops_agentpool_agent_info

I'm using the following flags when starting the exporter
- "--verbose"
- "--bind=:9393"
- "--azuredevops.access-token=xxxxx"
- "--azuredevops.organisation=xxxxx"
- "--blacklist.project=xxxxx"

I've set the PAT to have full access.

I'm using versio 0.8.4

Thanks

Azure DevOps job info

Hi

Is it possible to have the same info as in the metric
azure_devops_build_info

but for the job?

Because we would like to have information of all the jobs run in a agent with the status (failed, succeeded, canceled)

Thanks

Alberto

Certain Metrics missing from export

Hello, I am currently trying to use this exporter to collect metrics for our Azure DevOps organization. I have been able to successfully collect metrics using this docker image, but certain metrics are not present. Here are example metrics I have found that is completely missing (some of these are included in the example queries):

azure_devops_agentpool_agent_info
azure_devops_agentpool_usage

Is there a setting I have wrong?

How to restrict the program to specific metric while we pull from AZ devops

Hi Team,

First a fall, thanks a million for creating such a useful repo and sharing same with our people. Here, I like to know how can I restrict the metric which I pull from azure devops. Currently, we are getting a lot of irrelevant data and we are very sure in a long run this will cause a issue in storage part. Hence, from the beginning, we like to be more cautious about what the data we are going to pull to save space in HD. Pls let me know your comments.

Pls give step by step assistance. If possible, add screenshot or document. So that, people whoever like me get benefit.

Currently, I converted this repo into exe file using go build
Environment - WIndows

Build and Install instructions

How is this built and installed?

Why is there no binary in the releases?

Build agent metrics

Hi!

Are there such metrics or may be can they be added in future?

1. build time for build definition by build agent (or azure_devops_build_info but with agentPoolAgentID label ?)
2. builds count per build agents in pool (like azure_devops_stats_agentpool_builds but divided by agentPoolAgentID)

Thank you

[feauture request] monitoring free minutes in public ado agents

I don't see metric from pipelines->parallel jobs. It will be very usefull who want monitoring free public minutes in own projects :)

Get Projects issue

Hi all,

We are trying to get all projects from our Azure DevOps organization with azure-devops-exporter. However, it lists only 1000 projects, while in fact there are much more of them. The limit of projects was set to 2000, but that didn't help.

Could someone guide which code parts should be changed to overcome this issue? (main.go and project.go? Which functions?) I believe it's a missing paging functionality issue.

Br,

Self Hosted ADO fails due to misformed URL

The exporter throws errors trying to hit API endpoints due to a misformed URL when trying to use on-premises ADO. This happens because organization isn't added to the url path when --azuredevops.url= is defined.

/azure-devops-exporter --azuredevops.url=https://tfs.mydomain.biz --azuredevops.access-token=MyPATToken --azuredevops.organisation=DefaultCollection

ERRO[0016] response status code is 404 (expected 200), url: https://tfs.mydomain.biz/e7065b43-d240-4a6e-a9d7-aff9d09ec266/_apis/release/definitions?api-version=2.0&isDeleted=false&$top=100&$expand=environments,lastRelease  collector=Deployment project=Briefcase

Since we are specifying DefaultCollection the API endpoint should look like this:

https://tfs.mydomain.biz/DefaultCollection/e7065b43-d240-4a6e-a9d7-aff9d09ec266/_apis/release/definitions?api-version=2.0&isDeleted=false&$top=100&$expand=environments,lastRelease  collector=Deployment project=Briefcase

Missing metrics (and incorrect agentPoolId)

I don't seem to be getting the azure_devops_agentpool_queue_length metric.

I only see these metrics

I can't see any errors in the exporters logs. Is there any guidance on what permissions to enable for the PAT?

Get: auto redirect is disabled - azure-devops-exporter k8s

Hello,

could you please help, I am trying to deploy this exporter in k8s cluster and got CrashLoopBackOff on the pod.
Seems like exporter is not able connect to Azure REST API, I generated PAT with full access permissions.
From the kctl logs I got:

_

RESTY 2022/07/12 12:05:26 ERROR Get "https://spsprodweu1.vssps.visualstudio.com/_signin?realm=dev.azure.com&reply_to=https%3A%2F%2Fdev.azure.com%2F(Organization_Name)%2F_apis%2Fprojects%3F%2524top%3D100%26api-version%3D5.1&redirect=1&hid=cc01a906-d406-467d-bb70-56ce6c4df2a5&context=eyJodCI6MiwiaGlkIjoiZjYyMjRhNzAtMzFhYi00NDZlLTgxZWQtYzcwZDVlNDY1MmYwIiwicXMiOnt9LCJyciI6IiIsInZoIjoiIiwiY3YiOiIiLCJjcyI6IiJ90#ctx=eyJTaWduSW5Db29raWVEb21haW5zIjpbImh0dHBzOi8vbG9naW4ubWljcm9zb2Z0b25saW5lLmNvbSIsImh0dHBzOi8vbG9naW4ubWljcm9zb2Z0b25saW5lLmNvbSJdfQ2": auto redirect is disabled, Attempt 1

_

Regards,
M

PS. This link is working correctly in web browser - it is redirecting to Azure REST API (https://dev.azure.com/(Organization_name)/_apis/projects?%24top=100&api-version=5.1

Closed pull requests

It looks like the pull request collector is hard coded to only pull back active pull requests. Any objection to also including closed? Would be useful to be able to capture how many pull requests have been completed, time to complete, etc.