jenkinsci/opentelemetry-plugin

Monitor and observe Jenkins with OpenTelemetry.

Jenkins OpenTelemetry Plugin

Introduction

Monitor and observe Jenkins with OpenTelemetry.

Visualize jobs and pipelines executions as distributed traces:

Example pipeline execution trace of a SpringBoot Maven going application

Visualize Jenkins and pipeline health indicators:

Example Kibana dashboard of the Jenkins and CI jobs health

Troubleshoot Jenkins performances with distributed tracing of HTTPs requests:

Example Jenkins HTTP trace

Architecture

Using the OpenTelemetry Collector, you can use many monitoring backends to monitor Jenkins such as Jaeger, Zipkin, Prometheus, Elastic Observability and many others listed here.

Here are example architectures with Elastic, Jaeger, and Prometheus:

CI/CD Observability with Jaeger and Prometheus	CI/CD Observability with Elastic

Getting started

• Set up an OpenTelemetry endpoint such as the OpenTelemetry Collector
• Install the Jenkins OpenTelemetry plugin
• Configure the Jenkins OpenTelemetry plugin navigating to the "Manage Jenkins / Configure System" screen. In the OpenTelemetry section define:
  • "OTLP Endpoint": the hostname and port of the OpenTelemetry GRPC Protocol (OTLP GRPC) endpoint, typically an OpenTelemetry Collector or directly an Observability backend that supports the OTLP GRPC protocol
  • "Authentication": authentication mechanism used by your OTLP Endpoint
    • "Header Authentication" : name of the authentication header if header based authentication is used.
    • "Bearer Token Authentication": Bearer token when using header based authentication. Note that Elastic APM token authentication uses a "Bearer Token Authentication".
    • "No Authentication"
  • Check "Export OpenTelemetry configuration as environment variables" to easily integrate visibility in other build tools (see the otel-cli, the OpenTelemetry Maven extension, the OpenTelemetry Ansible integration...)
  • Visualization: add the backend used to visualize job executions as traces.
    • Elastic Observability
    • Jaeger
    • Zipkin
    • Custom Observability backend for other visualization solutions
• Set up Jenkins health dashboards on your OpenTelemetry metrics visualization solution. See details including guidance for Elastic Kibana here.

Example Jenkins OpenTelemetry configuration

Setup and Configuration

For details to set up Jenkins with Elastic, Jaeger or Prometheus, to configure the integration including using Jenkins Configuration as Code, see Setup and Configuration.

Troubleshooting and Optimizing Jenkins Jobs and Pipelines Using Tracing on the Builds

For details on how to explore and troubleshoot jobs and pipelines builds as traces, see Traces of Jobs and Pipeline Builds.

Example pipeline execution trace of a SpringBoot app built with Maven going through security checks with Snyk, deployed on a Maven repository and published as a Docker image

Troubleshooting Jenkins Performances Using Tracing on the HTTP Requests of the Jenkins Controller

For details to set up Jenkins with Elastic, Jaeger or Prometheus, to configure the integration including using Jenkins Configuration as Code, see Setup and Configuration.

Jenkins Security

Monitor access to Jenkins to detect anomalous behaviours.

For details, see the security logs, metrics, and trace attributes here.

Storing Jenkins Pipeline Logs in an Observability Backend

For details on how to store Jenkins pipelines build logs in an Observability backend like Elastic, see Storing Jenkins Pipeline Logs in an Observability Backend though OpenTelemetry.

Storing Jenkins pipeline logs in Elasticsearch and visualizing logs both in Kibana and through Jenkins GUI

Other CI/CD Tools supporting OpenTelemetry traces

List of other CI/CD tools that support OpenTelemetry traces and integrate with the Jenkins OpenTelemetryPlugin creating a distributed traces providing end to end visibility.

OpenTelemetry Maven Extension

The OpenTelemetry Maven Extension is a Maven extension to instrument with traces steps of Maven builds, including capturing details of the produced artifacts for traceability.

ℹ️ For seamless and turnkey integration of the trace of the Maven builds that use the OpenTelemetry Maven Extension with the Jenkins trace, consider in the Jenkins configuration to enable "Export OpenTelemetry configuration as environment variables".

OpenTelemetry Ansible Plugin

The OpenTelemetry Ansible Plugin is an Ansible callback to instrument with traces the tasks of Ansible playbooks.

ℹ️ For seamless and turnkey integration of the trace of the Ansible playbooks that use the OpenTelemetry plugin with the Jenkins trace, consider in the Jenkins configuration to enable "Export OpenTelemetry configuration as environment variables".

pytest-otel

The PyTest Otel Plugin is a PyTest plugin to report each PyTest test as a span of a trace.

ℹ️ For seamless and turnkey integration of the trace of the PyTest tests that use the OpenTelemetry plugin with the Jenkins trace, consider in the Jenkins configuration to enable "Export OpenTelemetry configuration as environment variables".

Otel CLI

The otel-cli is a command line wrapper to observe the execution of a shell command as an OpenTelemetry trace.

FAQ

Enrich your pipeline sh, bat, and powershell steps with meaningful explanation thanks to labels

If you use Jenkins pipelines in conjunction with the sh, bat, powershell steps, then it's highly recommended using the label argument to add a meaningful explanation thanks to step labels. Example:

node {
    sh(label: 'Maven verify', script: './mvnw deploy')
}

Using the OpenTelemetry OTLP/HTTP rather than OTLP/GRPC protocol

Navigate to the Jenkins OpenTelemetry Plugin configuration, in the "Advanced" section, add to the "Configuration Properties text area the following:

otel.exporter.otlp.protocol=http/protobuf

Support for disabling the Groovy Sandbox and accessing the Jenkins pipeline logs APIs while enabling the Jenkins OpenTelemetry Plugin

No test have been done on disabling the Groovy Sandbox and accessing the Jenkins pipeline logs APIs while enabling the Jenkins OpenTelemetry Plugin for the following reasons:

• Disabling the Groovy Sandbox is a very advanced use case due to the security implications of doing so
• The surface of Jenkins pipeline logs capabilities exposed by disabling the Groovy sandbox is very broad and goes way beyond the OpenTelemetyr plugin

If you are limited with the current capabilities of the Jenkins OpenTelemetry Plugin and consider opening up the Groovy sandbox to workaround these limitations, please prefer to reach out to us creating an enhancement request so we can work together at productizing the proper secured solution to your problem.

Learn More

• You can look at this video tutorial to get started:
• DevOpsWorld 2021 - Embracing Observability in Jenkins with OpenTelemetry

Demos

If you'd like to see this plugin in action with some other integrations then refer to the demos.

Contributing

Refer to our contribution guidelines

LICENSE

Licensed under Apache Software License 2, see LICENSE

opentelemetry-plugin's Issues

metrics for jenkins.agents.* are not generated

With the latest changes, the metrics for the number of agents (online and offline) are not populated.

Reliability Tests

Verify the reliability of the OpenTelemetry Plugin:

• Don't break Jenkins
• Don't pollute the logs

Links

• is blocked by #9
• is blocked by #16

Span name "Node Allocation" is not intuitive

Span name "Node Allocation" is not intuitive.

See Continuous Delivery Foundation vocabulary https://github.com/cdfoundation/sig-interoperability/blob/master/docs/vocabulary.md

Project
ArgoCD	Sync Wave	Sync Phase	Sync	N/A	Event	Application Controller
CircleCI	N/A	Step	Job	Workflow	Trigger	Executor
Codefresh	N/A	Step	Stage	Pipeline	Trigger	Runtime environment
Eiffel	Activity	Activity	Activity	Activity	Event	Environment
GitHub Actions	Action	Step	Job	Workflow	Event	Runner
GitLab CI/CD	N/A	Job	Stage	Pipeline	Trigger	Runner
Harness	N/A	Step	Pipeline	Workflow	Trigger	Delegate
Jenkins	N/A	Job	Stage	Pipeline	Trigger	Agent/Node
Jenkins X	N/A	Step	Stage	Pipeline	Trigger	Agent
Keptn	N/A	N/A	Task	Workflow	Event	Keptn-service
Screwdriver	N/A	Step	Job	Pipeline	Trigger	N/A
Spinnaker	N/A	Task	Stage	Pipeline	Trigger	Cluster
Tekton	N/A	Step	Task	Pipeline	Trigger	Resource (?)
Zuul	N/A	N/A	Job	Pipeline	Trigger	Node (?)

`git.username` reports the jenkins credentialsId rather than the git username

git.username reports the jenkins credentialsId rather than the git username

https://github.com/jenkinsci/opentelemetry-plugin/blob/master/src/main/java/io/jenkins/plugins/opentelemetry/job/AbstractGitStepHandler.java#L36

Identify the node provider

Feature Request

In a CI environment with different type of agents, such as:

• static workers
• ephemeral workers.

And also different cloud providers, such as:

• GCE
• EC2
• K8s

I'd like to identify the node spans with their provider therefore I can easily monitor the providers easily.

I don't know if the FlowNode can provide those details, but if it's possible, should the span have a new label like the ones specified in:

• https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/resource/semantic_conventions/cloud.md
• https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/resource/semantic_conventions/k8s.md

Create JobAction to visualise easily the link to the APM service

The current approach works for the build actions, but can we also enable the same approach for the job itself?

Therefore in the Job view we can see a link to the APM service?

Need to monitor and troubleshoot problems to launch build agents

Problem Description

The launch of build agents can have problems, administrators need to be alerted on, to monitor, and to troubleshoot these problems.

Example of problems

• Problem with the configuration of dynamic agents (invalid credentials...)
• Outage of underlying cloud provisioning infrastructure (no more VMs available, quota reached...)

Solutions

Probably related to hudson.slaves.ComputerListener#onLaunchFailure

but probably not enough.

hudson.slaves.CloudProvisioningListener#onFailure

Issues

Cloud Providers might not report errors with the provisioner:

• jenkinsci/google-compute-engine-plugin#261

Parallel steps are not properly chained to parent & children steps

Parallel steps are not properly chained to parent & children steps

labels.jenkins_pipeline_step_type for stages contain the name with special chars

The spans for the stage contains the value instead the stage type

See the span in json format

{
  "_index": "apm-7.12.0-span-000001",
  "_type": "_doc",
  "_id": "APU8CXgBEDp4T8ZhjhaJ",
  "_version": 1,
  "_score": null,
  "fields": {
    "span.name": [
      "Stage: Environment"
    ],
    "observer.name": [
      "instance-0000000004"
    ],
    "labels.jenkins_url": [
      "https://*********/"
    ],
    "trace.id": [
      "2643f01223dea633e394b5cd6b348816"
    ],
    "span.duration.us": [
      54770
    ],
    "processor.event": [
      "span"
    ],
    "agent.name": [
      "opentelemetry/java"
    ],
    "event.outcome": [
      "failure"
    ],
    "service.name": [
      "jenkins"
    ],
    "processor.name": [
      "transaction"
    ],
    "span.id": [
      "38cb0bec8f0feca4"
    ],
    "observer.version_major": [
      7
    ],
    "observer.hostname": [
      "8e14ded1d164"
    ],
    "span.type": [
      "app"
    ],
    "timestamp.us": [
      1615062665821357
    ],
    "observer.id": [
      "66b60432-bf3e-45ab-8000-409a38b7c546"
    ],
    "@timestamp": [
      "2021-03-06T20:31:05.821Z"
    ],
    "observer.ephemeral_id": [
      "2075549c-81ba-46e6-8e59-4ab02e1d2548"
    ],
    "observer.version": [
      "7.12.0"
    ],
    "ecs.version": [
      "1.8.0"
    ],
    "observer.type": [
      "apm-server"
    ],
    "labels.jenkins_pipeline_step_type": [
      "{ (Environment)"
    ],
    "parent.id": [
      "43d7474fa0a8cfea"
    ],
    "agent.version": [
      "0.16.0"
    ],
    "labels.service_namespace": [
      "jenkins"
    ]
  },
  "highlight": {
    "span.id": [
      "@kibana-highlighted-field@38cb0bec8f0feca4@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1615062665821
  ]
}

What do you think if we create a new label for the name of the stage and set the type as stage instead?

    "labels.jenkins_pipeline_step_type": [
      "stage"
    ],
    "labels.jenkins_pipeline_stage_name": [
      "Environment"
    ],

Traces for HTTP requests

Feature Request

Add tracing to Jenkins controller HTTP requests (instrument the Stapler web framework).

Add more features to the node/agent steps

Follow up #21

To answer some questions such as:

• Should the allocate and ready spans use the same `labels?
  • the allocate one is the only with the new attribute, but does it make sense to populate that attribute to the child span?
• Should we calculate the provisioning time between the allocate and the ready spans and provide a new attribute?

StackOverflowError

Version report

Jenkins versions report:

Jenkins: 2.263.4
OS: Linux - 2.6.32-754.36.1.el6.x86_64
---
opentelemetry:0.9

Reproduction steps

• Installed opentelemetry plugin 0.9
• Deleted a job build
• Observed the following in the logs

2021-05-03 16:27:52.627+0000 [id=128871]        WARNING h.model.listeners.RunListener#report: RunListener failed
java.lang.StackOverflowError
        at hudson.model.AbstractItem.getFullName(AbstractItem.java:477)
        at hudson.model.AbstractItem.getFullName(AbstractItem.java:477)
        at hudson.model.AbstractItem.getFullName(AbstractItem.java:477)
        at io.jenkins.plugins.opentelemetry.job.OtelTraceService$RunIdentifier.fromRun(OtelTraceService.java:306)
        at io.jenkins.plugins.opentelemetry.job.OtelTraceService.getSpan(OtelTraceService.java:70)
        at io.jenkins.plugins.opentelemetry.job.OtelTraceService.setupContext(OtelTraceService.java:219)
        at io.jenkins.plugins.opentelemetry.job.opentelemetry.OtelContextAwareAbstractRunListener.onDeleted(OtelContextAwareAbstract
RunListener.java:99)
        at io.jenkins.plugins.opentelemetry.job.MonitoringRunListener._onDeleted(MonitoringRunListener.java:243)
        at io.jenkins.plugins.opentelemetry.job.opentelemetry.OtelContextAwareAbstractRunListener.onDeleted(OtelContextAwareAbstract
RunListener.java:100)
        at io.jenkins.plugins.opentelemetry.job.MonitoringRunListener._onDeleted(MonitoringRunListener.java:243)
        at io.jenkins.plugins.opentelemetry.job.opentelemetry.OtelContextAwareAbstractRunListener.onDeleted(OtelContextAwareAbstract
RunListener.java:100)
        at io.jenkins.plugins.opentelemetry.job.MonitoringRunListener._onDeleted(MonitoringRunListener.java:243)
        at io.jenkins.plugins.opentelemetry.job.opentelemetry.OtelContextAwareAbstractRunListener.onDeleted(OtelContextAwareAbstract
RunListener.java:100)
...

Add more attributes to the runs

Dependencies

Feature Request

Add some attributes regarding the run in order to help with the filtering:

Multibranch pipeline

• detect if the transaction is related to:
  • a branch
  • pul request
  • tag

All

• Build description, people might use it to add more context to the build.
• build result (unstable, failure, aborted, success)
• build cause

For instance:

Update README.md documentation

README.md documentation is outdated.

root span name using the job name should exclude the PR number to remain a low cardinality value

The root span name in Multi Branch Pipelines is a high cardinality value due to the name of the Pull requests.
we should exclude the PR number from the span name.

See also:

• https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#span
• Span name: Both low-cardinality (grouping key) and human-readable (display name) #557
• Use low cardinality names for database spans #341

Capture declarative condition post conditions as spans (`always`, `cleanup`, `success`...)

Feature Request

Introduce wrapping spans for families of post conditions in declarative pipelines: always, changed, fixed, regression, aborted, success, unstable, unsuccessful, failure, cleanup.

• https://www.jenkins.io/doc/book/pipeline/syntax/#post-conditions
• https://www.jenkins.io/doc/pipeline/tour/post/

For instance:

pipeline {
  agent any
  stages {
    stage ('Hello') {
      steps {
        echo "hello World"
      }
    }
  }
  post { 
        always { 
            sh 'echo "always"'
        }
        cleanup {
            sh 'echo "cleanup"'
        }
    }
}

Produces:

Monitor Jenkins controller JVM metrics

Feature Request

Capture JVM monitoring metrics: Garbage Collector activity, Heap and Non Heap size, CPU, threads

Plugin label metadata in the attributes for some spans is not correct

Version report

LTS 2.289.1

Latest plugin version

Reproduction steps

Metadata for the plugin step is not correct for the googleStorageUpload step

While debugging the descriptor, it seems the descriptorId is coming from CoreSteps

While for some other plugin steps the information is populated correctly, see

Maybe the implementation in the google storage plugin is the reason and this is just a corner case

🚧 Parallel step does not work

⚠️

What

When using the parallel step to generate dynamically a bunch of parallel stages, the spans are not created

Screenshots

Configuration

• Jenkins LTs
• Latest version ->h ttps://github.com/jenkinsci/opentelemetry-plugin/commit/bc0eed04b7d257e9dceac017a8029082a150d573

Actions

• Create the minimal example
• Provide debug traces

Capability to add "logs" to spans

Problem Description

I want to manually add "events" / "logs" to my spans in my pipeline, similar to manually creating span events or log messages when instrumenting applications with distributed tracing.

Desire to enable more logs without modifying the pipeline.

Example use cases

• ...

Example with the OpenTelemetry APIs for Java

Span currentSpan = Span.current();
currentSpan.addEvent("my event", ...); // TODO realistic example

See Other Problem Statements

• Capability to manually create spans #90
• Capability to manually add attributes to spans #91

See also

• quarkusio/quarkus#9916 Allow logging via OpenTelemetry #9916

Proposed Solutions

Correlate pipeline logs with traces

• Enrich Jenkins pipeline logs with the trace_id and span_id + let the observability backend correlated logs and traces

addSpanEvent pipeline step

Similar to the OpenTelemetry API io.opentelemetry.api.trace.Span#addEvent(java.lang.String, io.opentelemetry.api.common.Attributes)

Some Steps are treated as transactions upon controller restart

With the latest changes some steps are processed as transactions:

{
  "_index": "apm-7.12.0-transaction-000001",
  "_type": "_doc",
  "_id": "VugXGHgBW07LL3qwDDIu",
  "_version": 1,
  "_score": null,
  "fields": {
    "transaction.name.text": [
      "Install super-linter"
    ],
    "observer.name": [
      "instance-0000000004"
    ],
    "labels.jenkins_url": [
      "https://*******/"
    ],
    "service.language.name": [
      "java"
    ],
    "transaction.result": [
      "OK"
    ],
    "transaction.sampled": [
      true
    ],
    "transaction.id": [
      "33855c83e8744d5d"
    ],
    "host.ip": [
      "35.232.43.180"
    ],
    "trace.id": [
      "79455033f8d2812bb5ffe84c614136f1"
    ],
    "transaction.span_count.dropped": [
      0
    ],
    "processor.event": [
      "transaction"
    ],
    "agent.name": [
      "opentelemetry/java"
    ],
    "event.outcome": [
      "success"
    ],
    "service.name": [
      "jenkins"
    ],
    "service.framework.name": [
      "jenkins"
    ],
    "processor.name": [
      "transaction"
    ],
...

NPE in SimpleSpanProcessor

Version report

Jenkins versions report:

Jenkins: 2.263.4
OS: Linux - 2.6.32-754.36.1.el6.x86_64
---
opentelemetry:0.10-beta

Reproduction steps

• Installed opentelemetry plugin 0.9
• Observed the following in the logs

2021-05-10 01:48:42.087+0000 [id=1367]  WARNING i.o.s.t.e.SimpleSpanProcessor#onEnd: Exporter threw an Exception
java.lang.NullPointerException
        at io.opentelemetry.proto.common.v1.AnyValue$Builder.setStringValue(AnyValue.java:899)
        at io.opentelemetry.exporter.otlp.internal.CommonAdapter.makeStringArrayAnyValue(CommonAdapter.java:132)
        at io.opentelemetry.exporter.otlp.internal.CommonAdapter.makeStringArrayKeyValue(CommonAdapter.java:72)
        at io.opentelemetry.exporter.otlp.internal.CommonAdapter.toProtoAttribute(CommonAdapter.java:36)
        at io.opentelemetry.exporter.otlp.internal.SpanAdapter.lambda$toProtoSpan$0(SpanAdapter.java:102)
        at io.opentelemetry.sdk.trace.AttributesMap.forEach(AttributesMap.java:56)
        at io.opentelemetry.exporter.otlp.internal.SpanAdapter.toProtoSpan(SpanAdapter.java:102)
        at io.opentelemetry.exporter.otlp.internal.SpanAdapter.groupByResourceAndLibrary(SpanAdapter.java:82)
        at io.opentelemetry.exporter.otlp.internal.SpanAdapter.toProtoResourceSpans(SpanAdapter.java:42)
        at io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter.export(OtlpGrpcSpanExporter.java:88)
        at io.opentelemetry.sdk.trace.export.SimpleSpanProcessor.onEnd(SimpleSpanProcessor.java:75)
        at io.opentelemetry.sdk.trace.RecordEventsReadableSpan.endInternal(RecordEventsReadableSpan.java:442)
        at io.opentelemetry.sdk.trace.RecordEventsReadableSpan.end(RecordEventsReadableSpan.java:422)
        at io.jenkins.plugins.opentelemetry.job.MonitoringRunListener._onFinalized(MonitoringRunListener.java:222)
        at io.jenkins.plugins.opentelemetry.job.opentelemetry.OtelContextAwareAbstractRunListener.onFinalized(OtelContextAwareAbstra
ctRunListener.java:59)
        at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:255)
        at hudson.model.Run.onEndBuilding(Run.java:2069)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:636)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:353)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)

Comments

Seems to me that the problem is caused by adding null value for sensitive parameters in here

opentelemetry-plugin/src/main/java/io/jenkins/plugins/opentelemetry/job/MonitoringRunListener.java

Lines 113 to 121 in 9d1e176

	for (ParameterValue parameter : parameters.getParameters()) {
	parameterNames.add(parameter.getName());
	parameterIsSensitive.add(parameter.isSensitive());
	if (parameter.isSensitive()) {
	parameterValues.add(null);
	} else {
	parameterValues.add(Objects.toString(parameter.getValue(), null));
	}
	}

which then leads to https://github.com/open-telemetry/opentelemetry-java/blob/e60857d369a1f10664833483cbb841abd5ee53c3/exporters/otlp/common/src/main/java/io/opentelemetry/exporter/otlp/internal/CommonAdapter.java#L129-L135
trying to add null value to the list of string parameters that is not supported by the protocol
https://github.com/open-telemetry/opentelemetry-proto/blob/de4fc37940d39370194fb774e634ca408dacd865/opentelemetry/proto/common/v1/common.proto#L24-L39

Capability to manually create spans

Problem description

I want to manually create spans in my pipeline, similar to manually creating spans when instrumenting applications with distributed tracing.

Example use cases

• Create a "grouping span" to group multiple atomic actions together.
  • Example: Get code from a complex Git repository incl fetching submodules
• Add events to the trace typically to add "logs"
  • Probably better solved using the concept of "span events" in OpenTelemetry API
• Add informations on shell steps to indicate the purpose of the shell step (e.g. an sh 'docker login -u $user -p $pass my.docker.registry')
  • May be better solved solving Capability to manually add attributes to spans #91

Example with the OpenTelemetry APIs for Java

Tracer tracer = null;
tracer.spanBuilder("Get code from Git (incl fetching sub modules)")
    .setAttribute("my.attribute", "my value")
    .startSpan();

See other problem statements

• Capability to manually add attributes to spans #91
• Capability to add "logs" to spans #92

Proposed solutions

createSpan pipeline step

pipeline {
    agent any
    stages {
        stage('Checkout') {
            steps {
                createSpan(name:'my-span', attributes:[TODO 'command':'git']) {
                    // TODO sequence of shell steps to `git`+ fetch git submodules
                    // Get some code from a GitHub repository
                    // git credentialsId: 'my-user-git', url: 'https://github.com/jglick/simple-maven-project-with-tests.git'
                }
            }
        }
        stage('Build') {
            steps {
                ...
            }
        }
    }
}

Feat: Allow for specifying multiple OTLP Headers (see `OTEL_EXPORTER_OTLP_HEADERS`)

Feature Request

Hey, thought I'd take this new plugin for a spin but ran into a limitation not being able to specify multiple headers.

We use Honeycomb.io, who support OTLP/GRPC natively. However you need to specify the following headers:

x-honeycomb-team: YOUR_API_KEY
x-honeycomb-dataset: YOUR_DATASET

Currently it looks like only a single header can be specified. Using the otel-collector will be a reasonable workaround for us for now.

Pipeline parameters are not reported as span attributes by Elastic APM

Version report

Jenkins and plugins versions report:

• Jenkins opentelemetry-plugin 0.10-beta
• Elastic Observability 7.12.1

Reproduction steps

• Create a parameterized pipeline and execute it
• Visualise the trace in Elastic APM, the fields ci.pipeline.parameter.name, ci.pipeline.parameter.sensitive, and ci.pipeline.parameter.value are not reported by Elastic APM as span attributes when Jaeger reports them.

pipeline {
    agent any
    
    parameters {
        string(name: 'PERSON', defaultValue: 'Mr Jenkins', description: 'Who should I say hello to?')
        text(name: 'BIOGRAPHY', defaultValue: '', description: 'Enter some information about the person')
        booleanParam(name: 'TOGGLE', defaultValue: true, description: 'Toggle this value')
        choice(name: 'CHOICE', choices: ['One', 'Two', 'Three'], description: 'Pick something')
        password(name: 'PASSWORD', defaultValue: 'SECRET', description: 'Enter a password')
    }
    stages {
        stage('Example') {
            steps {
                echo "Hello ${params.PERSON}"
                echo "Biography: ${params.BIOGRAPHY}"
                echo "Toggle: ${params.TOGGLE}"
                echo "Choice: ${params.CHOICE}"
                echo "Password: ${params.PASSWORD}"
            }
        }
    }
}

Results

Expected result as visible with Jaeger 1.22:

Actual result with Elastic APM 7.12.1:

Make Otel Metrics Exporter interval configurable

Feature Request

Make Otel Metrics Exporter interval configurable. Today it's hard coded at 60s

See

opentelemetry-plugin/src/main/java/io/jenkins/plugins/opentelemetry/OpenTelemetrySdkProvider.java

Line 124 in 0742019

int exportIntervalMillis = 60_000;

Collect cloud correlation attributes for the Jenkins build agents and controller

Collect cloud correlation attributes for the Jenkins build agents and controller

See https://github.com/open-telemetry/opentelemetry-java/tree/main/extensions

See io.jenkins.plugins.opentelemetry.computer.MonitoringComputerListener.GetComputerAttributes

NullPointerException in MonitoringComputerListener.postConstruct

2021-03-09 16:04:13.475+0000 [id=42]	WARNING	h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate Key[type=io.jenkins.plugins.opentelemetry.computer.MonitoringComputerListener, annotation=[none]]; skipping this component
java.lang.NullPointerException
	at io.jenkins.plugins.opentelemetry.computer.MonitoringComputerListener.postConstruct(MonitoringComputerListener.java:67)
Caused: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.ExtensionFinder$GuiceFinder$SezpozModule.onProvision(ExtensionFinder.java:596)
Caused: java.lang.RuntimeException: @PostConstruct public void io.jenkins.plugins.opentelemetry.computer.MonitoringComputerListener.postConstruct()
	at hudson.ExtensionFinder$GuiceFinder$SezpozModule.onProvision(ExtensionFinder.java:598)
	at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126)
	at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:87)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
Caused: com.google.inject.ProvisionException: Unable to provision, see the following errors:
1) Error notifying ProvisionListener hudson.ExtensionFinder$GuiceFinder$SezpozModule of io.jenkins.plugins.opentelemetry.computer.MonitoringComputerListener.
 Reason: java.lang.RuntimeException: @PostConstruct public void io.jenkins.plugins.opentelemetry.computer.MonitoringComputerListener.postConstruct()
1 error
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:52)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
	at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:440)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
	at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
	at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
	at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:402)
	at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:393)
	at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:346)
	at hudson.ExtensionList.load(ExtensionList.java:381)
	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:317)
	at hudson.ExtensionList.iterator(ExtensionList.java:172)
	at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:247)
	at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1635)
	at jenkins.model.Nodes$9.run(Nodes.java:351)
	at hudson.model.Queue._withLock(Queue.java:1398)
	at hudson.model.Queue.withLock(Queue.java:1275)
	at jenkins.model.Nodes.load(Nodes.java:346)
	at jenkins.model.Jenkins$13.run(Jenkins.java:3254)
	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1131)
	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Add plugin name and version for the span caused by a plugin pipeline step

Dependencies

Feature Request

I'm not sure whether this is something feasible!

Add the plugin name and version attributes for every span caused by a plugin pipeline step.

Why

This could help us to correlate misbehaviours when a plugin gets updated.

Stable CI & Jenkins Semantic Conventions

Review our semantic conventions to head toward interoperability with other CI/CD platforms.

Current source code:

• https://github.com/jenkinsci/opentelemetry-plugin/blob/opentelemetry-0.9/src/main/java/io/jenkins/plugins/opentelemetry/semconv/JenkinsOtelSemanticAttributes.java
• https://github.com/jenkinsci/opentelemetry-plugin/blob/opentelemetry-0.9/src/main/java/io/jenkins/plugins/opentelemetry/semconv/JenkinsSemanticMetrics.java

More on semantic conventions for CI/CD

• CD Foundation / SIG Interoperability / Vocabulary

Monitor Jenkins controller disk usage metrics

Feature Request

Capture Jenkins Disk Usage metrics.
The [Jenkins Simple Disk Usage Plugin] allows to monitor disk usage in the Jenkins Home at the at the job level.

See https://plugins.jenkins.io/cloudbees-disk-usage-simple/ and https://github.com/jenkinsci/prometheus-plugin/blob/prometheus-2.0.9/src/main/java/org/jenkinsci/plugins/prometheus/DiskUsageCollector.java

Leverage the sh/bat steps with the system that interact with

What

Allow to create span labels dynamically for the steps if they have an environment variable set.

Why

Some pipeline steps interact with third party services. This could help to enhance the transactions/spans with more information about what are the services they interact with.

Design

Env variables could be set dynamically based on the below nomenclature:

• OTEL_STEP_<ATTRIBUTE_NAME>

Then <ATTRIBUTE_NAME> will be replaced accordingly,

spanBuilder.setAttribute("OTEL_STEP_FOO", "foo")

Some ideas

Given the step:

pipeline {
  ...
  stages {
    stage('login') {
      steps {
        withCredentials([usernamePassword(credentialsId: 'docker-login',
                                          passwordVariable: 'pass',
                                          usernameVariable: 'user')]) {
          sh 'docker login -u $user -p $pass my.docker.registry'
        }
      }
   }
}

I'd like to create a span with the sh step, but also saying that the thirdparty system that interacts with is my.docker.registry.

Therefore, a potential implementation (if possible) could be if the env variable OTEL_STEP_SERVICE is set, see below:

pipeline {
  ...
  stages {
    stage('login') {
      steps {
        withCredentials([usernamePassword(credentialsId: 'docker-login',
                                          passwordVariable: 'pass',
                                          usernameVariable: 'user')]) {
          withEnv(['OTEL_STEP_SERVICE=my.docker.registry']) {
            sh 'docker login -u $user -p $pass $OTEL_STEP_SERVICE'
          }
        }
      }
   }
}

Then OTEL_STEP_SERVICE will help to identify the system that the step interacts with.

Division by zero exception when no swap memory allocated

Version report

Jenkins and plugins versions report: jenkins-opentelemetry-plugin 0.13

Division by zero when zero swap memory is allocated to the JVM

23.117 [id=121]	WARNING	i.o.s.m.e.IntervalMetricReader$Exporter#doRun: Exporter threw an Exception
java.lang.ArithmeticException: Division undefined
	at java.math.BigDecimal.divide(BigDecimal.java:1744)
	at io.jenkins.plugins.opentelemetry.init.JvmMonitoringInitializer.lambda$initialize$9(JvmMonitoringInitializer.java:148)
	at io.opentelemetry.sdk.metrics.AsynchronousInstrumentAccumulator.lambda$doubleAsynchronousAccumulator$2(AsynchronousInstrumentAccumulator.java:47)
	at io.opentelemetry.sdk.metrics.AsynchronousInstrumentAccumulator.collectAll(AsynchronousInstrumentAccumulator.java:86)
	at io.opentelemetry.sdk.metrics.AbstractAsynchronousInstrument.collectAll(AbstractAsynchronousInstrument.java:23)
	at io.opentelemetry.sdk.metrics.SdkMeter.collectAll(SdkMeter.java:102)
	at io.opentelemetry.sdk.metrics.SdkMeterProvider.collectAllMetrics(SdkMeterProvider.java:68)
	at io.opentelemetry.sdk.metrics.export.IntervalMetricReader$Exporter.doRun(IntervalMetricReader.java:131)
	at io.opentelemetry.sdk.metrics.export.IntervalMetricReader$Exporter.run(IntervalMetricReader.java:122)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)```


### Reproduction steps

Run the Jenkins Controller in an environment where no virtual memory is allocated to the Jenkins Controller JVM process.


### Results

Expected result:

Swap Memory % utilization don't fail (either not reporting anything or reporting 0% or reporting 100%)

Actual result:

Exception

23.117 [id=121] WARNING i.o.s.m.e.IntervalMetricReader$Exporter#doRun: Exporter threw an Exception
java.lang.ArithmeticException: Division undefined
at java.math.BigDecimal.divide(BigDecimal.java:1744)
at io.jenkins.plugins.opentelemetry.init.JvmMonitoringInitializer.lambda$initialize$9(JvmMonitoringInitializer.java:148)

Support configuring additional attributes for traces and metrics (see `OTEL_RESOURCE_ATTRIBUTES`)

Feature Request

Support configuration to add attributes to traces, similar to OTEL_RESOURCE_ATTRIBUTES (here).

Example use cases

• Qualify better the Jenkins instance
• Elastic needs a "transaction.type" (here)

Jenkins admins need dashboards and alerts on the health of the Jenkins platform

Problem Description

Jenkins admins need dashboards and alerts to solve the following problems:

• Overall health indicators of the health of the Jenkins platform
• Detect build agent allocation problems
• Detect Jenkins controller responsiveness problems
  • Web UI responsiveness
• Detect File System usage problems, GC problems, CPU problems on Jenkins controller...
  • It's not exactly user facing problems but we know these are critical indicators to look at.
  • Ideally we would identify symptoms of these root causes and monitor / alert on the symptoms

Capability to manually add attributes to spans

Problem description

I want to manually add attributes to spans in my pipeline, similar to manually adding attributes to spans when instrumenting applications with distributed tracing.

Example use cases

• Add informations on shell steps to indicate the purpose of the shell step (e.g. an sh 'docker login -u $user -p $pass my.docker.registry')
  • ❓ How to add attributes to atomic built-in steps ("non wrapping steps") like the sh step?

Exemple with the OpenTelemetry APIs for Java

Span currentSpan = Span.current();
currentSpan.setAttribute(""net.peer.name", "my.docker.registry");
currentSpan.setAttribute(""rpc.service", "docker");
currentSpan.setAttribute(""rpc.method", "login");

See other problem statements

• Capability to manually create spans #90
• Capability to add "logs" to spans #92

Proposed solutions

addAttributes() pipeline step

TODO

adding an attributes attribute to the built-in steps

TODO

Wrong root span name and value of the `ci.pipeline.id` attribute for traces of PR Builds on Multibranch Pipelines

Version report

Jenkins: 2.299
OS: Linux - 5.4.0-62-generic

Jenkins plugins versions

---

pmd:4.0.1-SNAPSHOT (private-8ae81512-jenkins)
analysis-core:1.97-SNAPSHOT (private-07ea7c8e-jenkins)
pipeline-github-lib:1.0
checkstyle:4.0.1-SNAPSHOT (private-46d69902-jenkins)
xunit:3.0.2
ssh-slaves:1.32.0
bitbucket:1.1.29
basic-branch-build-strategies:1.3.2
groovy:2.4
matrix-auth:2.6.7
ansicolor:1.0.0
kubernetes:1.30.0
configuration-as-code:1.51
metrics:4.0.2.8
email-ext:2.83
skip-notifications-trait:1.0.5
cobertura:1.16
jaxb:2.3.0.1
active-directory:2.24
cloudbees-credentials:3.3
authorize-project:1.4.0
build-timeout:1.20
timestamper:1.13
gradle:1.36
ant:1.11
hashicorp-vault-plugin:3.8.0
config-file-provider:3.8.0
job-dsl:1.77
warnings-ng:9.2.1
ldap:2.7
branch-api:2.6.4-SNAPSHOT (private-1b883b9a-gleb.samsonov)
ws-cleanup:0.39
jacoco:3.3.0
ec2:1.62
workflow-aggregator:2.6
windows-slaves:1.8
external-monitor-job:1.7
build-timestamp:1.0.3
ssh-credentials:1.19
pipeline-utility-steps:2.8.0
workflow-api:2.46
pam-auth:1.6
startup-trigger-plugin:2.9.3
git:4.7.2
workflow-cps-global-lib:2.21
scm-api:2.6.4
credentials:2.5
structs:1.23
junit:1.50
script-security:1.77
dtkit-api:3.0.0
mercurial:2.15
trilead-api:1.0.13
apache-httpcomponents-client-4-api:4.5.13-1.0
display-url-api:2.3.5
variant:1.4
handy-uri-templates-2-api:2.1.8-1.0
cloudbees-folder:6.15
snakeyaml-api:1.29.1
kubernetes-client-api:5.4.1
blueocean-config:1.24.7
blueocean-events:1.24.7
lockable-resources:2.11
matrix-project:1.19
workflow-job:2.41
workflow-basic-steps:2.23
blueocean-core-js:1.24.7
workflow-cps:2.92
token-macro:2.15
maven-plugin:3.12
workflow-durable-task-step:2.39
code-coverage-api:1.4.0
plain-credentials:1.7
blueocean-bitbucket-pipeline:1.24.7
blueocean-dashboard:1.24.7
authentication-tokens:1.4
workflow-step-api:2.23
github-branch-source:2.11.1
jackson2-api:2.12.3
blueocean-git-pipeline:1.24.7
credentials-binding:1.26
blueocean-github-pipeline:1.24.7
command-launcher:1.6
pipeline-model-definition:1.8.5
antisamy-markup-formatter:2.1
durable-task:1.37
workflow-multibranch:2.26
workflow-support:3.8
pipeline-build-step:2.13
blueocean-pipeline-editor:1.24.7
workflow-scm-step:2.13
jquery3-api:3.6.0-1
mailer:1.34
bouncycastle-api:2.20
aws-java-sdk:1.11.995
blueocean-display-url:2.4.1
font-awesome-api:5.15.3-3
node-iterator-api:1.5.0
pipeline-stage-view:2.19
pipeline-milestone-step:1.3.2
checks-api:1.7.0
kubernetes-credentials:0.9.0
resource-disposer:0.16
git-client:3.7.2
blueocean-rest-impl:1.24.7
blueocean-web:1.24.7
aws-credentials:1.29
forensics-api:1.1.0
blueocean-autofavorite:1.2.4
pipeline-input-step:2.12
blueocean-i18n:1.24.7
blueocean-pipeline-api-impl:1.24.7
blueocean-personalization:1.24.7
data-tables-api:1.10.25-1
echarts-api:5.1.2-2
pipeline-stage-step:2.5
bootstrap4-api:4.6.0-3
blueocean-jwt:1.24.7
jenkins-design-language:1.24.7
jdk-tool:1.5
plugin-util-api:2.3.0
jsch:0.1.55.2
analysis-model-api:10.2.5
blueocean-rest:1.24.7
blueocean-commons:1.24.7
git-server:1.9
pubsub-light:1.16
sse-gateway:1.24
ace-editor:1.1
blueocean-pipeline-scm-api:1.24.7
github:1.33.1
pipeline-rest-api:2.19
javadoc:1.6
momentjs:1.1.1
handlebars:3.0.8
github-api:1.123
docker-workflow:1.25
pipeline-model-extensions:1.8.5
pipeline-stage-tags-metadata:1.8.5
pipeline-model-api:1.8.5
favorite:2.3.3
popper-api:1.16.1-2
htmlpublisher:1.25
jira:3.1.3
pipeline-graph-analysis:1.11
okhttp-api:3.14.9
docker-commons:1.17
jjwt-api:0.11.2-9.c8b45b8bb173
opentelemetry:0.14
deployit-plugin:10.0.3
sshd:3.0.3
parameterized-trigger:2.41
run-condition:1.5
jquery:1.12.4-1
conditional-buildstep:1.4.1
delivery-pipeline-plugin:1.4.2
popper2-api:2.5.4-2
blueocean-jira:1.24.5
bootstrap5-api:5.0.1-2
caffeine-api:2.9.1-23.v51c4e2c879c8
statistics-gatherer:2.0.3
cloudbees-bitbucket-branch-source:2.9.9

Reproduction steps

Root Span Name and attribute ci.pipeline.id are reported as path/to/Build/pr-{number} when path/to/Build/job/PR-123/1 was expected.

while the attribute ci.pipeline.run.url reports the expected url with path/to/Build/job/PR-123/1

Intermittent java.util.ConcurrentModificationException

Version report

Jenkins and plugins versions report:

Jenkins: 2.263.1
OS: Windows Server 2016 - 10.0
---
opentelemetry:0.13

• What Operating System are you using (both controller, and any agents involved in the problem)?

- Windows Server 2016 - 10.0
- Linux - 4.14.200-155.322.amzn2.x86_64

Results

Actual result:

java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextNode(Unknown Source)
	at java.util.HashMap$EntryIterator.next(Unknown Source)
	at java.util.HashMap$EntryIterator.next(Unknown Source)
	at java.util.AbstractMap.toString(Unknown Source)
	at com.google.common.collect.AbstractMapBasedMultimap$AsMap.toString(AbstractMapBasedMultimap.java:1323)
	at com.google.common.collect.AbstractMultimap.toString(AbstractMultimap.java:253)
	at com.google.common.collect.ArrayListMultimap.toString(ArrayListMultimap.java:65)
	at java.lang.String.valueOf(Unknown Source)
	at java.lang.StringBuilder.append(Unknown Source)
	at io.jenkins.plugins.opentelemetry.job.OtelTraceService$RunSpans.toString(OtelTraceService.java:244)
	at java.lang.String.valueOf(Unknown Source)
	at java.lang.StringBuilder.append(Unknown Source)
	at io.jenkins.plugins.opentelemetry.job.OtelTraceService.lambda$getSpan$1(OtelTraceService.java:81)
	at java.util.logging.Logger.log(Unknown Source)
	at io.jenkins.plugins.opentelemetry.job.OtelTraceService.getSpan(OtelTraceService.java:81)
	at io.jenkins.plugins.opentelemetry.job.OtelEnvironmentContributor.buildEnvironmentFor(OtelEnvironmentContributor.java:26)
	at hudson.model.Run.getEnvironment(Run.java:2442)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getEnvironment(WorkflowRun.java:492)
	at hudson.plugins.git.GitSCM.getParamExpandedRepos(GitSCM.java:504)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1205)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1301)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

Add span traces steps to help with distributes traces in any Jenkins shared library

Feature Request

A Jenkins shared library step does not match any span since it is not a FlowNode, but a wrapper that called some other built-in steps (either the ones provided by the Jenkins Core/Workflow or plugins).

Let's support a couple of steps to enable the distributed traces in any shared library explicitly:

openSpan (with some parameters)
closeSpan (with some parameters)
createSpan (with some parameters and a closure)

To illustrate the above I'll use some a couple of examples (using the https://www.jenkins.io/doc/book/pipeline/shared-libraries/#defining-a-more-structured-dsl)

// vars/buildPlugin.groovy
def call(Map config) {
    node {
        git url: "https://github.com/jenkinsci/${config.name}-plugin.git"
        sh 'mvn install'
        mail to: '...', subject: "${config.name} plugin build", body: '...'
    }
}

The above-mentioned new steps should help us to create a span:

// vars/buildPlugin.groovy
def call(Map config) {
    String spanId = openSpan(...)
    node {
        git url: "https://github.com/jenkinsci/${config.name}-plugin.git"
        sh 'mvn install'
        mail to: '...', subject: "${config.name} plugin build", body: '...'
    }
    closeSpan(spanId: id)
}

Or using the closure:

// vars/buildPlugin.groovy
def call(Map config) {
    createSpan(...) {
        node {
            git url: "https://github.com/jenkinsci/${config.name}-plugin.git"
            sh 'mvn install'
            mail to: '...', subject: "${config.name} plugin build", body: '...'
        }
    }
}

Override global configuration remains previous broken configuration

Version report

Results

When updating the authorisation from authentication: "noAuthentication" to authentication: "otlpHeaderAuthentication" with a wrong configuration, the configuration cannot be updated afterwards and it remains the previous configuration.

java.lang.IllegalArgumentException: Invalid character '=' in key name 'authorization=bearer'
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:239)
	at io.grpc.Metadata$Key.validateName(Metadata.java:742)
	at io.grpc.Metadata$Key.<init>(Metadata.java:750)
	at io.grpc.Metadata$Key.<init>(Metadata.java:668)
	at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:959)
	at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:954)
	at io.grpc.Metadata$Key.of(Metadata.java:705)
	at io.grpc.Metadata$Key.of(Metadata.java:701)
	at io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporterBuilder.addHeader(OtlpGrpcSpanExporterBuilder.java:117)
	at io.jenkins.plugins.opentelemetry.authentication.HeaderAuthentication.configure(HeaderAuthentication.java:63)
	at io.jenkins.plugins.opentelemetry.OpenTelemetrySdkProvider.initializeForGrpc(OpenTelemetrySdkProvider.java:129)
	at io.jenkins.plugins.opentelemetry.OpenTelemetrySdkProvider.initialize(OpenTelemetrySdkProvider.java:216)
	at io.jenkins.plugins.opentelemetry.JenkinsOpenTelemetryPluginConfiguration.initializeOpenTelemetry(JenkinsOpenTelemetryPluginConfiguration.java:88)
	at io.jenkins.plugins.opentelemetry.JenkinsOpenTelemetryPluginConfiguration.setAuthentication(JenkinsOpenTelemetryPluginConfiguration.java:129)

Therefore there is no an easy way to update with a new configuration.

Actual result:

Add support for detecting Maven build type

Feature Request

Looks like the usual suspects like freestyle, pipeline and multi-branch are are correctly detected in the pipeline type but a still fairly common Maven one is missing.

Looks like all that's missing is a bit of code in the OtelUtils class to detect the run type as hudson.maven.AbstractMavenBuild similar to what is already done for freestyle build.

Elastic Observability Redirection URL is broken

Redirection URL no longer works

Example:
https://localhost:5601/app/apm/services/jenkins/transactions/view?rangeFrom=2021-03-24T11:25:53.019Z&rangeTo=2021-03-24T11:45:53.019Z&transactionName=scripted-pipeline-assigned-to-specific-node&transactionType=unknown&latencyAggregationType=avg

Scalability Tests

Scalability Tests

Dynamic service naming to support CI at Scale

Dependencies

NA

Feature Request

This is related to the discussion done in #88

@skundrik proposed another way to configure the service name based on the Jenkins URL. Something like grok to extract parts you want and combine them into service name.

Context

The current implementation is something that can now be configured with the Jenkins UI (Global settings) and JCasC.

The proposal could allow to provide an opinionated way to configure the service name instead allowing users to set the naming the would like.

Questions

I'd like to highlight some cases to consider about this approach and what the proposal should do in those cases:

• What if the URL is too long?
• What if the URL contains details that are specific for the deployment type (like the k8s FQDN)?
• What if the URL has not been configured?

Record agent/node steps

What

Trace the agent/node steps as spans

Why

Then we can start answering questions such as:

• How long does it take to provision the node?
• What's the agent with more calls?

Add ColumnView with the URLs

Dependencies

Feature Request

Use https://javadoc.jenkins.io/hudson/views/ListViewColumn.html to add a link in the Jobs View, something like:

Customise the service name to support CI at Scale

What

Currently the service name is hardcoded with the jenkins service name

See

opentelemetry-plugin/src/main/java/io/jenkins/plugins/opentelemetry/opentelemetry/resource/JenkinsResourceProvider.java

Lines 33 to 38 in 390ad7d

	Attributes attributes = Attributes.of(
	ResourceAttributes.SERVICE_NAMESPACE, JenkinsOtelSemanticAttributes.SERVICE_NAMESPACE_JENKINS,
	ResourceAttributes.SERVICE_NAME, JenkinsOtelSemanticAttributes.SERVICE_NAME_JENKINS,
	ResourceAttributes.SERVICE_VERSION, version,
	JenkinsOtelSemanticAttributes.JENKINS_URL, rootUrl
	);

and

opentelemetry-plugin/src/main/java/io/jenkins/plugins/opentelemetry/semconv/JenkinsOtelSemanticAttributes.java

Lines 79 to 87 in b355859

	/**
	* @see io.opentelemetry.semconv.resource.attributes.ResourceAttributes#SERVICE_NAME
	*/
	public static final String SERVICE_NAME_JENKINS = "jenkins";
	/**
	* @see io.opentelemetry.semconv.resource.attributes.ResourceAttributes#SERVICE_NAMESPACE
	*/
	public static final String SERVICE_NAMESPACE_JENKINS = "jenkins";

Let's support a customise option when configuring the OpenTelemetry to set what's the service name to be used.

Why

Several Jenkins coordinators to report to the same obs11 backend might clash if they have the same Jobs for any reason.

Proposals

Given the Jenkins URL then calculate the service name

Given the Jenkins URL https://foo.domain.com then creates the service name: foo.domain.com

Pros:

• Delegate the responsability to the plugin to decide the name based on the Jenkins URL.

Cons:

• If JenkinsURL is too verbose (IP address, k8s namespaces...) then it might be less user friendly.
• No customisation.

Provide a global settings to override the service name

Given a Global Settings called serviceName, with default value Jenkins, then update the service name.

Pros:

• User friendly
• More customisation.

Cons:

• Service name might not match with the Jenkins URL.

Another option ??

All metrics reported in Prometheus lack of contextual attributes (`host.name`, `service.name`...)

Plugin report

• Jenkins OpenTelemetry plugin version: 0.13
• OpenTelemetry Java SDK version: 1.2.0

Problem Description

All metrics reported by the Jenkins OpenTelemetry plugin lack of contextual attributes when ingested by Prometheus.
The cause is that the OpenTelemetry -> Prometheus integration ignores the OpenTelemetry Resource associated with the metrics and only captures the Metric Labels

ℹ️ The OpenTelemetry Metric data model specification don't indicate that "Resources" are part of the "Timeseries Model" (here):

Timeseries Model

In this low-level metrics data model, a Timeseries is defined by an entity consisting of several metadata properties:

• Metric name
• Label set
• Kind of point (integer, floating point, etc)
• Unit of measurement

Reproduction steps

Run Jenkins with the OpenTelemetry plugin and send OpenTelemetry data to an OpenTelemetry Collector with the Prometheus exporter. Verify the metrics exposed for Prometheus, they don't have the resource attributes as expected with the line of code

https://github.com/jenkinsci/opentelemetry-plugin/blob/opentelemetry-0.13/src/main/java/io/jenkins/plugins/opentelemetry/OpenTelemetrySdkProvider.java#L84

...
this.sdkMeterProvider = SdkMeterProvider.builder().setResource(resource).buildAndRegisterGlobal();

...
    private Resource buildResource() {
        return Resource.getDefault().merge(new JenkinsResourceProvider().createResource(null));
    }

Results in the Otel Collector Prometheus Receiver, lacking of contextual attributes

All metrics collected in Jenkins and ingested by Prometheus are lacking of contextual attributes that would enable to differentiate multiple Jenkins instances.

# HELP ci_pipeline_run_completed Job completed
# TYPE ci_pipeline_run_completed counter
ci_pipeline_run_completed 384
# HELP ci_pipeline_run_launched Job launched
# TYPE ci_pipeline_run_launched counter
ci_pipeline_run_launched 384
# HELP ci_pipeline_run_started Job started
# TYPE ci_pipeline_run_started counter
ci_pipeline_run_started 384
# HELP jenkins_agents_offline Number of offline agents
# TYPE jenkins_agents_offline gauge
jenkins_agents_offline 0
# HELP jenkins_agents_online Number of online agents
# TYPE jenkins_agents_online gauge
jenkins_agents_online 1
# HELP jenkins_agents_total Number of agents
# TYPE jenkins_agents_total gauge
jenkins_agents_total 1
# HELP jenkins_disk_usage_bytes Disk usage of first level folder in JENKINS_HOME.
# TYPE jenkins_disk_usage_bytes gauge
jenkins_disk_usage_bytes 6.86112768e+08
# HELP jenkins_queue_blocked Number of blocked items in queue
# TYPE jenkins_queue_blocked gauge
jenkins_queue_blocked 0
# HELP jenkins_queue_buildable Number of buildable items in queue
# TYPE jenkins_queue_buildable gauge
jenkins_queue_buildable 0
# HELP jenkins_queue_left Total count of left items
# TYPE jenkins_queue_left counter
jenkins_queue_left 768
# HELP jenkins_queue_time_spent_millis Total time spent in queue by items
# TYPE jenkins_queue_time_spent_millis counter
jenkins_queue_time_spent_millis 654213
# HELP jenkins_queue_waiting Number of waiting items in queue
# TYPE jenkins_queue_waiting gauge
jenkins_queue_waiting 0
# HELP process_cpu_load Process CPU load
# TYPE process_cpu_load gauge
process_cpu_load 0.011472335740346274
# HELP process_cpu_time Process CPU time
# TYPE process_cpu_time counter
process_cpu_time 5.77532942e+11
# HELP runtime_jvm_gc_count The number of collections that have occurred for a given JVM garbage collector.
# TYPE runtime_jvm_gc_count counter
runtime_jvm_gc_count{gc="G1 Old Generation"} 0
runtime_jvm_gc_count{gc="G1 Young Generation"} 218
# HELP runtime_jvm_gc_time Time spent in a given JVM garbage collector in milliseconds.
# TYPE runtime_jvm_gc_time counter
runtime_jvm_gc_time{gc="G1 Old Generation"} 0
runtime_jvm_gc_time{gc="G1 Young Generation"} 3803
# HELP runtime_jvm_memory_area Bytes of a given JVM memory area.
# TYPE runtime_jvm_memory_area gauge
runtime_jvm_memory_area{area="heap",type="committed"} 3.8797312e+08
runtime_jvm_memory_area{area="heap",type="max"} 4.294967296e+09
runtime_jvm_memory_area{area="heap",type="used"} 2.58488944e+08
runtime_jvm_memory_area{area="non_heap",type="committed"} 2.9888512e+08
runtime_jvm_memory_area{area="non_heap",type="used"} 2.73452656e+08
# HELP runtime_jvm_memory_pool Bytes of a given JVM memory pool.
# TYPE runtime_jvm_memory_pool gauge
runtime_jvm_memory_pool{pool="CodeHeap 'non-nmethods'",type="committed"} 2.555904e+06
runtime_jvm_memory_pool{pool="CodeHeap 'non-nmethods'",type="max"} 5.8368e+06
runtime_jvm_memory_pool{pool="CodeHeap 'non-nmethods'",type="used"} 2.440448e+06
runtime_jvm_memory_pool{pool="CodeHeap 'non-profiled nmethods'",type="committed"} 3.0867456e+07
runtime_jvm_memory_pool{pool="CodeHeap 'non-profiled nmethods'",type="max"} 1.22912768e+08
runtime_jvm_memory_pool{pool="CodeHeap 'non-profiled nmethods'",type="used"} 3.0779776e+07
runtime_jvm_memory_pool{pool="CodeHeap 'profiled nmethods'",type="committed"} 6.6125824e+07
runtime_jvm_memory_pool{pool="CodeHeap 'profiled nmethods'",type="max"} 1.22908672e+08
runtime_jvm_memory_pool{pool="CodeHeap 'profiled nmethods'",type="used"} 6.377856e+07
runtime_jvm_memory_pool{pool="Compressed Class Space",type="committed"} 2.1127168e+07
runtime_jvm_memory_pool{pool="Compressed Class Space",type="max"} 1.073741824e+09
runtime_jvm_memory_pool{pool="Compressed Class Space",type="used"} 1.6845888e+07
runtime_jvm_memory_pool{pool="G1 Eden Space",type="committed"} 9.5420416e+07
runtime_jvm_memory_pool{pool="G1 Eden Space",type="used"} 2.3068672e+07
runtime_jvm_memory_pool{pool="G1 Old Gen",type="committed"} 2.87309824e+08
runtime_jvm_memory_pool{pool="G1 Old Gen",type="max"} 4.294967296e+09
runtime_jvm_memory_pool{pool="G1 Old Gen",type="used"} 2.30177392e+08
runtime_jvm_memory_pool{pool="G1 Survivor Space",type="committed"} 5.24288e+06
runtime_jvm_memory_pool{pool="G1 Survivor Space",type="used"} 5.24288e+06
runtime_jvm_memory_pool{pool="Metaspace",type="committed"} 1.78208768e+08
runtime_jvm_memory_pool{pool="Metaspace",type="used"} 1.59607984e+08
# HELP system_memory_utilization System memory utilization (0.0 to 1.0)
# TYPE system_memory_utilization gauge
system_memory_utilization 0.9886684417724609
# HELP system_paging_usage System swap usage
# TYPE system_paging_usage gauge
system_paging_usage{state="free"} 9.86185728e+08
system_paging_usage{state="used"} 2.235039744e+09
# HELP system_paging_utilization System swap utilization (0.0 to 1.0)
# TYPE system_paging_utilization gauge
system_paging_utilization 0.69384765625

Results in Elastic, having contextual attributes

Contextual attributes passed as OpenTelemetry Resource: labels.jenkins_url: http://localhost:8080/jenkins/, labels.service_namespace: jenkins, host.hostname...

{
  "_index": "apm-7.13.0-metric-000001",
  "_type": "_doc",
  "_id": "uqTQ03kBP_hj0IByzUfI",
  "_version": 1,
  "_score": null,
  "fields": {
    "jenkins.disk.usage.bytes": [
      758106110
    ],
    "jenkins.agents.online": [
      1
    ],
    "observer.name": [
      "instance-0000000003"
    ],
    "labels.jenkins_url": [
      "http://localhost:8080/jenkins/"
    ],
    "service.node.name": [
      "MacBook-Pro.localdomain"
    ],
    "host.hostname": [
      "MacBook-Pro.localdomain"
    ],
    "jenkins.queue.waiting": [
      1
    ],
    "service.language.name": [
      "java"
    ],
    "jenkins.queue.time_spent_millis": [
      469281
    ],
    "ci.pipeline.run.launched": [
      327
    ],
    "ci.pipeline.run.completed": [
      324
    ],
    "processor.event": [
      "metric"
    ],
    "agent.name": [
      "opentelemetry/java"
    ],
    "host.name": [
      "MacBook-Pro.localdomain"
    ],
    "host.os.type": [
      "macos"
    ],
    "service.name": [
      "jenkins"
    ],
    "processor.name": [
      "metric"
    ],
    "observer.version_major": [
      7
    ],
    "observer.hostname": [
      "20a3cc61eb94"
    ],
    "jenkins.agents.total": [
      1
    ],
    "metricset.name": [
      "app"
    ],
    "jenkins.agents.offline": [
      0
    ],
    "jenkins.queue.left": [
      653
    ],
    "event.ingested": [
      "2021-06-03T21:39:07.334Z"
    ],
    "observer.id": [
      "86b784f7-e659-45eb-9c85-85967e455027"
    ],
    "@timestamp": [
      "2021-06-03T21:39:06.763Z"
    ],
    "service.version": [
      "2.235.5"
    ],
    "jenkins.queue.blocked": [
      0
    ],
    "ci.pipeline.run.started": [
      327
    ],
    "observer.ephemeral_id": [
      "1254778f-91de-4c2a-8135-8a9dcb98184b"
    ],
    "observer.version": [
      "7.13.0"
    ],
    "host.os.platform": [
      "darwin"
    ],
    "ecs.version": [
      "1.8.0"
    ],
    "observer.type": [
      "apm-server"
    ],
    "jenkins.queue.buildable": [
      1
    ],
    "agent.version": [
      "1.2.0"
    ],
    "labels.service_namespace": [
      "jenkins"
    ]
  },
  "sort": [
    1622756346763
  ]
}

On Jenkins restart, jobs resuming cause "VerifyException: No span found for my-pipeline #yy""

2021-03-03 10:59:32.119+0000 [id=38]	INFO	jenkins.InitReactorRunner$1#onAttained: Started initialization
2021-03-03 10:59:32.536+0000 [id=52]	INFO	jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2021-03-03 10:59:32.599+0000 [id=42]	INFO	j.b.a.SecurityProviderInitializer#addSecurityProvider: Initializing Bouncy Castle security provider.
2021-03-03 10:59:32.846+0000 [id=42]	INFO	j.b.a.SecurityProviderInitializer#addSecurityProvider: Bouncy Castle security provider initialized.
2021-03-03 10:59:39.245+0000 [id=50]	INFO	jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
2021-03-03 10:59:39.261+0000 [id=45]	INFO	jenkins.InitReactorRunner$1#onAttained: Started all plugins
2021-03-03 10:59:44.443+0000 [id=48]	INFO	i.j.p.o.OpenTelemetrySdkProvider#initializeForGrpc: OpenTelemetry initialized with GRPC endpoint http://localhost:8200, authenticationHeader: BearerTokenAuthentication{tokenId='elastic-apm-secret-token'}
2021-03-03 10:59:44.452+0000 [id=51]	INFO	jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2021-03-03 10:59:44.537+0000 [id=43]	INFO	jenkins.InitReactorRunner$1#onAttained: System config loaded
2021-03-03 10:59:45.362+0000 [id=41]	INFO	jenkins.InitReactorRunner$1#onAttained: System config adapted
2021-03-03 10:59:45.363+0000 [id=45]	INFO	jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2021-03-03 10:59:45.363+0000 [id=49]	INFO	jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
2021-03-03 10:59:45.380+0000 [id=68]	INFO	hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Download metadata
2021-03-03 10:59:45.385+0000 [id=68]	INFO	hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Download metadata. 4 ms
2021-03-03 10:59:45.829+0000 [id=40]	INFO	o.s.c.s.AbstractApplicationContext#prepareRefresh: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@49427a32: display name [Root WebApplicationContext]; startup date [Wed Mar 03 11:59:45 CET 2021]; root of context hierarchy
2021-03-03 10:59:45.830+0000 [id=40]	INFO	o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@49427a32]: org.springframework.beans.factory.support.DefaultListableBeanFactory@506888dd
2021-03-03 10:59:45.846+0000 [id=40]	INFO	o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@506888dd: defining beans [authenticationManager]; root of factory hierarchy
2021-03-03 10:59:46.060+0000 [id=40]	INFO	o.s.c.s.AbstractApplicationContext#prepareRefresh: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@7425668a: display name [Root WebApplicationContext]; startup date [Wed Mar 03 11:59:46 CET 2021]; root of context hierarchy
2021-03-03 10:59:46.060+0000 [id=40]	INFO	o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@7425668a]: org.springframework.beans.factory.support.DefaultListableBeanFactory@2e01d1b8
2021-03-03 10:59:46.061+0000 [id=40]	INFO	o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@2e01d1b8: defining beans [filter,legacy]; root of factory hierarchy
2021-03-03 10:59:46.149+0000 [id=40]	INFO	jenkins.model.Jenkins#setInstallState: Install state transitioning from: DEVELOPMENT to : DEVELOPMENT
2021-03-03 10:59:46.159+0000 [id=40]	INFO	jenkins.InitReactorRunner$1#onAttained: Completed initialization
2021-03-03 10:59:48.926+0000 [id=74]	INFO	o.j.p.workflow.job.WorkflowRun#finish: test-scripted-pipeline-with-parallel #88 completed: FAILURE
2021-03-03 10:59:48.977+0000 [id=74]	WARNING	h.model.listeners.RunListener#report: RunListener failed
com.google.common.base.VerifyException: No span found for test-scripted-pipeline-with-parallel #88
	at com.google.common.base.Verify.verify(Verify.java:122)
	at com.google.common.base.Verify.verifyNotNull(Verify.java:157)
	at io.jenkins.plugins.opentelemetry.job.opentelemetry.OtelContextAwareAbstractRunListener.onCompleted(OtelContextAwareAbstractRunListener.java:48)
	at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:610)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$800(WorkflowRun.java:137)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:1037)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1473)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:489)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-03 10:59:49.022+0000 [id=74]	WARNING	h.model.listeners.RunListener#report: RunListener failed
com.google.common.base.VerifyException: No span found for test-scripted-pipeline-with-parallel #88
	at com.google.common.base.Verify.verify(Verify.java:122)
	at com.google.common.base.Verify.verifyNotNull(Verify.java:157)
	at io.jenkins.plugins.opentelemetry.job.opentelemetry.OtelContextAwareAbstractRunListener.onFinalized(OtelContextAwareAbstractRunListener.java:59)
	at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:255)
	at hudson.model.Run.onEndBuilding(Run.java:2042)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:636)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$800(WorkflowRun.java:137)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:1037)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1473)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:489)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-03 10:59:49.931+0000 [id=74]	INFO	o.j.p.workflow.job.WorkflowRun#finish: test-declarative-pipeline #150 completed: FAILURE
2021-03-03 10:59:49.932+0000 [id=74]	WARNING	h.model.listeners.RunListener#report: RunListener failed
com.google.common.base.VerifyException: No span found for test-declarative-pipeline #150
	at com.google.common.base.Verify.verify(Verify.java:122)
	at com.google.common.base.Verify.verifyNotNull(Verify.java:157)
	at io.jenkins.plugins.opentelemetry.job.opentelemetry.OtelContextAwareAbstractRunListener.onCompleted(OtelContextAwareAbstractRunListener.java:48)
	at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:610)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$800(WorkflowRun.java:137)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:1037)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1473)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:489)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-03 10:59:49.945+0000 [id=74]	WARNING	h.model.listeners.RunListener#report: RunListener failed
com.google.common.base.VerifyException: No span found for test-declarative-pipeline #150
	at com.google.common.base.Verify.verify(Verify.java:122)
	at com.google.common.base.Verify.verifyNotNull(Verify.java:157)
	at io.jenkins.plugins.opentelemetry.job.opentelemetry.OtelContextAwareAbstractRunListener.onFinalized(OtelContextAwareAbstractRunListener.java:59)
	at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:255)
	at hudson.model.Run.onEndBuilding(Run.java:2042)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:636)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$800(WorkflowRun.java:137)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:1037)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1473)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:489)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-03 10:59:49.979+0000 [id=74]	INFO	o.j.p.workflow.job.WorkflowRun#finish: unit-test-simple-parallel-scripted-pipeline #90 completed: FAILURE
2021-03-03 10:59:49.979+0000 [id=74]	WARNING	h.model.listeners.RunListener#report: RunListener failed
com.google.common.base.VerifyException: No span found for unit-test-simple-parallel-scripted-pipeline #90
	at com.google.common.base.Verify.verify(Verify.java:122)
	at com.google.common.base.Verify.verifyNotNull(Verify.java:157)
	at io.jenkins.plugins.opentelemetry.job.opentelemetry.OtelContextAwareAbstractRunListener.onCompleted(OtelContextAwareAbstractRunListener.java:48)
	at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:610)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$800(WorkflowRun.java:137)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:1037)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1473)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:489)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-03 10:59:49.986+0000 [id=74]	WARNING	h.model.listeners.RunListener#report: RunListener failed
com.google.common.base.VerifyException: No span found for unit-test-simple-parallel-scripted-pipeline #90
	at com.google.common.base.Verify.verify(Verify.java:122)
	at com.google.common.base.Verify.verifyNotNull(Verify.java:157)
	at io.jenkins.plugins.opentelemetry.job.opentelemetry.OtelContextAwareAbstractRunListener.onFinalized(OtelContextAwareAbstractRunListener.java:59)
	at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:255)
	at hudson.model.Run.onEndBuilding(Run.java:2042)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:636)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$800(WorkflowRun.java:137)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:1037)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1473)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:489)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-03 10:59:50.032+0000 [id=74]	INFO	o.j.p.workflow.job.WorkflowRun#finish: test-scripted-pipeline #189 completed: FAILURE
2021-03-03 10:59:50.032+0000 [id=74]	WARNING	h.model.listeners.RunListener#report: RunListener failed
com.google.common.base.VerifyException: No span found for test-scripted-pipeline #189
	at com.google.common.base.Verify.verify(Verify.java:122)
	at com.google.common.base.Verify.verifyNotNull(Verify.java:157)
	at io.jenkins.plugins.opentelemetry.job.opentelemetry.OtelContextAwareAbstractRunListener.onCompleted(OtelContextAwareAbstractRunListener.java:48)
	at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:610)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$800(WorkflowRun.java:137)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:1037)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1473)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:489)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-03 10:59:50.044+0000 [id=74]	WARNING	h.model.listeners.RunListener#report: RunListener failed
com.google.common.base.VerifyException: No span found for test-scripted-pipeline #189
	at com.google.common.base.Verify.verify(Verify.java:122)
	at com.google.common.base.Verify.verifyNotNull(Verify.java:157)
	at io.jenkins.plugins.opentelemetry.job.opentelemetry.OtelContextAwareAbstractRunListener.onFinalized(OtelContextAwareAbstractRunListener.java:59)
	at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:255)
	at hudson.model.Run.onEndBuilding(Run.java:2042)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:636)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$800(WorkflowRun.java:137)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:1037)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1473)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:489)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-03 10:59:50.082+0000 [id=74]	INFO	o.j.p.workflow.job.WorkflowRun#finish: test-trace-environment-variables-in-shell-step #70 completed: FAILURE
2021-03-03 10:59:50.082+0000 [id=74]	WARNING	h.model.listeners.RunListener#report: RunListener failed
com.google.common.base.VerifyException: No span found for test-trace-environment-variables-in-shell-step #70
	at com.google.common.base.Verify.verify(Verify.java:122)
	at com.google.common.base.Verify.verifyNotNull(Verify.java:157)
	at io.jenkins.plugins.opentelemetry.job.opentelemetry.OtelContextAwareAbstractRunListener.onCompleted(OtelContextAwareAbstractRunListener.java:48)
	at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:610)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$800(WorkflowRun.java:137)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:1037)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1473)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:489)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-03 10:59:50.094+0000 [id=74]	WARNING	h.model.listeners.RunListener#report: RunListener failed
com.google.common.base.VerifyException: No span found for test-trace-environment-variables-in-shell-step #70
	at com.google.common.base.Verify.verify(Verify.java:122)
	at com.google.common.base.Verify.verifyNotNull(Verify.java:157)
	at io.jenkins.plugins.opentelemetry.job.opentelemetry.OtelContextAwareAbstractRunListener.onFinalized(OtelContextAwareAbstractRunListener.java:59)
	at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:255)
	at hudson.model.Run.onEndBuilding(Run.java:2042)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:636)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$800(WorkflowRun.java:137)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:1037)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1473)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:489)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-03 10:59:50.110+0000 [id=33]	INFO	hudson.WebAppMain$3#run: Jenkins is fully up and running
2021-03-03 10:59:50.127+0000 [id=74]	INFO	o.j.p.workflow.job.WorkflowRun#finish: test-scripted-pipeline-with-failure #77 completed: FAILURE
2021-03-03 10:59:50.128+0000 [id=74]	WARNING	h.model.listeners.RunListener#report: RunListener failed
com.google.common.base.VerifyException: No span found for test-scripted-pipeline-with-failure #77
	at com.google.common.base.Verify.verify(Verify.java:122)
	at com.google.common.base.Verify.verifyNotNull(Verify.java:157)
	at io.jenkins.plugins.opentelemetry.job.opentelemetry.OtelContextAwareAbstractRunListener.onCompleted(OtelContextAwareAbstractRunListener.java:48)
	at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:610)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$800(WorkflowRun.java:137)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:1037)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1473)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:489)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-03 10:59:50.133+0000 [id=74]	WARNING	h.model.listeners.RunListener#report: RunListener failed
com.google.common.base.VerifyException: No span found for test-scripted-pipeline-with-failure #77
	at com.google.common.base.Verify.verify(Verify.java:122)
	at com.google.common.base.Verify.verifyNotNull(Verify.java:157)
	at io.jenkins.plugins.opentelemetry.job.opentelemetry.OtelContextAwareAbstractRunListener.onFinalized(OtelContextAwareAbstractRunListener.java:59)
	at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:255)
	at hudson.model.Run.onEndBuilding(Run.java:2042)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:636)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$800(WorkflowRun.java:137)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:1037)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1473)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:489)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-03 11:00:16.595+0000 [id=144]	INFO	o.j.p.workflow.job.WorkflowRun#finish: test-scripted-pipeline #190 completed: SUCCESS
2021-03-03 11:00:16.662+0000 [id=137]	INFO	o.j.p.workflow.job.WorkflowRun#finish: test-scripted-pipeline-with-failure #78 completed: FAILURE
2021-03-03 11:00:17.648+0000 [id=72]	INFO	o.j.p.workflow.job.WorkflowRun#finish: unit-test-simple-parallel-scripted-pipeline #91 completed: SUCCESS
2021-03-03 11:00:18.121+0000 [id=142]	INFO	o.j.p.workflow.job.WorkflowRun#finish: test-trace-environment-variables-in-shell-step #71 completed: SUCCESS
2021-03-03 11:00:24.427+0000 [id=142]	INFO	o.j.p.workflow.job.WorkflowRun#finish: test-scripted-pipeline-with-parallel #89 completed: SUCCESS
2021-03-03 11:00:25.583+0000 [id=142]	INFO	o.j.p.workflow.job.WorkflowRun#finish: test-declarative-pipeline #151 completed: SUCCESS

jenkins.* metrics are always 0

Version report

Jenkins and plugins versions report:

2.189.1 with latest master build.

• What Operating System are you using (both controller, and any agents involved in the problem)?

k8s

Reproduction steps

• Create some agents
• Search for any metrics jenkins.agents.online > 0

Results

Expected result:

There will be some metrics when a new agent is connected

Actual result:

0