The OpenTelemetry Operator is an implementation of a Kubernetes Operator.
The operator manages:
- OpenTelemetry Collector
- auto-instrumentation of the workloads using OpenTelemetry instrumentation libraries
You can install Opentelemetry Operator via Helm Chart from the opentelemetry-helm-charts repository. More information is available in here.
To install the operator in an existing cluster, make sure you have cert-manager
installed and run:
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
Once the opentelemetry-operator
deployment is ready, create an OpenTelemetry Collector (otelcol) instance, like:
kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: simplest
spec:
config: |
receivers:
otlp:
protocols:
grpc:
http:
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 15
batch:
send_batch_size: 10000
timeout: 10s
exporters:
logging:
service:
pipelines:
traces:
receivers: [otlp]
processors: []
exporters: [logging]
EOF
WARNING: Until the OpenTelemetry Collector format is stable, changes may be required in the above example to remain compatible with the latest version of the OpenTelemetry Collector image being referenced.
This will create an OpenTelemetry Collector instance named simplest
, exposing a jaeger-grpc
port to consume spans from your instrumented applications and exporting those spans via logging
, which writes the spans to the console (stdout
) of the OpenTelemetry Collector instance that receives the span.
The config
node holds the YAML
that should be passed down as-is to the underlying OpenTelemetry Collector instances. Refer to the OpenTelemetry Collector documentation for a reference of the possible entries.
At this point, the Operator does not validate the contents of the configuration file: if the configuration is invalid, the instance will still be created but the underlying OpenTelemetry Collector might crash.
The Operator does examine the configuration file to discover configured receivers and their ports. If it finds receivers with ports, it creates a pair of kubernetes services, one headless, exposing those ports within the cluster. The headless service contains a service.beta.openshift.io/serving-cert-secret-name
annotation that will cause OpenShift to create a secret containing a certificate and key. This secret can be mounted as a volume and the certificate and key used in those receivers' TLS configurations.
As noted above, the OpenTelemetry Collector format is continuing to evolve. However, a best-effort attempt is made to upgrade all managed OpenTelemetryCollector
resources.
In certain scenarios, it may be desirable to prevent the operator from upgrading certain OpenTelemetryCollector
resources. For example, when a resource is configured with a custom .Spec.Image
, end users may wish to manage configuration themselves as opposed to having the operator upgrade it. This can be configured on a resource by resource basis with the exposed property .Spec.UpgradeStrategy
.
By configuring a resource's .Spec.UpgradeStrategy
to none
, the operator will skip the given instance during the upgrade routine.
The default and only other acceptable value for .Spec.UpgradeStrategy
is automatic
.
The CustomResource
for the OpenTelemetryCollector
exposes a property named .Spec.Mode
, which can be used to specify whether the collector should run as a DaemonSet
, Sidecar
, or Deployment
(default). Look at this sample for reference.
A sidecar with the OpenTelemetry Collector can be injected into pod-based workloads by setting the pod annotation sidecar.opentelemetry.io/inject
to either "true"
, or to the name of a concrete OpenTelemetryCollector
, like in the following example:
kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: sidecar-for-my-app
spec:
mode: sidecar
config: |
receivers:
jaeger:
protocols:
thrift_compact:
processors:
exporters:
logging:
service:
pipelines:
traces:
receivers: [jaeger]
processors: []
exporters: [logging]
EOF
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: myapp
annotations:
sidecar.opentelemetry.io/inject: "true"
spec:
containers:
- name: myapp
image: jaegertracing/vertx-create-span:operator-e2e-tests
ports:
- containerPort: 8080
protocol: TCP
EOF
When there are multiple OpenTelemetryCollector
resources with a mode set to Sidecar
in the same namespace, a concrete name should be used. When there's only one Sidecar
instance in the same namespace, this instance is used when the annotation is set to "true"
.
The annotation value can come either from the namespace, or from the pod. The most specific annotation wins, in this order:
- the pod annotation is used when it's set to a concrete instance name or to
"false"
- namespace annotation is used when the pod annotation is either absent or set to
"true"
, and the namespace is set to a concrete instance or to"false"
The possible values for the annotation can be:
- "true" - inject
OpenTelemetryCollector
resource from the namespace. - "sidecar-for-my-app" - name of
OpenTelemetryCollector
CR instance in the current namespace. - "my-other-namespace/my-instrumentation" - name and namespace of
OpenTelemetryCollector
CR instance in another namespace. - "false" - do not inject
When using a pod-based workload, such as Deployment
or Statefulset
, make sure to add the annotation to the PodTemplate
part. Like:
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
labels:
app: my-app
annotations:
sidecar.opentelemetry.io/inject: "true" # WRONG
spec:
selector:
matchLabels:
app: my-app
replicas: 1
template:
metadata:
labels:
app: my-app
annotations:
sidecar.opentelemetry.io/inject: "true" # CORRECT
spec:
containers:
- name: myapp
image: jaegertracing/vertx-create-span:operator-e2e-tests
ports:
- containerPort: 8080
protocol: TCP
EOF
When using sidecar mode the OpenTelemetry collector container will have the environment variable OTEL_RESOURCE_ATTRIBUTES
set with Kubernetes resource attributes, ready to be consumed by the resourcedetection processor.
The operator can inject and configure OpenTelemetry auto-instrumentation libraries. Currently DotNet, Java, NodeJS and Python are supported.
To use auto-instrumentation, configure an Instrumentation
resource with the configuration for the SDK and instrumentation.
kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: my-instrumentation
spec:
exporter:
endpoint: http://otel-collector:4317
propagators:
- tracecontext
- baggage
- b3
sampler:
type: parentbased_traceidratio
argument: "0.25"
python:
env:
# Required if endpoint is set to 4317.
# Python autoinstrumentation uses http/proto by default
# so data must be sent to 4318 instead of 4317.
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://otel-collector:4318
dotnet:
env:
# Required if endpoint is set to 4317.
# Dotnet autoinstrumentation uses http/proto by default
# See https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/blob/888e2cd216c77d12e56b54ee91dafbc4e7452a52/docs/config.md#otlp
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://otel-collector:4318
EOF
The values for propagators
are added to the OTEL_PROPAGATORS
environment variable.
Valid values for propagators
are defined by the OpenTelemetry Specification for OTEL_PROPAGATORS.
The value for sampler.type
is added to the OTEL_TRACES_SAMPLER
envrionment variable.
Valid values for sampler.type
are defined by the OpenTelemetry Specification for OTEL_TRACES_SAMPLER.
The value for sampler.argument
is added to the OTEL_TRACES_SAMPLER_ARG
environment variable. Valid values for sampler.argument
will depend on the chosen sampler. See the OpenTelemetry Specification for OTEL_TRACES_SAMPLER_ARG for more details.
The above CR can be queried by kubectl get otelinst
.
Then add an annotation to a pod to enable injection. The annotation can be added to a namespace, so that all pods within that namespace wil get instrumentation, or by adding the annotation to individual PodSpec objects, available as part of Deployment, Statefulset, and other resources.
Java:
instrumentation.opentelemetry.io/inject-java: "true"
NodeJS:
instrumentation.opentelemetry.io/inject-nodejs: "true"
Python:
instrumentation.opentelemetry.io/inject-python: "true"
DotNet:
instrumentation.opentelemetry.io/inject-dotnet: "true"
OpenTelemetry SDK environment variables only:
instrumentation.opentelemetry.io/inject-sdk: "true"
The possible values for the annotation can be
"true"
- inject andInstrumentation
resource from the namespace."my-instrumentation"
- name ofInstrumentation
CR instance in the current namespace."my-other-namespace/my-instrumentation"
- name and namespace ofInstrumentation
CR instance in another namespace."false"
- do not inject
If nothing else is specified, instrumentation is performed on the first container available in the pod spec. In some cases (for example in the case of the injection of an Istio sidecar) it becomes necessary to specify on which container(s) this injection must be performed.
For this, it is possible to fine-tune the pod(s) on which the injection will be carried out.
For this, we will use the instrumentation.opentelemetry.io/container-names
annotation for which we will indicate one or more pod names (.spec.containers.name
) on which the injection must be made:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-deployment-with-multiple-containers
spec:
selector:
matchLabels:
app: my-pod-with-multiple-containers
replicas: 1
template:
metadata:
labels:
app: my-pod-with-multiple-containers
annotations:
instrumentation.opentelemetry.io/inject-java: "true"
instrumentation.opentelemetry.io/container-names: "myapp,myapp2"
spec:
containers:
- name: myapp
image: myImage1
- name: myapp2
image: myImage2
- name: myapp3
image: myImage3
In the above case, myapp
and myapp2
containers will be instrumented, myapp3
will not.
By default, the operator uses upstream auto-instrumentation libraries. Custom auto-instrumentation can be configured by overriding the image fields in a CR.
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: my-instrumentation
spec:
java:
image: your-customized-auto-instrumentation-image:java
nodejs:
image: your-customized-auto-instrumentation-image:nodejs
python:
image: your-customized-auto-instrumentation-image:python
dotnet:
image: your-customized-auto-instrumentation-image:dotnet
The Dockerfiles for auto-instrumentation can be found in autoinstrumentation directory. Follow the instructions in the Dockerfiles on how to build a custom container image.
You can configure the OpenTelemetry SDK for applications which can't currently be autoinstrumented by using inject-sdk
in place of (e.g.) inject-python
or inject-java
. This will inject environment variables like OTEL_RESOURCE_ATTRIBUTES
, OTEL_TRACES_SAMPLER
, and OTEL_EXPORTER_OTLP_ENDPOINT
, that you can configure in the Instrumentation
, but will not actually provide the SDK.
instrumentation.opentelemetry.io/inject-sdk: "true"
The operator allows specifying, via the feature gates, which languages the Instrumentation resource may instrument.
These feature gates must be passed to the operator via the --feature-gates
flag.
The flag allows for a comma-delimited list of feature gate identifiers.
Prefix a gate with '-' to disable support for the corresponding language.
Prefixing a gate with '+' or no prefix will enable support for the corresponding language.
If a language is enabled by default its gate only needs to be supplied when disabling the gate.
Language | Gate | Default Value |
---|---|---|
.NET | operator.autoinstrumentation.dotnet |
enabled |
Language not specified in the table are always supported and cannot be disabled.
The OpenTelemetry Operator comes with an optional component, the Target Allocator (TA). When creating an OpenTelemetryCollector Custom Resource (CR) and setting the TA as enabled, the Operator will create a new deployment and service to serve specific http_sd_config
directives for each Collector pod as part of that CR. It will also change the Prometheus receiver configuration in the CR, so that it uses the http_sd_config from the TA. The following example shows how to get started with the Target Allocator:
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: collector-with-ta
spec:
mode: statefulset
targetAllocator:
enabled: true
config: |
receivers:
prometheus:
config:
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: [ '0.0.0.0:8888' ]
exporters:
logging:
service:
pipelines:
metrics:
receivers: [prometheus]
processors: []
exporters: [logging]
Behind the scenes, the OpenTelemetry Operator will convert the Collector’s configuration after the reconciliation into the following:
receivers:
prometheus:
config:
global:
scrape_interval: 1m
scrape_timeout: 10s
evaluation_interval: 1m
scrape_configs:
- job_name: otel-collector
honor_timestamps: true
scrape_interval: 10s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
follow_redirects: true
http_sd_configs:
- follow_redirects: false
url: http://collector-with-ta-targetallocator:80/jobs/otel-collector/targets?collector_id=$POD_NAME
exporters:
logging:
service:
pipelines:
metrics:
receivers: [prometheus]
processors: []
exporters: [logging]
Note how the Operator added a global
section and a new http_sd_configs
to the otel-collector
scrape config, pointing to a Target Allocator instance it provisioned.
More info on the TargetAllocator can be found here.
The OpenTelemetry Operator follows the same versioning as the operand (OpenTelemetry Collector) up to the minor part of the version. For example, the OpenTelemetry Operator v0.18.1 tracks OpenTelemetry Collector 0.18.0. The patch part of the version indicates the patch level of the operator itself, not that of OpenTelemetry Collector. Whenever a new patch version is released for OpenTelemetry Collector, we'll release a new patch version of the operator.
By default, the OpenTelemetry Operator ensures consistent versioning between itself and the managed OpenTelemetryCollector
resources. That is, if the OpenTelemetry Operator is based on version 0.40.0
, it will create resources with an underlying OpenTelemetry Collector at version 0.40.0
.
When a custom Spec.Image
is used with an OpenTelemetryCollector
resource, the OpenTelemetry Operator will not manage this versioning and upgrading. In this scenario, it is best practice that the OpenTelemetry Operator version should match the underlying core version. Given a OpenTelemetryCollector
resource with a Spec.Image
configured to a custom image based on underlying OpenTelemetry Collector at version 0.40.0
, it is recommended that the OpenTelemetry Operator is kept at version 0.40.0
.
We strive to be compatible with the widest range of Kubernetes versions as possible, but some changes to Kubernetes itself require us to break compatibility with older Kubernetes versions, be it because of code incompatibilities, or in the name of maintainability. Every released operator will support a specific range of Kubernetes versions, to be determined at the latest during the release.
We use cert-manager
for some features of this operator and the third column shows the versions of the cert-manager
that are known to work with this operator's versions.
The OpenTelemetry Operator might work on versions outside of the given range, but when opening new issues, please make sure to test your scenario on a supported version.
OpenTelemetry Operator | Kubernetes | Cert-Manager |
---|---|---|
v0.75.0 | v1.19 to v1.26 | v1 |
v0.74.0 | v1.19 to v1.26 | v1 |
v0.73.0 | v1.19 to v1.26 | v1 |
v0.72.0 | v1.19 to v1.26 | v1 |
v0.71.0 | v1.19 to v1.25 | v1 |
v0.70.0 | v1.19 to v1.25 | v1 |
v0.69.0 | v1.19 to v1.25 | v1 |
v0.68.0 | v1.19 to v1.25 | v1 |
v0.67.0 | v1.19 to v1.25 | v1 |
v0.66.0 | v1.19 to v1.25 | v1 |
v0.64.1 | v1.19 to v1.25 | v1 |
v0.63.1 | v1.19 to v1.25 | v1 |
v0.62.1 | v1.19 to v1.25 | v1 |
v0.61.0 | v1.19 to v1.25 | v1 |
v0.60.0 | v1.19 to v1.25 | v1 |
v0.59.0 | v1.19 to v1.24 | v1 |
v0.58.0 | v1.19 to v1.24 | v1 |
v0.57.2 | v1.19 to v1.24 | v1 |
v0.56.0 | v1.19 to v1.24 | v1 |
v0.55.0 | v1.19 to v1.24 | v1 |
v0.54.0 | v1.19 to v1.24 | v1 |
v0.53.0 | v1.19 to v1.24 | v1 |
v0.52.0 | v1.19 to v1.23 | v1 |
Please see CONTRIBUTING.md.
In addition to the core responsibilities the operator project requires approvers and maintainers to be responsible for releasing the project. See RELEASE.md for more information and release schedule.
Approvers (@open-telemetry/operator-approvers):
- Benedikt Bongartz, Red Hat
- Yuri Oliveira Sa, Red Hat
Emeritus Approvers:
- Anthony Mirabella, AWS
- Dmitrii Anoshin, Splunk
- Jay Camp, Splunk
- James Bebbington, Google
- Owais Lone, Splunk
- Pablo Baeyens, DataDog
Target Allocator Maintainers (@open-telemetry/operator-ta-maintainers):
- Anthony Mirabella, AWS
- Kristina Pathak, Lightstep
- Sebastian Poxhofer
Maintainers (@open-telemetry/operator-maintainers):
- Jacob Aronoff, Lightstep
- Pavol Loffay, Red Hat
- Vineeth Pothulapati, Timescale
Emeritus Maintainers
- Alex Boten, Lightstep
- Bogdan Drutu, Splunk
- Juraci Paixão Kröhling, Grafana Labs
- Tigran Najaryan, Splunk
Learn more about roles in the community repository.
Thanks to all the people who already contributed!