[Documentation]: OTEL tracing

[jay-steyn]

[Documentation]: OTEL tracing

[bwoebi]

Regarding 1., also people who want to use OTel API for their code, but also want to use the automatic integrations dd-trace-php provides.

Do you know if there a plans at Datadog to make an application instrumented with dd-trace-php, able to send signals (or at least traces) to an OTEL collector with an otlp receiver? Or will Datadog fully embrace the OpenTelemetry PHP API and SDKs and keep dd-trace-php as a RAD, out-of-the-box library, for a smooth onboarding with Datadog?

Neither is planned, at least currently. Technically, we provide the raw span data with \dd_trace_serialize_closed_spans(), which could be used at the end of a request to transform our spans into a format otlp understands.

But yes, overall our target are definitely Datadog users. We don't aim to explicitly prevent using our extension for non-Datadog purposes, but we do not officially support it either.

I see the big picture, but with the current state of the documentation, I did not understand that.

Yes, I agree, we're going to look at improving the documentation accordingly, as you're definitely not the first one to misunderstand the capabilities we provide.

[PROFeNoM]

Hey @jay-steyn 👋

At first glance, it appears to me that you aren't using the datadog-agent, especially given the two following pieces of information:

"agent_url": "collector:4317"
[...]
                               Diagnostics                               
agent_error => Received HTTP/0.9 when not allowed
Diagnostic checks => failed

As illustrated in this documentation, to leverage the Datadog instrumentation, you'll need to use the Datadog Agent. This way, you'll be able to get both OTel-generated and DDog-generated spans in the Datadog UI.

I believe you should be able to install the Datadog Agent for your docker application using our public documentation (e.g., this one), but I'll be here to help you out if you need any help in the process 😃

---

For instance, if you are using docker compose and want to trace a service called wordpress, you could fundamentally have the following:

Your datadog-agent service:

  datadog-agent:
    image: datadog/agent:latest
    environment:
      DD_API_KEY: <your API key>
      DD_SITE: <your site, certainly datadoghq.com or datadoghq.eu>
      DD_LOGS_ENABLED: true
    networks:
      - default
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
      #- /opt/datadog-agent/run:/opt/datadog-agent/run:rw

and your traced wordpress service:

  wordpress:
    image: whatever
    networks:
      - default
    environment:
      <OTel configs, for instance>
      DD_AGENT_HOST: datadog-agent
      DD_TRACE_OTEL_ENABLED: true
      <Other DD configs as needed, for instance>
    depends_on:
      - datadog-agent

This is just an example, but this should give you the rough picture :)

[jay-steyn]

Hi, okay thanks for this.

I thought you could use Datadog instrumentation WITHOUT the need for the DD Agent and using the OTEL collector exclusively (refer to this #2339)

[bwoebi]

Hey @jay-steyn - no, it's just about the ability to send spans created by OTEL to datadog.

And the Jaeger traces in the OTEL PR were just to demonstrate that using OTEL spans with an external collector still works when dd-trace-php is enabled, i.e. nothing breaks by the mere presence of dd-trace-php.

[jay-steyn]

Hey @jay-steyn - no, it's just about the ability to send spans created by OTEL to datadog.

And the Jaeger traces in the OTEL PR were just to demonstrate that using OTEL spans with an external collector still works when dd-trace-php is enabled, i.e. nothing breaks by the mere presence of dd-trace-php.

Okay thanks. I will look at another approach.

[clemherreman]

Hello @bwoebi, I stumbled upon @jay-steyn's issue because I was confused too.

Did I (eventually :) ) understand correctly that Custom Instrumentation of PHP Applications with the OpenTelemetry API (and #2332) is to allow to have both (1) dd-trace-php instrumentation, collected by a Datadog agent, and (2) OpenTelemetry SDK instrumentation, collected by either a Datadog agent or an OTEL collector+Datadog exporter, all in the same application?

This is done by implementing in the dd-trace-php library an OpenTelemetry-compatible tracer, and using that tracer in both instrumentation : the dd-trace-php one, and the OpenTelemetry SDK one. Is that correct?

So that means that an application, instrumented with only dd-trace-php, cannot send traces to an OTEL collector with an otlp receiver ? 

Thank you for your time :)

[PROFeNoM]

The purpose of #2332 is to 1. allow the use of other Datadog Products (e.g., Profiling, AppSec) on OTel-generated spans without having to do anything, 2. leverage Datadog's auto-instrumentation, and 3. be able to use the OTel API to instrument your application with open standards (manual or automatic).

have both (1) dd-trace-php instrumentation, collected by a Datadog agent, and (2) OpenTelemetry SDK instrumentation, collected by either a Datadog agent or an OTEL collector+Datadog exporter, all in the same application?

1. dd-trace-php instrumentation, collected by a Datadog agent,: Datadog spans created from automatic or manual instrumentation will be collected by the Datadog Agent
2. OpenTelemetry SDK instrumentation, collected by either a Datadog agent or an OTEL collector+Datadog exporter: OTel-created spans (i.e., Spans created using the OTel API, whether it comes from manual or automatic instrumentation) will use Datadog spans under the hood. The Datadog agent will collect these Datadog spans.

This is done by implementing in the dd-trace-php library an OpenTelemetry-compatible tracer

Technically speaking, Datadog's PHP Library is replacing at runtime part of the OTel SDK to ensure interoperability of the APIs—i.e., consistency of information exchange between the Datadog API and the OTel API. Strictly speaking, we are not implementing OTel's Tracer; we are implementing its spans and associated contexts.

and using that tracer in both instrumentation: the dd-trace-php one and the OpenTelemetry SDK one. Is that correct?

Datadog spans are used in both instrumentations, so could you say that the Datadog Tracer is used in both instrumentations... It depends on what you exactly mean by "tracer."

So that means that an application instrumented with only dd-trace-php cannot send traces to an OTEL collector with an OTLP receiver?

That's not the point, indeed.
Spans created from the dd-trace-php (i.e., using Datadog's API or from automatic instrumentation) won't be sent to, for instance, an OTel collector.
However, spans created using the OTel API should be sent to your OTel collector, as they have always been.

Does that answer your questions? 🙏 Don't hesitate to ask questions. If you're unsure, others may not understand either, so we must improve our documentation and communication. Feedback is very much welcome.

[clemherreman]

Thank you @PROFeNoM this is much clearer to me, with your detailed explanations.

1. allow the use of other Datadog Products (e.g., Profiling, AppSec) on OTel-generated spans without having to do anything, 2. leverage Datadog's auto-instrumentation, and 3. be able to use the OTel API to instrument your application with open standards (manual or automatic).

So, the main target are people that:

• have an application instrumented by OTEL API
• uses Datadog
• wants to use additionnal features, like Profiling or AppSec, on that application, without re-instrumenting with dd-trace-php ?

I see the need, sadly it's not my need :)

Technically speaking, Datadog's PHP Library is replacing at runtime part of the OTel SDK to ensure interoperability of the APIs—i.e., consistency of information exchange between the Datadog API and the OTel API. Strictly speaking, we are not implementing OTel's Tracer; we are implementing its spans and associated contexts.

I see the big picture, but with the current state of the documentation, I did not understand that.

Spans created from the dd-trace-php (i.e., using Datadog's API or from automatic instrumentation) won't be sent to, for instance, an OTel collector.
However, spans created using the OTel API should be sent to your OTel collector, as they have always been.

Do you know if there a plans at Datadog to make an application instrumented with dd-trace-php, able to send signals (or at least traces) to an OTEL collector with an otlp receiver? Or will Datadog fully embrace the OpenTelemetry PHP API and SDKs and keep dd-trace-php as a RAD, out-of-the-box library, for a smooth onboarding with Datadog?

dd-trace-php have been really stable, well maintained so far, and I don't know if https://github.com/open-telemetry/opentelemetry-php will be as good. They hit their first v1.0 on october which is not that long ago.

_Also, I want to use observability to hunt bug in my code, I don't want to have to hunt bugs in my 3rd party observability code 🙄 _

Does that answer your questions? 🙏 Don't hesitate to ask questions. If you're unsure, others may not understand either, so we must improve our documentation and communication. Feedback is very much welcome.

Very much so, thank you. It's a real treat to have answer directly from the developers, and that fast.