`ArgumentError` thrown from auto-instrumentation code when running Datadog in parallel

[anandhasudhan]

`ArgumentError` thrown from auto-instrumentation code when running Datadog in parallel

[arielvalentin]

Thank you @anandhasudhan for the report.

We do not provide product support for vendor specific distributions.

Please open an issue with Datadog or contact them via your support contract: https://github.com/DataDog/dd-trace-rb/issues

[anandhasudhan]

@arielvalentin I've created a bug report with the dd-trace-rb team and they are currently investigating. (link)

I just wanted to provide additional context here in case it helps. I had initialized 1_datadog.rb first and then 2_opentelemetry.rb. I tried changing the order to initialize 1_opentelemetry.rb first and then 2_datadog.rb. The opposite seems to happen. ie. open telemetry code ends up calling data dog code and throws the ArgumentError.

Argument Error Stack trace

ArgumentError: wrong number of arguments (given 2, expected 3)
/usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/contrib/http/instrumentation.rb:91:in `annotate_span_with_response!': wrong number of arguments (given 2, expected 3) (ArgumentError)
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-instrumentation-net_http-0.21.1/lib/opentelemetry/instrumentation/net/http/patches/instrumentation.rb:39:in `block (2 levels) in request'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-instrumentation-net_http-0.21.1/lib/opentelemetry/instrumentation/net/http/patches/instrumentation.rb:38:in `tap'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-instrumentation-net_http-0.21.1/lib/opentelemetry/instrumentation/net/http/patches/instrumentation.rb:38:in `block in request'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-1.1.0/lib/opentelemetry/trace.rb:72:in `block in with_span'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-1.1.0/lib/opentelemetry/context.rb:87:in `with_value'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-1.1.0/lib/opentelemetry/trace.rb:72:in `with_span'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-1.1.0/lib/opentelemetry/trace/tracer.rb:29:in `in_span'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-instrumentation-net_http-0.21.1/lib/opentelemetry/instrumentation/net/http/patches/instrumentation.rb:31:in `request'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/contrib/http/instrumentation.rb:55:in `block in request'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/trace_operation.rb:192:in `block in measure'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/span_operation.rb:150:in `measure'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/trace_operation.rb:192:in `measure'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/tracer.rb:380:in `start_span'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/tracer.rb:160:in `block in trace'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/context.rb:43:in `activate!'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/tracer.rb:159:in `trace'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing.rb:16:in `trace'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-1.10.1/lib/datadog/tracing/contrib/http/instrumentation.rb:40:in `request'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-exporter-otlp-0.23.0/lib/opentelemetry/exporter/otlp/exporter.rb:163:in `block (2 levels) in send_bytes'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-exporter-otlp-0.23.0/lib/opentelemetry/exporter/otlp/exporter.rb:237:in `measure_request_duration'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-exporter-otlp-0.23.0/lib/opentelemetry/exporter/otlp/exporter.rb:163:in `block in send_bytes'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-exporter-otlp-0.23.0/lib/opentelemetry/exporter/otlp/exporter.rb:131:in `block in around_request'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-common-0.19.7/lib/opentelemetry/common/utilities.rb:92:in `block in untraced'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-1.1.0/lib/opentelemetry/context.rb:87:in `with_value'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-common-0.19.7/lib/opentelemetry/common/utilities.rb:91:in `untraced'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-exporter-otlp-0.23.0/lib/opentelemetry/exporter/otlp/exporter.rb:131:in `around_request'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-exporter-otlp-0.23.0/lib/opentelemetry/exporter/otlp/exporter.rb:155:in `send_bytes'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-exporter-otlp-0.23.0/lib/opentelemetry/exporter/otlp/exporter.rb:89:in `export'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-sdk-1.1.0/lib/opentelemetry/sdk/trace/export/batch_span_processor.rb:187:in `block in export_batch'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-sdk-1.1.0/lib/opentelemetry/sdk/trace/export/batch_span_processor.rb:187:in `synchronize'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-sdk-1.1.0/lib/opentelemetry/sdk/trace/export/batch_span_processor.rb:187:in `export_batch'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-sdk-1.1.0/lib/opentelemetry/sdk/trace/export/batch_span_processor.rb:170:in `block in work'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-sdk-1.1.0/lib/opentelemetry/sdk/trace/export/batch_span_processor.rb:159:in `loop'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-sdk-1.1.0/lib/opentelemetry/sdk/trace/export/batch_span_processor.rb:159:in `work'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/opentelemetry-sdk-1.1.0/lib/opentelemetry/sdk/trace/export/batch_span_processor.rb:180:in `block in reset_on_fork'
	from /usr/src/app/vendor/bundle/ruby/2.6.0/gems/hamelin-1.1.3/lib/hamelin/core_ext/threads.rb:14:in `block in initialize'

I came across this PR where it indicates that opentelemetry tracers have been built using the datadog auto-instrumentation library as a starting point and wonder if that could have any impact here.

[arielvalentin]

@anandhasudhan Yes, you are correct that the inspiration for the OTel instrumentation libraries came from DD Trace and was largely contributed by non-Datadog engineers.

We do not have any intention on maintaining interoperability with DD trace other than using it as a reference when implementing OTel Instrumentation.

Datadog does not provide any engineering support to the OTel Ruby instrumentation since that donation, and I will reiterate that we do not provide guarantees for compatibility with other distributions or 3rd party tracers. If we choose to, we will only provide interoperability with what is available in the specification, which is limited to non-vendor distributions of OpenTracing and OpenCensus.

In addition to that, we do not provide support for EoL versions of Ruby like 2.6 so if this is an issue with e.g. Net::HTTP having changed its method signature between versions, then you will have to downgrade to an earlier version of the instrumentations where that was compatible for both DD Trace and OTel.