Errors During Instrumentation (JMS and others)

[datadog]

Errors During Instrumentation (JMS and others)

[realark]

@tony-schellenberg I don't think failing to resolve net.bytebuddy.dynamic.Nexus would affect the JMS instrumentation. I think in this case that's not actually an error due to the class being dynamically generated.

I can take a look through your full debug log if you can send those.

[tony-schellenberg]

I think I can do that. Do you just need all the DataDog lines?

[realark]

Yes, that will do it. Thanks!

[tony-schellenberg]

I'm just looking at the logs. Is there a secure way for me to transfer them to you? I don't think there's anything sensitive in them but it's probably better if I don't just post them. Should I open a ticket with support and provide them there and give you the ticket number here?

[realark]

Yeah going through the support system and posting the ticket number is the best way to get those logs to me.

[tony-schellenberg]

I have posted a log to ticket #171383

[realark]

I took a look through the logs. I think the missing Nexus class is a red herring. I don't see the JMS Instrumentation attempting to apply. Which JMS client (and version) are you running with?

[tony-schellenberg]

The JMS implementation is com.sun.messaging.mq:imq:3.5. The jms version is 1.1.

I know this is an old client. In our code, we are using the javax type such as javax.jms.MessageProducer with this library.

[tylerbenson]

@tony-schellenberg Is that even available in maven central? I only see version 4.2+ there.
We'd be happy to run it through some tests to verify compatibility, but that becomes a challenge if it's not in maven central.

[tony-schellenberg]

I don't know if it's there or not. We had the product since before it was OpenMQ back when it was paid. We have our own Maven server that mirrors maven central but also has other artifacts so this one might not be generally available.

Given that it supports the javax.jms interfaces, can you think of a reason that it wouldn't instrument? Is there something that I can look for in it or a way that I can run the tests?

[tylerbenson]

@tony-schellenberg When running with debug logging enabled, I'd expect to see these kinds of messages:

[Test worker] DEBUG datadog.trace.agent.tooling.Instrumenter$Default - Applying instrumentation: okhttp -- datadog.trace.instrumentation.okhttp3.OkHttp3Instrumentation on sun.misc.Launcher$AppClassLoader@15db9742
[Test worker] DEBUG datadog.trace.agent.tooling.AgentInstaller$LoggingListener - Transformed class okhttp3.OkHttpClient -- sun.misc.Launcher$AppClassLoader@15db9742
[Test worker] DEBUG datadog.trace.bootstrap.ExceptionLogger - Failed to handle exception in instrumentation

Another message you should search for in the logs is Instrumentation muzzled, which might give indication as to why the library might be skipped.

[tony-schellenberg]

I can't see anything in the logs referring to instrumentation being muzzled.

Looking at this https://github.com/DataDog/dd-trace-java/blob/master/dd-java-agent/instrumentation/jms/src/main/java/datadog/trace/instrumentation/jms/JMSMessageProducerInstrumentation.java#L41

I unpacked the imq library and did a bit of digging. There is a concrete implementation that implements javax.jms.MessageProducer. I'm don't have much understanding of how the auto instrumentation works. Is there some other obvious reason why it wouldn't be able to find it that I can dig into?

[realark]

Generally the way our instrumentation works:

1. A matcher sees a class it could potentially instrument (like in your jms-producer link)
2. The classpath is checked to ensure it's safe to apply instrumentation.
3. Instrumentation applies or logs a failure to apply.

Any of these stages would have produced something in the log file.

I think the issue is we're skipping instrumentation because the class name starts with com.sun. We skip "core" namespaces for performance reasons.

https://github.com/datadog/dd-trace-java/blob/master/dd-java-agent/agent-tooling/src/main/java/datadog/trace/agent/tooling/AgentInstaller.java#L70

Can you confirm the package-namespace of the client you're trying to instrument?

[tony-schellenberg]

I think you might have figured it out. The one class I'm looking at so far is com.sun.messaging.jmq.jmsclient.MessageProducerImpl

[realark]

Great, that explains it. We should be good to whitelist com.sun.messaging.. Do you think that will cover your client's namespace?

[tony-schellenberg]

From what I can see of it, I think that should be good. Thanks for helping me out with this.

[tylerbenson]

Fix is included in 0.17.0.