Unique database servers per database host on map

[datadog]

Unique database servers per database host on map

[vgarmash-chwy]

Thank you. Additional questions:

1. What if my application connects to multiple databases? How the dd.service.mapping should be configured for multiple databases of the same Postgres or MySQL type?
2. How to add HAProxy, F5 (or any other SNMP resource) to the service map? Would be great in some near future to see examples of configuration for every integration that Datadog supports.
3. Would it be possible to monitor and see on Service Map the replication or any other types of connections between 2 databases? How the database could set its own unique service name?

Can someone in Datadog create article about how service maps work, where they got their names for services from and how to customize them.

[vgarmash-chwy]

@alxch- Thank you. I am very aware of these pages and all others that can be found in Google. The problem is the none of those pages are talking about from where service names are coming and how to change those. Now, thanks to @tylerbenson we know that DD_SERVICE_MAPPING is playing a crucial role in fixing incorrect names. Would be nice to see deeper article about all integrations too.

[vgarmash-chwy]

I just got those questions responded by the DD support. Short story is none of those things are currently supported and not clear if they would ever be. Here:

Hi Slava,

Thanks for reaching out and sorry for the delay on this thread.

1. What if my application connects to multiple databases? How the dd.service.mapping should be configured for multiple databases of the same Postgres type?
   As I explained previously, the default service name for a database service is its type, extracted from the jdbc connection url.
   We can supersede the default service name for a type of database with the system property dd.service.mapping. Currently, you can not have two databases of the same type called by the same JVM with two different service names. I can open a feature request on your behalf to discuss with engineering and product management if that's possibility that you would like to see in our java dd tracer. I will need a priority for it: nice to have/medium or high.

2. How to add HAProxy, F5 (or any other SNMP resource) to the service map? Would be great in some near future to see example for every integration that Datadog supports.

Our service map shows the services for which we get /have gotten traces and the interaction with each other.
As for the previous point, I can open a feature request on your behalf to discuss with engineering and product management. I will need a priority for it: nice to have/medium or high. Would you mind giving me more details on your expectations regarding this subject?

3. Would it be possible to monitor and see on Service Map the replication or any other types of connections between 2 databases? How the database could set its own unique service name?

I don't think that's currently possible. I'll double check with the APM Team.

Best Regards,
Cécile

It is disappointing because you would think that Service Map would draw full picture of your enterprise infrastructure (one of the marketing promises Ashley Miller made in their keynote https://www.youtube.com/watch?v=ozESALTbYaI)

[vgarmash-chwy]

Hey all! As an update we're taking a look at this here: #866

Very good. Seems like someone found the JDBC specification ;). Good job! Looking forward to it!

[tylerbenson]

@darin-holloway thanks for pointing that out. I assumed that Mariadb was just like Mysql. Adding all those additional formats took some time, but glad we caught it now. The PR is updated to support those.

[tylerbenson]

You could configure each host to map their mysql service instance to be host specific using dd.service.mapping (documented here)

[vgarmash-chwy]

@tylerbenson Thank you for responding, but the page you are referring to is not explaining meaning of "dd.service.mapping", but rather just mention it.

I am personally digging through all publicly shared datadog pages to understand how the code need to me tagged or configured in order to build nice service map as your guys show in presentations (https://www.datadoghq.com/videos/dash-keynote-creating-context-with-service-maps/).

So if I understand it correctly, in service.mapping we can specify all dependent services which our service would need? Do you have somewhere more detailed examples for applications with multiple databases, caches etc?

[tylerbenson]

Sorry it isn't explained better. Basically that config basically renames the service for any reported span from the key to the value of the config. For example, I could configure it with the following environment variable DD_SERVICE_MAPPING=mysql:config-service-mysql-db to have all instances of mysql spans be renamed.

This would eventually be fed into and updated in service maps, since the span relationships are the source of that data.

Does that help?

[tylerbenson]

For 1, we don't really have a good solution for that. Do you have any suggestions? The best I can think of is for you to manually override the service name in your code.

Beyond that, service maps are not my area of expertise. I suggest you contact support directly for further assistance.

[alxch-]

You may find more information in this article: https://www.datadoghq.com/blog/service-map/. You can also check this doc: https://docs.datadoghq.com/tracing/visualization/services_map/

• Basically, nodes on the service map are APM services and edges are calls from one service to another. There are parent/child relationships between spans in your traces, and that makes the edges.

If you need help on these topics, you can reach out to support team so that they can help you.

[aantono]

I've just opened a feature request with the DataDog Support Team on this topic.
We did discuss the DD_SERVICE_MAPPING route, but unfortunately that solution does not scale when you have hundreds of different applications that are using a common/shared bootstrap mechanism (container/launcher), as it is impossible to know what mappings should be configured for each. It also does not support the use-case, mentioned above, when a single application talks to more than one instance of the same DB kind (like 2 different Mongo DBs).

Some proposed solutions (from my side) were to introduce another JVM agent property (or multiple, depending on how sophisticated we want to get this), to enable use of DB Schema names/Message Queues/AWS SDK Resources/etc instead of the default/static DDTags.SERVICE_NAME

For example, for Mongo DB instead of using static mongo (https://github.com/DataDog/dd-trace-java/blob/master/dd-java-agent/instrumentation/mongo-3.1/src/main/java/datadog/trace/instrumentation/mongo/DDTracingCommandListener.java#L112) we can use the event.getDatabaseName() value (https://github.com/DataDog/dd-trace-java/blob/master/dd-java-agent/instrumentation/mongo-3.1/src/main/java/datadog/trace/instrumentation/mongo/DDTracingCommandListener.java#L94).
The same could apply for JDBC "services", for example instead of sqlserver name (the return value of dbInfo.getType() - https://github.com/DataDog/dd-trace-java/blob/master/dd-java-agent/instrumentation/jdbc/src/main/java/datadog/trace/instrumentation/jdbc/PreparedStatementInstrumentation.java#L128) we can return the JDBC URL (or even better, some cleaned-up human readable variant of it... i.e. instead of jdbc:sqlserver://SchemaName.dbhost.com:1433 return something like SchemaName. Though it will be a bit tricky, as different JDBC drivers have different URL schema patterns, so we will need to parse the names out depending on the variant, i.e. MySQL, Postgres, H2, SqlServer, etc. Tricky, but doable, and only need to load the patterns once, after that it's smooth sailing). Similarly things like RabbitMQ can use an Exchange as an alternative/uniquely distinguishing name.

The same goes for Java AWS SDK, which right now is just a dumping ground for ALL communications with AWS, which is hardly of any use! Instead, why not use the AWS Service name - awsServiceName as a disambiguating representation? (https://github.com/DataDog/dd-trace-java/blob/master/dd-java-agent/instrumentation/aws-java-sdk-1.11.0/src/main/java/datadog/trace/instrumentation/aws/v0/SpanDecorator.java#L48) Then we will see things like AmazonKinesisFirehose, 'Amazon S3', AmazonDynamoDBv2, etc. IMHO, those are a LOT more useful than the current namings.

I would be willing to contribute to a PR effort to get this support all across different instrumentations we there is support from DataDog.

[realark]

Hi @vgarmash-chewy, would you mind taking support up on their offer to submit a feature request? This would be a very useful feature to have, but we'll need to devote some first-class time to work on it. A feature request through support is the fastest way to get this prioritized.

[vgarmash-chwy]

@realark The future request has been opened. Please reach out to Cécile Terpin (Datadog customer support), who opened those for us.

[aantono]

@realark. We’ve been also working with Nelson Adams to get this feature requested from our end, so he can also provide some details. (Our offer to help with a PR is also on 😉)

[andrewsouthard1]

Hey all! As an update we're taking a look at this here: #866

[tylerbenson]

@vgarmash-chewy unfortunately parsing JDBC urls are non-trivial since they're very inconsistent.

If anyone has particular Driver implementations that are important to them, please voice your opinion. Additionally, it would be helpful if you could provide a sample JDBC URL string (please don't include passwords).

Thanks!

[vgarmash-chwy]

Yes, I agree. If you look into https://download.oracle.com/otn-pub/jcp/jdbc-4_1-mrel-spec/jdbc4.1-fr-spec.pdf, paragraph 9.4.1, you'll see this:

The JDBC API defines a set of properties to identify and describe a DataSource
implementation. The actual set required for a specific implementation depends on
the type of DataSource object, that is, whether it is a basic DataSource object, a
ConnectionPoolDataSource object, or an XADataSource object. The only
property required for all DataSource implementations is description.

and then it following with list of standard properties, including "databaseName", which you are trying to read. I think it can work. I agree with you that it is not easy task since JDBC vendors are trying to hide as much as they can from the client.
Although if you can find the way to configure most of known JDBC drivers to provide Datadog with the proper database name, you can describe it in the documentation so who wants to get better Services Map, would follow your recommendation. You can try to use databaseName or maybe even put some label into the "description" property of the datasource, which you can parse in your instrumenter.

[darin-holloway]

Beyond vanilla Mysql Connector/J, we would need support for

Mariadb/aurora https://mariadb.com/kb/en/library/about-mariadb-connector-j/#connection-strings

ex:

jdbc:(mysql|mariadb):[replication:|failover:|sequential:|aurora:]//<hostDescription>[,<hostDescription>...]/[database][?<key1>=<value1>[&<key2>=<value2>]]

[tylerbenson]

Is anyone able to do a quick validation (we plan to release this early next week)?
Please download the latest snapshot here: https://oss.jfrog.org/artifactory/oss-snapshot-local/com/datadoghq/dd-java-agent/0.30.0-SNAPSHOT/dd-java-agent-0.30.0-SNAPSHOT.jar

To enable:

• System Property: -Ddd.trace.db.client.split-by-instance=true
• Environment Variable: DD_TRACE_DB_CLIENT_SPLIT_BY_INSTANCE=true

Thanks!

#881

[labbati]

Hello everybody. Version 0.30.0 is out which will let you assign a service name matching the database instance name to all db clients spans, as described by @tylerbenson.