Is OpenTracing suitable for streaming/publish-subscribe systems?

[opentracing]

Is OpenTracing suitable for streaming/publish-subscribe systems?

[yurishkuro]

OpenTracing can be used with streaming / pub-sub. We have an internal project integrating tracing with Kafka, for example.

I think the main question is what you want to use tracing for in this scenario. For system analysis the lineage is often very important. With OpenTracing, we can correlate messages processed by different nodes in the pipeline and build an execution graph, similar to the call graph we get from regular RPC use case.

[OStrekalovsky]

And how we can handle situation when multiple services which consume the same source message want to finish started span which is single per message from source and generate differenct spans as result of processing whole trace? As i read, span (which is created when source message is created) may be closed only once. Or there exists another approach to create spans for that case?
It there an implementation of OpenTracing that can handle this case?

[yurishkuro]

In our model the span does not represent the time message spends in the queue. Spans are meant to be single-process, so there is a span, started and finished, in the producer that encapsulates the work of putting the message into the queue. Each consumer starts its own span upon reading the message, and those spans encapsulate the work done by each consumer. You can think of it in terms of AWS's lambda architecture - a handler function does not even need to know how it gets the message, from a queue or via an RPC call, therefore its OpenTracing instrumentation should not be different in each scenario either.

[OStrekalovsky]

So that "short life spans" should be linked only by means of "FollowsFrom" reference type?
And how can i calculate for example total time from creating source message to the end of processing? Only by traverse from the end to the beggining of the trace and calculate it by myself? Is there an existed tracing system that can handle that for me?

[yurishkuro]

FollowsFrom or yet another non-standard ref type. FollowsFrom ref type usually keeps the next span in the same trace. Sometimes you want the consumer to start a new trace, but link to the parent trace via some custom span ref type.

When spans are in the same trace, e.g. Zipkin will still display the total duration of the trace for you.

[mvpmvh]

Is there an example/snippet of tracing spans using FollowsFrom (or some other means) with published messages? I need a visual