Reproduction steps:
Run the code with loki and grafana for checking how the logs are delivered.
#[tokio::main]
async fn main() {
let (loki, task) = tracing_loki::layer("{loki_url_here}", HashMap::new(), HashMap::new()).unwrap();
tokio::spawn(task);
tracing_subscriber::registry().with(loki).init();
let span = error_span!("main", foo = "bar");
async {
// Notice that this log event has `foo` field just like the parent span
tracing::info!(foo = "baz", bruh = "other", "log event");
}
.instrument(span)
.await;
}
The log event in Grafana will be seen as:
{
"message": "log event",
// As you can see, the field value from the log event is lost, and contains only span's value
"foo": "bar",
"bruh": "other",
"_spans": [
"main"
],
"_target": "app",
"_module_path": "app",
"_file": "app/src/main.rs",
"_line": 10
}
This happens because spans are flattened into a single object. I am pretty sure it is done due to LogQL
limitation that it doesn't support arrays.
Potential Solutions
Unconditionally name all fields that come from spans with the following approach:
[{span_name}_[{span_index}_]]{field_name}
span_name
and span_index
can be omitted if the resulting log event's field_name
is entirely unique.
span_index
is the number index in the list of spans. It can be omitted if there is no other span with the exact same name in the event. I suppose we can have several spans of the same name if the code that creates spans names them the same, or there were recursive function calls.
With this approach, we will optimize for the regular case where it is very unlikely that there are collisions in field names. The downside is that the information from the same span can now be available under different field names dynamically if there are collisions, which makes LogQL queries for the same field name not workable.
Maybe tracing_loki
could expose ways to format the resulting log event from spans and log event data in a configurable way so that users could decide how they want to resolve conflicts and how to lay out the log events in Loki at all? It could be exposed in terms of a callback that formats a JSON value as the most flexible way to provide this capability. This will also allow renaming the standard fields _file/_line/_module_path
or dropping them if not needed just as users decide.