hrxi/tracing-loki

A tracing layer for shipping logs to Grafana Loki

tracing-loki

A tracing layer for Grafana Loki.

Documentation

https://docs.rs/tracing-loki

Usage

Add this to your Cargo.toml:

[dependencies]
tracing-loki = "0.1"

Example

use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;
use std::process;
use url::Url;

#[tokio::main]
async fn main() -> Result<(), tracing_loki::Error> {
    let (layer, task) = tracing_loki::builder()
        .label("host", "mine")?
        .extra_field("pid", format!("{}", process::id()))?
        .build_url(Url::parse("http://127.0.0.1:3100").unwrap())?;

    // We need to register our layer with `tracing`.
    tracing_subscriber::registry()
        .with(layer)
        // One could add more layers here, for example logging to stdout:
        // .with(tracing_subscriber::fmt::Layer::new())
        .init();

    // The background task needs to be spawned so the logs actually get
    // delivered.
    tokio::spawn(task);

    tracing::info!(
        task = "tracing_setup",
        result = "success",
        "tracing successfully set up",
    );

    Ok(())
}

tracing-loki's Issues

[Feature Request]: graceful shutdown

i'm using tracing-loki in axum based web service, can't figure out how to make sure loki message flush to server before shutdown.

(if this is still not supported, maybe we can add shutdown method to tracing_loki::Layer. i can make a PR if this is ok.)

Https danger_accept_invalid_certs

I'm sorry if this is already possible, looking to ignore and invalid certificates and I could not figure out how to implement this. It appears that it's not exposed and I didn't really want to modify the current source.

Is it something that can be done already?

Span/Trace ID missing in `SerializedEvent`

Hey,

Maybe i have the wrong way of wanting to use Loki together with Tempo for tracing, but i feel like having a Span or Trace ID on SerializedEvent would make lots of sense to be able to look up one log line inside of a tracing system like Tempo or Jaeger.

Question: Possible to not miss logs without sleep?

I'm pretty new to Rust so sorry if this should be obvious.

Looking at the example given in examples it calls tokio::time::sleep(Duration::from_secs(1)).await;. Now if I remove that line my loki instance will not receive a log. In my understanding that is because the background reporting thread is killed when the program exits - before it has a chance to deliver the log to loki.

So my question is: Is there a way to ensure all logs have been delivered before exiting the program? Thanks.

Content is always escaped?

Hey there, first of all thanks so much for this! Such a big help in day to day logging needs.

I'm currently writing a quite involved interface that exchanges data between an ecommerce platform and an ERP system and makes heavy use of logging.

One thing I noticed is that all log lines that get sent to loki are fully escaped.
Is this a Grafana thing, or are they escaped within this crate? I would love to view the logs unescaped for easier handling.

My setup (roughly):

    let (layer, task) = tracing_loki::layer(
        loki_url,
        vec![host, app, log_env].into_iter().collect(),
        Default::default(),
    )?;
    tracing_subscriber::registry()
        .with(layer)
        .with(tracing_subscriber::fmt::Layer::new())
        .with(tracing_subscriber::EnvFilter::from_str(
            "info,tiberius=error,hyper=error",
        )?)
        .init();

Example (a bit mangled to hammer down the point):

                let diff = format_text_diff(&jp.body_html, &sp.body_html);
                warn!(
                    "Beschreibungstext bei {} abweichend.
Abweichungen: {diff}

JTL: '{}',
Shopify: '{}'",
                    jp.handle, jp.body_html, sp.body_html
                );

What gets displayed in grafana:

"Beschreibungstext bei besser-grillen abweichend.\nAbweichungen: Gelöscht in Position 261:\t\t\" \"Hinzugefügt in Position 261:\t\" \"\n\nJTL: ....',\nShopify: ...'\n\nTrace:\n \n ...\n"

In the CLI the correct, unescaped output from the tracing_subscriber layer is shown.

For completeness: I made a small wrapper to include the backtrace in warnings. Shouldn't modiy any escaping tho.

#[macro_export(local_inner_macros)]
macro_rules! warn {
    // warn!("a {} event", "log")
    ($($arg:tt)+) => {{
        let mut a = std::format!($($arg)+);
        a += "\n\nTrace:\n";
        let b = std::backtrace::Backtrace::force_capture();
        let bs = b.to_string();
        a += &bs;
        log::warn!("{a}")
    }};
}

HTTP status client error (429 Too Many Requests)

I am getting this:

2023-10-09T13:04:07.696099Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=1 backoff_time=0ns error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)
2023-10-09T13:04:07.702310Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=2 backoff_time=500ms error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)
2023-10-09T13:04:09.162408Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=1 backoff_time=0ns error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)
2023-10-09T13:04:09.176558Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=1 backoff_time=0ns error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)
2023-10-09T13:04:09.178456Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=2 backoff_time=500ms error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)
2023-10-09T13:04:10.427208Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=1 backoff_time=0ns error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)
2023-10-09T13:04:10.431438Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=2 backoff_time=500ms error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)
2023-10-09T13:04:11.967807Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=1 backoff_time=0ns error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)
2023-10-09T13:04:11.970397Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=2 backoff_time=500ms error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)

let (loki_layer, loki_task_ctl, loki_task) = tracing_loki::builder()
    .label("host", gethostname::gethostname().to_str().unwrap())
    .unwrap()
    .build_controller_url(
        tracing_loki::url::Url::parse("http://localhost:3100").unwrap(),
    )
    .unwrap();
tokio::spawn(loki_task);

loki.yml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - store: tsdb
      object_store: filesystem
      schema: v12
      index:
        prefix: index_
        period: 24h

analytics:
  reporting_enabled: false

strip ansi codes from messages

not really an issue, more a question.

My logs contain a lot of ansi esc codes to have a colorful stdout outpout. This is hard to read in Grafana.
I would like to strip the ansi codes from messages before shipping them to loki. How should I proceed?

Support using event targets as labels

I had a hell of a time figuring out how to actually filter events by target in Grafana, because it appears they're not provided as labels: https://github.com/hrxi/tracing-loki/blob/master/src/lib.rs#L241

Having dug into Loki more, I understand why this choice was made (as it may exceed cardinality limits) but I would at least like to have the option.

Renaming target to _target also just makes it more annoying to use the json parser, as I have to do | json target="_target" instead of just | json target.

Missing support for Loki Multi-Tenancy

Hi, i was playing with this project and i quickly found out that its missing support multi-tenancy in loki.

I have already started working on fix.

My implementation involves passing tenant_id: Option<String> to layer function and subsequently to BackgroundTask::new() and setting a default header for http_client.
I need to test it out and after that i'll do PR.

Thanks Sam

Fields of spans and log events that collide with each other are lost

Reproduction steps:

Run the code with loki and grafana for checking how the logs are delivered.

#[tokio::main]
async fn main() {
    let (loki, task) = tracing_loki::layer("{loki_url_here}", HashMap::new(), HashMap::new()).unwrap();
    tokio::spawn(task);
    tracing_subscriber::registry().with(loki).init();
 
    let span = error_span!("main", foo = "bar");

    async {
        // Notice that this log event has `foo` field just like the parent span
        tracing::info!(foo = "baz", bruh = "other", "log event");
    }
    .instrument(span)
    .await;
}

The log event in Grafana will be seen as:

{
  "message": "log event",
  // As you can see, the field value from the log event is lost, and contains only span's value
  "foo": "bar",
  "bruh": "other",
  "_spans": [
    "main"
  ],
  "_target": "app",
  "_module_path": "app",
  "_file": "app/src/main.rs",
  "_line": 10
}

This happens because spans are flattened into a single object. I am pretty sure it is done due to LogQL limitation that it doesn't support arrays.

Potential Solutions

Unconditionally name all fields that come from spans with the following approach:

[{span_name}_[{span_index}_]]{field_name}

span_name and span_index can be omitted if the resulting log event's field_name is entirely unique.
span_index is the number index in the list of spans. It can be omitted if there is no other span with the exact same name in the event. I suppose we can have several spans of the same name if the code that creates spans names them the same, or there were recursive function calls.

With this approach, we will optimize for the regular case where it is very unlikely that there are collisions in field names. The downside is that the information from the same span can now be available under different field names dynamically if there are collisions, which makes LogQL queries for the same field name not workable.

Maybe tracing_loki could expose ways to format the resulting log event from spans and log event data in a configurable way so that users could decide how they want to resolve conflicts and how to lay out the log events in Loki at all? It could be exposed in terms of a callback that formats a JSON value as the most flexible way to provide this capability. This will also allow renaming the standard fields _file/_line/_module_path or dropping them if not needed just as users decide.

[FEATURE REQUEST]: 2 Way SSL

I have my current Loki server set up to demand client certificates for authentication, although it seems that this library makes HTTPS possible, its not clear if I can configure tracing_loki to provide certificates to Loki during the handshake. Just looking for clarification if that's possible.

Crash when running in docker

Hey there, I want to migrate my logging to loki, and successfully got my code to use this crate locally. I can see all the interesting things in grafana, all my test run, everything is great.

Then I deploy to AWS EC2 with docker-compose, and my app won't run anymore. I don't even get panic, the process is just killed. I only get a 139 exit code, which apparently is a docker code for a SEGV (https://stackoverflow.com/a/69763409).

After adding a few printlns to my code, it seems that this line is responsible for the crash:

    let (layer, task) = tracing_loki::layer(
        loki_url,
        vec![host, app, log_env].into_iter().collect(),
        Default::default(),
    )?;

I basically copied it from the examples. Do you know what could be going wrong here?

Potential bug in reserved label name

Reading the source code, I suppose this line is wrong. Shouldn't it check for "level" label name instead?

tracing-loki/src/lib.rs

Lines 397 to 399 in 07fac0d

	if labels.contains_key("label") {
	return Err(Error(ErrorI::ReservedLabelLevel));
	}

Dependencies versions

Hello, first of all, thank you for creating this crate!
Do you have a specific reason to use such specific versions in the Cargo.toml?

I'd suggest the requirement can be relaxed a bit for crates that have a proper semantic versioning workflow. From:

...
tokio = { version = "1.17.0", features = ["sync"] }
tokio-stream = "0.1.8"
tracing = "0.1.32"
...

To:

...
tokio = { version = "1", features = ["sync"] }
tokio-stream = "0.1"
tracing = "0.1"
...

This gives downstream crates a little bit of leeway about shared dependencies.

Would you consider reviewing a PR addressing this?

snappy encoding is actually not infallible

I got this crash when my app was generating lots of logs: snappy encoding is infallible: TooBig { given: 4894810188, max: 4294967295 }

According to the snappy documentation encode() can actually fail if the input is too big.

I think tracing-loki should not panic in such case but just ignore the logs. And maybe log a warning?

Add license

Noticed that the crate states it's licensed under MIT/Apache 2.0 but there's no reference to that in the repo itself, would be best to add a LICENSE file to ensure there's no confusion. Thanks for the crate!