christophergrant/databricks-opentelemetry

databricks-opentelemetry's Issues

Expose Structured Streaming backlog metrics

Streaming sources in Spark emit backlog metrics that indicate how behind a stream is. These metrics are helpful in determining a stream's performance relative to its source.

Streaming sources such as Autoloader, Kinesis, Kafka, and Delta emit these metrics.

This issue is to track efforts related to exposing these as metrics that can be centralized using OpenTelemetry.

Grafana automated install

Capture cluster event log events

This issue is to track any efforts related to emitting metrics or logs that relate to a cluster's event log.

Tracing examples and configs when model serving / calling LLM

When serving models / LLMs, latency is often very important but can be difficult to attribute, especially in complicated systems with multiple steps and systems.

Logs can be super helpful for troubleshooting, but collating logs across multiple systems isn't very easy and logs aren't easily consumed by humans. Tracing is often seen as a better alternative as it gives gantt charts out of the box, even with multiple systems with tools like Jaeger.

This issue is to track showing how to implement tracing for Databricks ML serving applications.

Jaeger automated install

Expose advanced scan metrics

It's helpful to detect things like cloud storage throttling, request count, etc. We should find a way to expose these metrics to OpenTelemetry.