imandra-ai/ocaml-opentelemetry

Instrumentation for https://opentelemetry.io

Opentelemetry

This project provides an API for instrumenting server software using opentelemetry, as well as connectors to talk to opentelemetry software such as jaeger.

• library opentelemetry should be used to instrument your code and possibly libraries. It doesn't communicate with anything except a backend (default: dummy backend)
• library opentelemetry-client-ocurl is a backend that communicates via http+protobuf with some collector (otelcol, datadog-agent, etc.)

License

MIT

Features

• basic traces
• basic metrics
• basic logs
• nice API
• interface with lwt
• sync collector relying on ocurl
  • batching, perf, etc.
• async collector relying on ocurl-multi
• interface with logs (carry context around)
• implicit scope (via ambient-context)

Use

For now, instrument traces/spans, logs, and metrics manually:

module Otel = Opentelemetry
let (let@) f x = f x

let foo () =
  let@ scope = Otel.Trace.with_  "foo"
      ~attrs:["hello", `String "world"] in
  do_work();
  Otel.Metrics.(
    emit [
      gauge ~name:"foo.x" [int 42];
    ]);
  do_more_work();
  ()

Setup

If you're writing a top-level application, you need to perform some initial configuration.

1. Set the service_name;
2. configure our ambient-context dependency with the appropriate storage for your environment — TLS, Lwt, Eio ... (see their docs for more details);
3. and install a Collector (usually by calling your collector's with_setup function.)

For example, if your application is using Lwt, and you're using ocurl as your collector, you might do something like this:

let main () =
  Otel.Globals.service_name := "my_service";
  Otel.GC_metrics.basic_setup();

  Ambient_context.with_storage_provider (Ambient_context_lwt.storage ()) @@ fun () ->
  Opentelemetry_client_ocurl.with_setup () @@ fun () ->
  (* … *)
  foo ();
  (* … *)

Configuration

The library is configurable via Opentelemetry.Config, via the standard opentelemetry env variables, or with some custom environment variables.

• OTEL_EXPORTER_OTLP_ENDPOINT sets the http endpoint to send signals to
• OTEL_OCAML_DEBUG=1 to print some debug messages from the opentelemetry library ide
• OTEL_RESOURCE_ATTRIBUTES sets a comma separated list of custom resource attributes

Collector opentelemetry-client-ocurl

This is a synchronous collector that uses the http+protobuf format to send signals (metrics, traces, logs) to some other collector (eg. otelcol or the datadog agent).

Collector opentelemetry-client-cohttp-lwt

This is a Lwt-friendly collector that uses cohttp to send signals to some other collector (e.g. otelcol). It must be run inside a Lwt_main.run scope.

Opentelemetry-trace

The optional library opentelemetry.trace, present if trace is installed, provides a collector for trace. This collector forwards and translates events from trace into opentelemetry. It's only useful if there also is also a OTEL collector.

License

MIT

Semantic Conventions

Not supported yet.

• metrics
• traces
• resources

ocaml-opentelemetry's Issues

opentelemetry-cohttp: make sure to emit data before termination

I was playing around with this little example that uses opentelemetry_client_cohttp_lwt:

let run () =
  let open Lwt.Syntax in
  let* () =
    Opentelemetry_lwt.Trace.with_ "my trace" @@ fun _scope ->
    print_endline "bar" |> Lwt.return
  in
  Lwt.return_unit

let () =
  Opentelemetry_lwt.Globals.service_name := "my service";
  let config =
    Opentelemetry_client_cohttp_lwt.Config.make ~debug:true
      ~url:"http://localhost:4318" ()
  in
  Opentelemetry_client_cohttp_lwt.with_setup ~config () @@ fun () ->
  Lwt_main.run (run ())

And was surprised to not see any traces in the Jaeger UI (same example with ocurl works fine), especially because I could see a send span in logs

bar                                    
send spans { resource = ... }
opentelemetry: exiting…

Adding a little delay fix the issue

diff --git a/src/bin/main.ml b/src/bin/main.ml
index fc28bf1..1416c72 100644
--- a/src/bin/main.ml
+++ b/src/bin/main.ml
@@ -6,6 +6,7 @@ let run () =
     Opentelemetry_lwt.Trace.with_ "my trace" @@ fun _scope ->
     print_endline "bar" |> Lwt.return
   in
+  let* () = Lwt_unix.sleep 2.0 in
   Lwt.return_unit
 
 let () =

My understanding is that the cleanup function doesn't have the time to finish before the program terminate.

let cleanup () =
  if !debug_ then Printf.eprintf "opentelemetry: exiting…\n%!";
  Lwt.async (fun () ->
      let* () = emit_all_force httpc encoder in
      Httpc.cleanup httpc;
      Lwt.return ())

I see that we use a Lwt.async, which doesn't wait for the resolution of the promise. See for example this code:

let cleanup () =
  Lwt.async (fun () ->
      let open Lwt.Syntax in
      print_endline "start cleanup";
      (* simulate delay *)
      let* () = Lwt_unix.sleep 2.0 in
      print_endline "end cleanup";
      Lwt.return_unit)

let run () =
  print_endline "hello";
  cleanup () |> Lwt.return

let () = Lwt_main.run (run ())

And the output:

hello                                  
start cleanup

So in the case of opentelemetry_client_cohttp_lwt, the let* () = emit_all_force httpc encoder doesn't have time to complete before the program terminates.

Should we make sure that all data is emitted before the program terminates (both after "normal" completion and after explicit interruption)?

I guess one way to fix this would be to make the cleanup function return a unit Lw.t to be able to properly wait for its resolution. But the problem is that this is part of a common interface that different backends implement, so I don't think it's possible to force a lwt value. Maybe we can make it parametric or something?

`dune build` error: library "curl" not found

ocaml-opentelemetry/src/client-ocurl/dune

Line 4 in f436a1b

(libraries opentelemetry opentelemetry.atomic curl pbrt threads mtime

Is it a typo of ocurl?

http semantic conventions

https://opentelemetry.io/blog/2023/http-conventions-declared-stable/

use new thread-local-storage where needed

Collector that uses EIO

Thanks for this API and the backends! It's a great project. In addition to the existing collectors, it would be great to have one that uses EIO, which is nicer than Lwt.

I was also wondering if you had any advice for how best to integrate opentelemetry. In particular, I would rather use the cohttp-lwt collector over the ocurl one since it doesn't require having libcurl, but since our codebase already uses lwt in multiple places, with multiple Lwt_main.run scopes, adding a new toplevel Lwt_main.run scope is fairly annoying.

Should we handle redirection

I was testing the cohttp client with the jaeger all in one image. I set http://localhost:4318/ as the config.url, which makes the client to send traces to http://localhost:4318//v1/traces (notice the double //. Also see #38).

I didn't notice any errors in the terminal, but no traces showed up. It turns out that this is because a 301 redirect to /v1/traces was returned, and we do not handle redirects in the code.

I think we should handle redirects or at least fail/log something out to the console. wdyt?

metrics for self diagnostic

https://opentelemetry.io/docs/reference/specification/error-handling/#self-diagnostics

• spans for batch export?
• metrics for export
  • number of dropped items
  • batch queue size
  • number of spans/metrics exported so far
  • always export errors?

track all environment variables for config

https://opentelemetry.io/docs/reference/specification/sdk-environment-variables/

Allow for more verbose debugging information

At the moment, enabling debugging via OTEL_OCAML_DEBUG=true ./binary.exe only prints information like the following:

opentelemetry: send http POST to http://127.0.0.1:5996/v1/traces (963B)
opentelemetry: send http POST to http://127.0.0.1:5996/v1/traces (963B)

I'd like, additionally, the ability to print on every single span entered and exited — effectively a verbose mode.

Is this better attached to OTEL_OCAML_DEBUG=true, implemented as a new payload like OTEL_OCAML_DEBUG=verbose, or as a separate env-var, like OTEL_OCAML_VERBOSE=true?

support exceptions

https://opentelemetry.io/docs/specs/semconv/exceptions/exceptions-spans/

occasional deadlock in another program

I suspect I get an occasional deadlock in another project that is caused by the threaded ocurl backend.

I think it might be helpful to refactor the code so that:

• implementations with thread and without thread are totally distinct
• implementation with thread has few, if any, lock. Ideally just a thread-safe queue.