open-telemetry/opentelemetry-cpp

The OpenTelemetry C++ Client

OpenTelemetry C++

The C++ OpenTelemetry client.

Project Status

Stable across all 3 signals i.e. Logs, Metrics, and Traces.

See Spec Compliance Matrix to understand which portions of the specification has been implemented in this repo.

Supported C++ Versions

Code shipped from this repository generally supports the following versions of C++ standards:

• ISO/IEC 14882:2014 (C++14)
• ISO/IEC 14882:2017 (C++17)
• ISO/IEC 14882:2020 (C++20)

Any exceptions to this are noted in the individual README.md files.

Please note that supporting the C Programming Language is not a goal of the current project.

Supported Development Platforms

Our CI pipeline builds and tests on following x86-64 platforms:

Platform	Build type
ubuntu-22.04 (GCC 10, GCC 12, Clang 14)	CMake, Bazel
ubuntu-20.04 (GCC 9.4.0 - default compiler)	CMake, Bazel
ubuntu-20.04 (GCC 9.4.0 with -std=c++14/17/20 flags)	CMake, Bazel
macOS 12.7 (Xcode 14.2)	Bazel
Windows Server 2019 (Visual Studio Enterprise 2019)	CMake, Bazel
Windows Server 2022 (Visual Studio Enterprise 2022)	CMake

In general, the code shipped from this repository should build on all platforms having C++ compiler with supported C++ standards.

Dependencies

Please refer to Dependencies.md for OSS Dependencies and license requirements.

Installation

Please refer to INSTALL.md.

Getting Started

As an application owner or the library author, you can find the getting started guide and reference documentation on opentelemetry-cpp.readthedocs.io

The examples/simple directory contains a minimal program demonstrating how to instrument a small library using a simple processor and console exporter, along with build files for CMake and Bazel.

Contributing

See CONTRIBUTING.md

We meet weekly, and the time of the meeting alternates between Monday at 13:00 PT and Wednesday at 9:00 PT. The meeting is subject to change depending on contributors' availability. Check the OpenTelemetry community calendar for specific dates and Zoom meeting links.

Meeting notes are available as a public Google doc. For edit access, get in touch on Slack.

Maintainers (@open-telemetry/cpp-maintainers):

• Ehsan Saei
• Lalit Kumar Bhasin, Microsoft
• Marc Alff, Oracle
• Tom Tan, Microsoft

Approvers (@open-telemetry/cpp-approvers):

• Josh Suereth, Google
• WenTao Ou, Tencent

Emeritus Maintainer/Approver/Triager:

• Alolita Sharma
• Emil Mikulic
• Jodee Varney
• Johannes Tax
• Max Golovanov
• Reiley Yang
• Ryan Burn

Thanks to all the people who have contributed

Release Schedule

See the release notes for existing releases.

See the project milestones for details on upcoming releases. The dates and features described in issues and milestones are estimates, and subject to change.

opentelemetry-cpp's Issues

Prometheus Exporter HTTP Server Support

Is your feature request related to a problem?
Yes. Currently, Java and Python do not implement an HTTP server in their language’s Prometheus exporter while node.js and Go do implement a server. For the C++ implementation, we have a choice between implementing the HTTP server or not implementing it. Benefits of implementing the HTTP service include that it is user friendly and conforms to the Prometheus exporter structure (Java and Python split it up into two pieces). Cons of implementing the HTTP service include this implementation could be the extra work that comes with creating it, managing it, and closing it.

Describe the solution you'd like
Implementing the HTTP server would make use of the C++ Prometheus exporter easier for end users. It also would make it easier for members in the future to contribute when working on related features.

Describe alternatives you've considered
Alternatively, not implementing the HTTP server would still be a feasible option. Java and Python both don’t implement an HTTP server in their exporter, instead opting to split up the implementation into two separate pieces.

Add exporter support for spans that aren't stored in a single contiguous segment

The current exporter implementation assumes, that spans are passed as a contiguous array. This might not be the case for spans stored in circular buffers (#52).

The exporter interface should be adapted to support exporting spans that aren't stored as a contiguous array.

Add a Changelog or Release Schedule

Is your feature request related to a problem?

Currently, there is no changelog for the C++ client. New contributors therefore have to pore over the codebase to determine which project features have been implemented. When looking for ways to contribute, they are forced to scroll through the issues list without any context as to which ones are priorities for the community.

Describe the solution you'd like

Adding a changelog similar to the ones present in the Java and Python clients (under the "Release Schedule" sections) gives new contributors a sense of how mature the project is and which features are currently under development. They also see which issues are slated for development either in the present of the near future and can focus their efforts there.

Describe alternatives you've considered

Tagging issues with priority or building out a feature completeness matrix will mitigate the current issues but still fail to outline the project's overarching trajectory which is important for contributors as they can gauge where and when their expertise will be most impactful.

Add external documentation for API/SDK functionality

Is your feature request related to a problem?
Yes. Currently there is minimal documentation about the built functionality in the C++ API/SDK. This can make it difficult for users and contributors alike to get started with the library.

Describe the solution you'd like
I would like some external documentation similar to Python’s readthedocs (https://opentelemetry-python.readthedocs.io/en/stable/index.html) pages. This feature is easily expandable and will allow for more documentation to be created as more functionality is built out.

Describe alternatives you've considered

1. Simply using the OpenTelemetry specification docs (https://github.com/open-telemetry/opentelemetry-specification). I believe this approach is sub-optimal for two reasons. First, not all functionality listed in the specification has been built in the C++ repository. This approach can make it difficult for contributors and users to gauge which features are built and which are not. Second, the languages OpenTelemetry supports are very different and each have their own idiomatic approaches to implementing the spec. The specification documents does not elaborate on the C++ specific implementation details. This can cause confusion for contributors and users.

2. Docs generation tools such as Doxygen (https://www.doxygen.nl/index.html). However, these tools provide very low-level documentation not suitable for many users. These docs can also be difficult to navigate. Because of this, they should be used to bolster existing documentation, not stand on their own.

Additional context
I believe examples are the best place to start adding documentation. This helps users understand what the library has to offer and how they can use the library in their own programs. Additionally, it gives contributors an idea of how the components interact and the purpose of each component. This combined with Doxygen would create quite comprehensive documentation.

Clarify the scope of nostd

In OpenTelemetry C++ we've introduced nostd for ABI compatibility, basically everything exposed by the API package will not have dependency on std::* types and C++ exceptions, as none of them would give ABI compatibility guarantee.

To keep the project manageable, we would need to avoid pitfalls such like maintaining a complete set of C++ standard library, or implement a subset and eventually drift away from the standard. This issue serves the purpose of listing the nostd types that we introduced, and the reason why we need them.

nostd::function_ref

nostd::pair

nostd::shared_ptr

nostd::span

nostd::string_view

nostd::unique_ptr

nostd::variant

Decide on requirements for C++ library

A few topics that need decisions:

• Bare C or C++ interface.
• If C++ then minimum C++ version to target (e.g. C++98, 11 or newer).
• Header-only library or no. If not, do we provide precompiled binaries and for which platforms.
• Do we depend on C++ std or no.
• Are exceptions allowed in the API and in the implementation or not.
• What are external dependencies (other libraries that this library will use and that become a transitive dependency for end users).

Opinions are welcome here in the form of comments and we will also discuss this in SIG meetings.

Enhance the existing exporter interface

The current naive exporter interface has a few shortcomings that need to be addressed:

• It must work with span circular buffers where the data isn't stored in a single contiguous segment.
• It must support forking (needs something like a restart function).
• It must/should support streaming exporters.

Clarify ABI requirements for the exporter interface

This is intended for continuing the discussion we had in the meeting on March 2nd.

The SDK user must be able to register span processors on the TraceProvider. Several span processors (and thus several exporters) can be registered at the same time. The SDK must allow users to implement and configure custom processors (and, obviously exporters).

Usually, exporters and other user defined objects will be located in a separate binary. Those binaries could either be linked statically with the SDK or dynamically loaded into the same process as the SDK.

For dynamically loaded exporter binaries, we have to decide how to define ABI requirements, similar to the API part of OpenTelemetry C++.

The main possibilities I see:

No ABI requirements are defined

In this case, the user has to take care that the SDK and the exporter binary are compatible. When using the standard SDK provided by OpenTelemetry C++, the user always has the possibility to compile the SDK in a way that is compatible with the exporter binary.

The same ABI requirements as for the API apply

This would restrict us in defining and changing the interface. However, in that scenario users would have the possibility to combine pre-compiled SDKs and exporter binaries that were compiled with different STL versions.

For reference, here's the related section in the spec, and here's a sketch of a C++ span exporter interface.

Document the supported compiler and platform version

Brought up by @maxgolov, something similar to this should exist in the README.md file:

Operating System Compiler
Mac OS X 10.11.6 Clang Xcode 8.3
Mac OS X 10.12.6 Clang Xcode 9.0, 9.1
Mac OS X 10.13.3 Clang Xcode 9.2, 9.3, 10.0, 10.1
Raspbian GNU/Linux 8 (jessie) GCC 4.9.2 (armv7l)
Ubuntu 14.04.* LTS GCC 4.8.*, 4.9.4
Ubuntu 14.04.1 LTS GCC 5.x.x
Ubuntu 16.04 LTS GCC 5.x.x (armv7l)
Windows 10 Android Studio/Gradle
Windows Server 2016 Visual Studio 2017

Implement Zipkin exporter

Refer to https://github.com/open-telemetry/opentelemetry-python/tree/master/ext/opentelemetry-ext-zipkin.

Implement metrics API

cmake.c++20.test fails in the Docker container in both bazel versions 3.2.0 and 3.3.0.

Is your feature request related to a problem?
Running the CI tests through GitHub actions results in all tests passing successfully. However, running the CI tests manually in the Docker container through terminal results in multiple CMake errors throughout all CMakeLists.txt in the repository caused by the cmake.c++20.test. The error produced reads "CXX_STANDARD is set to invalid value '20'". This error happens in both bazel version 3.2.0 and bazel version 3.3.0.

Describe the solution you'd like
The readme should inform readers on how to run the CI tests on GitHub Actions, instead of informing readers on how to run the CI tests manually through the Docker container.

Describe alternatives you've considered
Alternatively, a note can be put into the readme that describes the issue with running the cmake.c++20.test manually, and how it passes when run through GitHub Actions.

Choose C++ style guideline

C++ is a big language that can be used in many different ways. We need to choose a style and stick to it.

A couple popular starting points (can be chosen as is or as the starting point for our own style):

• https://google.github.io/styleguide/cppguide.html
• http://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines

Other style suggestions are welcome.

Implement Jaeger exporter

Implement span exporter for Jaeger.
Refer to https://github.com/open-telemetry/opentelemetry-python/tree/master/ext/opentelemetry-ext-jaeger.

Metrics Label Storage Convention

Is your feature request related to a problem?

Currently, different languages use varying data structures to store Metric labels. Java, for example, uses a string while python uses a dictionary (roughly equivalent to the unordered map in C++). These decisions were made based on the strengths of each language and the performance/availability of data structures in that particular language.

Describe the solution you'd like

It would be helpful to developers working on metrics and exporters if the community weighed the different approaches then issued explicit guidance on this topic as it would prevent the use of clashing data types when integrating parts of the library.

Describe alternatives you've considered

By using a string to hold labels, we can improve the user experience in some ways since those adding metrics to their code will not have to create a cumbersome map (or other key value data structure) each time they want to add labels to their metrics. Additionally, this improves readability of code as it is easier to see the labels in a string rather than a map, especially in C++. Finally, the performance of strings is likely superior as they are a lightweight type which can be compared relatively easily. Checking the contents of a map for equality is more difficult.

A map is beneficial because it doesn’t enforce an ordering on the labels. For example, the label sets “A:1, B:1” and “B:1, A:1” are equal in terms of content and a map would reflect that while a string would treat these as unique sets. Depending on preference, this could be seen as an advantage or disadvantage. Also, it may be easier to modify label sets using a map as users can add and remove certain attributes as they wish without complicated string parsing.

Badges and Licensing on the Repo Readme

Is your feature request related to a problem?
In an effort to have the OpenTelemetry C++ SDK be as up-to-date and consistent with the other more established SDKs, (Python, Java) there should be some sort of status tracking on the main Readme.md file, so that any observer of the repo will be able to quickly and efficiently know of the progress, reliability, and licensing of the codebase.

Describe the solution you'd like
We could simply make use of the open source repo-badges (https://github.com/dwyl/repo-badges), version badges (https://github.com/badges/shields) and license badges (https://gist.github.com/lukas-h/2a5d00690736b4c3a7ba) and add them to the current readme, so that the documentation would look similar to its sister repositories: (screenshot of the Python Readme badges for reference)

Add CI workflow using GitHub Actions

Is your feature request related to a problem?

Currently the OpenTelemetry C++ client (https://github.com/open-telemetry/opentelemetry-cpp) uses CircleCI and a Docker container to automatically build and test the project when pull requests are issued against the master branch of the main repository. The goal of continuous integration is to quickly build and run tests to provide testing feedback and information in a transparent, concise way. CircleCI fails in this regard as it obscures test output and even prevents contributors from running certain tests if they are using a “Free plan” account.

Proposed solution I'd like to see

Using a GitHub actions workflow to run CI pipelines to provide more testing transparency, more control, and an easy-to-use pipeline compared to CircleCI. Additionally, more workflows could be added to GitHub actions to provide services such as code coverage, automatic deployment, and seamless integration with other GitHub services.

Transparency

CircleCI allows anyone to view the console output when building and running tests but the results are not shown anywhere on the GitHub repository. To see testing output: You must go to a different website, navigate a different user interface, then sift through thousands of lines of console output.

In contrast, using GitHub actions would mean that all testing output is located directly on the repository’s GitHub page, making it easier for contributors to find and read.

Control

GitHub Actions’ integration with other GitHub features means you can have finer control over the CI pipeline. For example, certain workflows can be set to only run on a new release. Workflows can even be used to close stale issues and pull requests.

Ease-of-Use

The current CI pipeline using CircleCI will automatically run when pull requests and commits are issued against the main repository. But if a contributor forks the repository, unless they set up a CircleCI account and link it to their repo, CI will not be activated and tests must be run manually by the contributor.

GitHub Actions works out of the box on a forked repository and can be easily configured to run a test workflow each time a commit is issued. This would help individual contributors test their code and ensure high quality before submitting a pull request against the main repository.

Alternatives considered

Currently the OpenTelemetry C++ client (https://github.com/open-telemetry/opentelemetry-cpp) uses CircleCI and a Docker container to automatically build and test the project when pull requests are issued against the master branch of the main repository. However, for the reasons stated above using GitHub Actions workflows for CI would provide a better user experience for all developers contributing to the code base.

Build gRPC using CMake

Context
The OTLP exporter uses gRPC to communicate with the OpenTelemetry Collector. Currently we are building gRPC with Bazel, but not with CMake.

Problem
The OTLP exporter tests run on Windows using CMake only.

Solution
Build gRPC using CMake (see gRPC documentation). Add OTLP exporter tests to exporters/otlp/CMakeLists.txt.

Add sampling API

Add sampling API which follows https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/api-sampling.md.

Update CONTRIBUTING.md with internal examples

In PR 97 I added a Useful Resources section in CONTRIBUTING.md that generally point outwards to other Open Telemetry projects and their examples. This Issue will serve as a reminder to update these resources as examples and exporters are added to the C++ repository.

More Issues on Intermediate Storage of Prometheus Exporter Design

Is your feature request related to a problem?
Yes. In order to guarantee the current Prometheus Exporter design works, we need to figure out the following problems about the intermediate storage:

• What if the storage is full when export() function is called? Is the new batch of data discarded or will some data in the storage be replaced? What is the replacement strategy?
• What is the default collection capacity? How to determine it? Are there alternatives you are suggesting?
• Is capacity (size of the storage) configurable (such as an environment parameter) ?
• What should be the replacement strategy when the storage is full? LRU? LFU? FIFO?
• For the fake dump, where should we dump the data? Shall we discard the data directly?

Describe the solution you'd like
We don’t have solution for these problems yet, but we prefer the following mechanism:

• If the free space is not enough for adding the coming batch of data, we are going to discard the coming data. We think the older metrics data are more important.
• We think it’s better to make the storage size configurable, but we have no idea about the default size.
• In this solution, we are not going to replace any data in the storage, so we don’t consider which replacement strategy to use.

Describe alternatives you've considered
There are two alternatives:

1. Still use memory bound, and replace old data if the storage is full when new data comes.
   1. The default storage capacity is still a pending problem.
   2. If we are going to replace data, we will also replace from tail of the storage. They are newer metrics data and we think newer data are less important then older data.
2. Use a time boundary instead.
   1. We were suggested to use a time boundary initially, which means that we set a timeout, and use a timer to keep track of the time elapsed after last pull. After the pre-defined time threshold, do a fake pull to flush data. When the user pull data successfully, set the timer to zero can keep track of the time again. We don’t think this solution is better than a memory boundary for the following problems:
      1. Where would the fake flush dumps data to? Does the fake flush discard this data?
      2. From the Prometheus side, is it possible that metrics data would be found missing (due to the fake flush)?
      3. What are the standards to choose a reasonable timeout?
      4. We still need to use some data structure to hold the data, so the size of the collection is still limited. What if the collection is full before a fake flush?
3. We are leaning towards using a memory boundary (limiting the storage size).

Additional context
The same context with issue #158

Add context API and implementation

Add context API which follows https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/context/context.md. Related to #74.

References:

• OpenTelemetry Python Implementation

Add Prometheus exporter

Add Prometheus exporter to prove metrics SDK. Consider adding console exporter for metrics.

Implement OTLP exporters

Implement OTLP trace exporter and metrics exporters.

Refer to https://github.com/open-telemetry/opentelemetry-python/tree/master/ext/opentelemetry-ext-otcollector.

Add a badge for CI status

Description:

Is your feature request related to a problem?
Most of the existing open-telemetry repositories (java/javascript/python/collector/go/ruby/rust) display a badge for CI status. It would be very useful to have a CI badge for the CPP repo too.

Describe the solution you'd like.
As a developer contributing to open-telemetry, I recommend to add a CI status badge on the README.md of C++ repo. Because CI can increases the speed of development while ensuring the quality of the code that teams deploy.
And by adding a badge to README.md, a developer will be able to know CI status easily by access CI badge.

Describe alternatives you've considered.
As seen in the Rust repo, we may provide a CI status badge which links to “Actions” page instead of linking to CircleCI website as well.

Investigate Boost license implications

A portion of nostd::variant currently requires a Boost-licensed header.
This may be an issue for some commercial products.

Two alternatives to consider:

• research if we can get that portion code contributed under Open Telemetry (Apache-style) license. That way we still satisfy the ABI stability guarantees.

• the other alternative is to compile with std #67 . That approach means that commercial products building from source with standard library cannot safely interop with other prebuilt Open Telemetry modules.

atomic_shared_ptr.h doesn't compile with gcc-9.3 / C++20

Implementation suggests that it's C++20 compatible here:

opentelemetry-cpp/sdk/include/opentelemetry/sdk/common/atomic_shared_ptr.h

Line 14 in 9e5924c

* This wrapper relies on std::atomic for C++20, a mutex for gcc 4.8, and

However, if the project is built with

set(CMAKE_CXX_STANDARD 20)

Then the build is broken here:

opentelemetry-cpp/sdk/include/opentelemetry/sdk/trace/tracer.h

Line 47 in 9e5924c

opentelemetry::sdk::AtomicSharedPtr<SpanProcessor> processor_;

With the following error:

[ 50%] Building CXX object sdk/src/trace/CMakeFiles/opentelemetry_trace.dir/tracer_provider.cc.o
In file included from /mnt/c/work/opentelemetry-cpp/sdk/include/opentelemetry/sdk/common/atomic_shared_ptr.h:3,
                 from /mnt/c/work/opentelemetry-cpp/sdk/include/opentelemetry/sdk/trace/tracer.h:3,
                 from /mnt/c/work/opentelemetry-cpp/sdk/include/opentelemetry/sdk/trace/tracer_provider.h:9,
                 from /mnt/c/work/opentelemetry-cpp/sdk/src/trace/tracer_provider.cc:1:
/usr/include/c++/9/atomic: In instantiation of ‘struct std::atomic<std::shared_ptr<opentelemetry::v0::sdk::trace::SpanProcessor> >’:
/mnt/c/work/opentelemetry-cpp/sdk/include/opentelemetry/sdk/common/atomic_shared_ptr.h:33:35:   required from ‘class opentelemetry::v0::sdk::AtomicSharedPtr<opentelemetry::v0::sdk::trace::SpanProcessor>’
/mnt/c/work/opentelemetry-cpp/sdk/include/opentelemetry/sdk/trace/tracer.h:47:54:   required from here
/usr/include/c++/9/atomic:191:21: error: static assertion failed: std::atomic requires a trivially copyable type
  191 |       static_assert(__is_trivially_copyable(_Tp),

Clarify ABI requirements

Building on top of #24, I added an example of two libraries that are built with OpenTelemetry API and different STLs (libc++ versus libstdc++): g-easy@5dc5f2e

Using them both appears to result in a link error due to the OpenTelemetry API using std::shared_ptr.

Please help me understand whether I'm doing something wrong, or whether we need to change the requirements around STL use.

Does not build at head

Describe your environment

bazel --version
bazel 3.0.0

Steps to reproduce

git clone https://github.com/census-instrumentation/opencensus-cpp.git
cd opencensus-cpp
bazel build //...

What is the expected behavior?
Things build.

What is the actual behavior?

ERROR: /home/simon/opencensus-cpp/examples/http/BUILD:21:1: every rule of type cc_binary implicitly depends upon the target '//tools/cpp:toolchain', but this target could not be found because of: no such package 'tools/cpp': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
 - /home/simon/opencensus-cpp/tools/cpp
ERROR: Analysis of target '//examples/http:server' failed; build aborted: no such package 'tools/cpp': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
 - /home/simon/opencensus-cpp/tools/cpp
INFO: Elapsed time: 0.107s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)

HTTP client API for exporter developers

Different OS may have alternate implementations of HTTP stack:

• Windows: WinInet, Win 10 (WinRT) HTTP client, WinHttp
• Linux: libcurl, poco, etc.
• Mac OS X: libcurl, poco, Apple SDK URLRequest
• Android: Java HttpRequest, Google HTTP client, OkHTTP
• iOS: Apple SDK URLRequest
  etc.
• Chromium HTTP client

An exporter that supports HTTP requires a minimum set of methods:

• create an instance of HTTP client
• the client must have ability to create a request with a set of custom headers, method and request body
• it must provide async execution, with a callback that allows to obtain HTTP response state/result, status code, response headers and response body
• the callback should, as a minimum, support ability to inspect individual HTTP state machine events (SSL handshake, attempt to connect, sending request, receiving response, etc.), as well as ability to receive the response body / response status code.

Const-correctness aside, common interface structure could define how an exporter developer may design and implement their own HTTP client.

HTTP client does not have intrinsic knowledge of the binary serialization protocol itself (be that gRPC, protobuf, JSON, MSFT Bond, etc.), but can be invoked by exporter implementation to trigger the data upload. The callback associated with each async HTTP request may further handling the retry policy in case if data collection end-point returns something else than 200 OK.

Basic set of interfaces needed to implement the HTTP client:

    class IHttpClient
    {
    public:
        IHttpRequest* CreateRequest();
        void SendRequestAsync(IHttpRequest* request, IHttpResponseCallback* callback);
        void CancelRequestAsync(std::string const& id);
        void CancelAllRequests();
    };

    class IHttpRequest
    {
    public:
        std::string& GetId();
        void SetMethod(std::string const& method);
        void SetUrl(std::string const& url);
        HttpHeaders& GetHeaders();
        void SetBody(std::vector<uint8_t>& body);
        std::vector<uint8_t>& GetBody();
    };

    class HttpHeaders : public std::multimap<std::string, std::string>
    {
    public:
	void set(std::string const& name, std::string const& value);
	void add(std::string const& name, std::string const& value);
	std::string const& get(std::string const& name);
	bool has(std::string const& name);
    }

    class IHttpResponse
    {
    public:
        std::string& GetId();
        HttpResult GetResult();
        unsigned GetStatusCode();
        HttpHeaders& GetHeaders();
        std::vector<uint8_t>& GetBody();
    };

    class IHttpResponseCallback
    {
    public:
        void OnHttpResponse(IHttpResponse* response);
        void OnHttpStateEvent(HttpStateEvent state, ...);
    };

W3C TraceContext compliance

Implement W3C TraceContext and get 100% pass rate from the validation service https://github.com/w3c/trace-context/tree/master/test.

CI should be implemented which verifies the TraceContext compliance.

Use of exceptions in API should be revisited

There's an issue with API headers related to std::exception and exception handling. For a library that is built with exceptions enabled - there's been a breaking change between vs2017 and vs2019 on Windows, related to size optimizations in __CxxFrameHandler4 ...

In short: despite the effort invested in nostd - we cannot provide ABI stability guarantee that a static library with dynamic runtime (/MD) that uses exceptions, will be compatible with a product built with a different (older) compiler on Windows.

Related issues:
microsoft/dotnet#1121
microsoft/vcpkg#11619
microsoft/vcpkg#10617
reactos/reactos#1995

There are several options to solve this:

• Entirely avoid exceptions in any "standard" prebuilt implementation of OT, especially in API headers.
• Ensure that we use /d2FH4- flag if compiling on vs2019, to provide backwards-compat with vs2017.

Current violations in API headers that will break ABI compatibility across default build flag settings for vs2017 vs vs2019, rendering nostd ABI-incompatible if exceptions are being used:

• nostd/span.h
• nostd/string_view.h - OK, it has the #if __EXCEPTIONS check (causing the process to terminate if out-of-range params get passed to it).
• nostd/variant.h - uses exceptions
• plugin/detail/utility.h - OK, it has the #if __EXCEPTIONS check.

This issue presents a practical challenge to achieve ABI compatibility between libraries built with vs2017 vs 2019. Despite the fact that nostd API surface is ABI-stable - there are additional requirements that must be met.

Question on Use of Virtual Functions

When writing our design for the metrics API and SDK, our team working on this implementation (@Brandon-Kimberly , @ankit-bhargava , and @HudsonHumphries ) were confused about when to make a function virtual and when to make it purely virtual. To me, it makes sense to make the function purely virtual if it is crucial to the purpose of the class that it is in but I am very interested to hear others’ thoughts.

What was decided for the tracing portion of the library?

cc @reyang @alolita

Add labels to issues

Is your feature request related to a problem?
Yes. Currently there are very few labels on the issues open in this repository. This makes it very hard to navigate the issues, especially for a newcomer to the project.

Describe the solution you'd like
For the current issues the Maintainers would need to go through the open issues and label them. For future issues,
whenever an issue is opened, descriptive labels should be added to them.

Describe alternatives you've considered
A great way to assist the Maintainers in adding labels is have contributors add their own labels when creating an issue. This would make the work of adding labels extremely easy for the Maintainers. When creating an issue there could be a reminder to add labels to it.

Additional context
Many other Open Telemetry projects have many labels on there issues, here are a few:
JavaScript
.NET
Collector

Beta readiness tracker

This is a tracking issue for the requirements outlined in the beta release document: https://docs.google.com/document/d/1eviggoIguOS89dgKL-8ntUOCfPP6FLWAoQkGoBDsld0/edit#

Traces

• Stable API of every spec'd feature
• SDK Implementation of every spec'd feature
  • Basic tracing
  • Named tracers
  • Tracers use OTEP 66 Context mechanism - This is an implementation detail and does not affect the API

Resources

• Stable API of every spec'd feature
• SDK Implementation of every spec'd feature

Context Propagation

• Stable API of every spec'd feature
  • Basic context mechanism
  • Global Propagators API
  • Global Context API (get and set active context)
• SDK Implementation of every spec'd feature
  • Basic context mechanism
  • Global Propagators

Exporters

• OpenTelemetry Native - Collector Exporter
• Prometheus
• Jaeger ( stretch goal)
• Zipkin

Examples

• Manual tracing hello world example
• HTTP client/server using an instrumented framework
• gRPC client/server

Implement OpenTracing shim

Refer to https://github.com/open-telemetry/opentelemetry-python/tree/v1.10.0/shim/opentelemetry-opentracing-shim.

Add Doxygen for C++ OpenTelemetry API/SDK documentation

Is your feature request related to a problem?
Yes. Currently the C++ repository contains little internal documentation due to ongoing development. This can make it difficult for prospective contributors to get started with the project.

Describe the solution you'd like
The incorporation of, and requirement to use, Doxygen will help contributors constructively create documentation as features are built out. This makes it easier to build complete and comprehensive documentation.

Describe alternatives you've considered
I explored two other alternatives to using Doxygen:

1. Using a different documentation generator. Other docs generators exist for C++ but none have quite the popularity or accessibility of Doxygen. Doxygen is open-source, works on all major OS platforms, is compatible with a variety of languages, and is the de facto standard for C++ documentation generation. Contributors will likely be more comfortable and more experienced with using Doxygen compared to another documentation generators.
2. Not using a documentation generator. This approach is sub-optimal because it requires contributors to retroactively create documentation for built functionality. This documentation may never be created and is therefore not helpful to newcomers. Additionally, Doxygen can help to produce comprehensive, low-level documentation on class structure and class functions. This type of documentation is time-consuming and tedious to manually create.

Additional context
Other OpenTelemetry repositories have included docs generator tools into their code (such as Java with Javadoc).

Implement Baggage ( prev: CorrelationContext )

https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/correlationcontext/api.md

Implement Cloud Trace exporter

Refer to https://github.com/open-telemetry/opentelemetry-python/tree/master/ext/opentelemetry-exporter-cloud-trace

Data structure usage for intermediate storage in Prometheus Exporter

Is your feature request related to a problem?
In a proposed Prometheus Exporter in C++, intermediate storage between exporting data and collecting data must be implemented due to its nature as a pull based exporter. However, it is unclear whether or not containers from STL such as vectors can be used or not, though usage of STL seems not to be limited when building an exporter.

Describe the solution you'd like
If STL is available, a vector would be used as the container for intermediate storage in the Prometheus Exporter. Vectors have the added benefit of being contiguous in memory, allowing for deep copies to be made easily using a function like memcpy(). This is important for concurrency, which needs to be guaranteed when collecting data.

Describe alternatives you've considered
Alternatively, if STL is not available, then we could manage an array internally as our intermediate storage container. This would be more prone to various errors, but if implemented correctly, should function the same as a vector.

Additional context
In order to fit in the Metrics API data pipeline, we decided to add an intermediate storage in our Prometheus Exporter design. The controller calls the export() function and sends out the data. It is natural for push based exporters, because then the data can be pushed directly to the target system in the export() function. For Prometheus, it's different because Prometheus uses a pull based exporter. We need to wait for a pull request from the Prometheus server before serving the data, but we don't know when the request will come, so we need some place to store and hold the data.

Implement a stdout exporter

Implement an exporter that just writes all span data it receives to stdout (or, ideally, to any other configurable file).

A stub for such an exporter is available in the simple example. The stub in the example should be replaced.

function_ref.h implementation uses a feature that's removed in C++20

This code would no longer compile in strict mode with C++20 enabled in Visual Studio 2019 (16.6.2, tools v142):

opentelemetry-cpp/api/include/opentelemetry/nostd/function_ref.h

Line 66 in 9e5924c

std::is_convertible<typename std::result_of<F &(Args...)>::type, R>::value,

std::result_of has been permanently removed in C++20. Reference:
https://en.cppreference.com/w/cpp/types/result_of

Implementation needs to be rewritten.

One possible approach is to backport std::result_of into nostd::result_of

The other implementation approach is to rework this code to completely avoid the construct.

Visual Studio 2019 compiler takes very strict stance on removed features. There's also a temporary workaround flag that allows to enable deprecated features ( _HAS_DEPRECATED_RESULT_OF ). But I believe it is incorrect for common SDK API header to rely on a feature that's been permanently removed from C++ standard.

I also tested this to confirm that recent gcc-9 and clang-10 are both OK with that deprecated construct, even if a module is compiled in C++20 mode.. But what Visual C++ is doing (not supporting removed features) - seems like the right thing to do.

Add forking support to the exporters and processors

Exporters and processors should be fork safe. In case of a fork,

• no data should be duplicated, and
• vendor implementers must have the possibility to re-initialize open connections (sockets) or files.

This could be implemented via restart functions in span processors and exporters.

Add Codecov Badge to README.md

Is your feature request related to a problem?
OpenTelemetry’s Python, JavaScript, Collector, Operator, Go, Java and PHP repository all have a Codecov badge which provide a quick link to details about code coverage. The C++ SDK repository does not have a Codecov badge yet.

Describe the solution you'd like
As a developer contributing to the C++ SDK, I’d like to add a Codecov badge. In general, badges help in increasing the readability of the README.md file because they provide key metrics about the health of the repositories.
For the Codecov badge, developers can learn about the effectiveness of the tests and the overall quality of the project by quickly scanning the attached badge.

Describe alternatives you've considered
Codecov badge can be acquired from codecov.io and added easily to README.md. We should also consider adding other badges for build status, license, and versions.

Make a Quick-Start Guide

Is your feature request related to a problem?

New contributors currently have little guidance on what the library is or how to use it. As of now, the only way to learn about basic instrumentation and collection is to either carefully examine the code or work over the single tracer example. This, however, covers a very narrow set of features. Users must reference other language documentation for a comprehensive overview of OpenTelemetry features with example usage.

Describe the solution you'd like

The Java quick-start is a model example which marries explanations of fundamental OpenTelemetry concepts with short, digestible code blocks. After reading over this, users have enough information to get immediate results from the OpenTelemetry library. The quick-start also teaches users what their options are and builds a solid enough foundation for them to look over more technical documentation on their own.

Describe alternatives you've considered

Without a quick-start, users can look over documentation and examples independently. It takes longer to obtain useful results from the library and its much harder to reconcile the specification requirements with their concrete applications. Users also do not have a single location to refer back to when integrating the library into their workflows which can be frustrating and time consuming.

The location of Doxygen configuration file

Try to solve issue #109

Is your feature request related to a problem?
Yes. Currently the C++ repository contains little internal documentation, and we are trying to integrate Doxygen into the repository so that we can easily generate and publish technical documentation.

Describe the solution you'd like
In the repository there is already a folder called docs, however it looks like a collection of documents about project introduction, project setup, and policies, not for technical documentation. I prefer to create a new folder called doxygen to generate the configuration file and technical documentation.

Describe alternatives you've considered
Put doxygen file in the docs folder as well, so that all kinds of document are in the same directory.

Additional context
In Prometheus C++ client repository, it puts the doxygen configuration file in a doc folder.

Streaming SDK proposal

Hello Everyone,

I work on the OpenTracing C++ specification and the instrumentations for NGINX and Envoy. Recently, I've been working with LightStep to improve the performance of their C++ tracer. Our efforts to reduce instrumentation cost and improve collection throughput led us to a more efficient and streamlined design that I propose we adopt as one of the SDKs for OpenTelemetry. The key components of the design are 1) we remove intermediate storage on span objects and instead serialize eagerly as methods are called and 2) we use a domain-specific load balancing algorithm built upon non-blocking sockets, vectored-io, and io multiplexing.

Note: This only discusses the design as it releates to tracing. I plan to updated it to include metrics as well as the proposal progresses.

Design

At a high level, the design looks like this

1. To each span we associate a chain of buffers that represent the span's protobuf serialization and we build up the serialization as methods on the span are called.
2. When a span is finished, its serialization chain gets moved into a lock-free multiple-producer, single-consumer circular buffer (or discarded if the buffer is full).
3. A separate recorder thread regularly flushes the serializations in the circular buffer and streams them to multiple endpoints using the HTTP/1.1 chunked encoding (one chunk per span) with domain-specific load balancing that works as follows:
   1. When the recorder flushes, it picks an available endpoint connection and makes a non-blocking vectored-io write system call to the connection's socket with the entire contents of the circular buffer.
   2. If the amount of data written is less than the size of the circular buffer, the recorder binds any remaining partially written span to the connection (to be written later when its socket is available for writing again) and repeates (i) with one of the other available connections.
   3. If no connections are available for writing, it blocks with epoll (or the target platform's equivalent) until one of the connections is available for writing again.

Here's a diagram of the architecture.

LightStep's C++ tracer provides an example implementation of this design. These are the main components

1. The serialization chain for a span's protobuf serialization.
2. The span class with no intermediate data storage and eager serialization
3. The lock-free circular buffer for buffering spans in the recorder.
4. The streaming recorder with domain-specific load balancing

Rational

Serializing eagerly in (1) instead of storing data in intermediary structures that get serialized later when the span is finished, eliminates unnecessary copying and allows us to avoid small heap allocations. This leads to much lower instrumentation cost. As part of my work on the LightStep tracer, I developed microbenchmarks that compare the eager serialization approach to a more traditional approach that stores data in protobuf-generated classes and serializes later when traces are uploaded. For a span with 10 small key-values attached, I got these measurements for the cost of starting then finishing a span that show much better performance for the eager serialization approach:

-----------------------------------------------------------------------------
Benchmark                                      Time           CPU Iterations
-----------------------------------------------------------------------------
BM_SpanSetTag10/late-serialization           5979 ns       4673 ns     148715
BM_SpanSetTag10/eager-serialization           845 ns        845 ns     833609

Using a lock-free circular buffer in (2) allows the tracer to remain performant in high concurrency scenarios. We've found that mutex-protected span buffering causes signficant contention when multiple threads finish spans concurrently. I benchmarked the creation of 4000 spans partioned evenly across a varying number of threads and got these results for a mutex-protected buffer:

-----------------------------------------------------------------------------
Benchmark                                      Time           CPU Iterations
-----------------------------------------------------------------------------
BM_SpanCreationThreaded/mutex/1 thread   4714869 ns      57876 ns       1000
BM_SpanCreationThreaded/mutex/2 threads  7351604 ns     104345 ns       1000
BM_SpanCreationThreaded/mutex/4 threads  8019629 ns     185328 ns       1000
BM_SpanCreationThreaded/mutex/8 threads  8582282 ns     328126 ns       1000

The mutex-protected buffer shows slower performance when we use more than a single thread. By comparison, lock-free buffering doesn't have any such degradation

-----------------------------------------------------------------------------
Benchmark                                      Time           CPU Iterations
-----------------------------------------------------------------------------
BM_SpanCreationThreaded/stream/1 thread  1120340 ns      55628 ns      11704
BM_SpanCreationThreaded/stream/2 threads 1375883 ns     111743 ns       7542
BM_SpanCreationThreaded/stream/4 threads 1672191 ns     182614 ns       3970
BM_SpanCreationThreaded/stream/8 threads 1007622 ns     265792 ns       2606

By using multiple load-balanced endpoint connections, the transport in (3) allows for spans to be uploaded at a high rate without dropping data. The domain-specific load balancer takes advantage of the property that spans can be routed to any collection endpoint and naturally adapts to back-pressure from the endpoints to route data to where its most capable of being received. Consider what happens as a collection endpoint starts to reach its capacity to process spans:

1. At the network level, the endpoint ACKs the TCP packets of span data a slower rate.
2. From the recorder side, the vectored-write system calls for that endpoint will send less data and the socket will block.
3. The recorder will then naturally send spans to other endpoints until epoll reports that the overloaded endpoint's socket is no longer blocked and it's capable of receiving data.

Dependencies

A goal of the SDK is to have as minimal a set of dependencies as possible. Because the eager serialization approach uses manual serialization code instead of the protobuf-generated classes, we can avoid requiring protobuf as a dependency. Lightstep's current implementation uses libevent and c-ares for portable asynchronous networking and dns resolution -- but those parts of the code could be hid behind an interface in a way that would allow alternative libraries to be used or platform-specific implementations.

Customization Points

Because this approach never generates a SpanData-like structure with accessors, the customization points are different than the traditional exporter approach. The main point of customization point would be the serialization functions where a vendor could provide alternative implementations to write to a different wire format.

Next steps

The LightStep implementation could be adopted into a default OpenTelemetry tracer and SDK that uses the opentelemetry-proto format but provides a customization point for the serialization. This SDK would prioritize efficiency and high-throughput. While it might not be the right choice for all use cases, I would expect the OpenTelemetry C++ API to be flexible enough to support a variety of different impelmentations, so other use cases could be serviced by either an alternative impelmentation of the OpenTelemetry API or a different SDK (if we decide to support more than one).

Ability to use Standard Library instead of opentelemetry nostd classes

Due to ABI stability requirements OpenTelemetry C++ SDK provides the standard implementation of 'common' std-alike classes under nostd namespace:

• function ref
• shared_ptr
• span
• string_view
• unique_ptr
• variant

However, in some environments it may be of benefit to alias the nostd types to standard library, esp. if compiler has full support of C++17. And if the environment the SDK is being built for has no need to consume any external prebuilt exporters.

One way to achieve this is a custom compile-time build option (cmake flag) that aliases nostd with std namespace. Tests could be added to verify that the SDK still works well with recent version of Visual Studio, Apple llvm-clang, gcc, etc. (in essence, replacing the nostd implementations with std).

Update CI Tests to Allow SDK Exceptions

Is your feature request related to a problem?
During Monday's weekly meeting it was determined that, according to the spec, the SDK should be allowed to throw exceptions (i.e. when constructing samplers with invalid arguments). However, there is currently a CI test in place that is too strict and does not allow exceptions to thrown in any situation. This test needs to be updated to be more fine grained to prevent API functions from throwing across ABI boundaries, but to allow exceptions in situations like this.

Describe the solution you'd like
The CI tests should allow throwing exceptions in cases where they doesn't cross ABI boundaries, while still failing for exceptions that do.

Additional context
Current test: bazel build --copt=-fno-exceptions $BAZEL_OPTIONS //...

Implement a batching span processor

In addition to the simple span processor, a batching span processor needs to be implemented.

The batching span processor must conform to spec requirements.

For buffering spans, the circular buffer introduced in #52 should be used. For timer functionality, the dispatcher introduced in #63 should be used.