lightstep / lightstep-tracer-cpp Goto Github PK
View Code? Open in Web Editor NEWThe Lightstep distributed tracing library for C++
Home Page: https://lightstep.com
License: MIT License
The Lightstep distributed tracing library for C++
Home Page: https://lightstep.com
License: MIT License
As this comment notes HTTP header names are case insensitive. When using a HTTPHeadersReader
, baggage keys are "detected" case-insensitively, but the case of the header, whatever it happened to be, is preserved when the baggage is stored in the map. Depending on what your http library does to header names, this means you can't use BaggageItem
reliably. For instance if I say
SetBaggageItem("somekey", "somevalue")
in a Go service, when I extract that in a C++ service I need to say BaggageItem("Somekey")
because the http
code in Go is going to turn the header info Ot-Baggage-Somekey
.
I don't think this is the intended behavior of baggage to require folks to not use header names to carry the baggage key name (since it is impossible to preserve the baggage key name case in that case). I know that I can use ForeachBaggageItem
for this scenario with my own case insensitive comparison, but again, it seems against the "spirit" of providing BaggageItem
at all then.
The C++ streaming recorder seems to "never" stop making requests, even if there are no spans to send. This is mostly harmless I think, and it makes sense to optimize for the assumptions of sending spans as opposed to not.
At shutdown of a service though, the fact there is always a http request in flight to the satellites (whether there are spans to send or not) means the Tracer shutdown is almost never clean. It seems (based on my reading of the code) that we Flush, sending spans if there are any, and then the socket is closed before the response of the satellite arrives.
It may also be possible that some of the time response arrives, reaches the code in OnReadable
and then reconnects and then that request is aborted abruptly.
I appreciate we do not want to block the process indefinitely on tracer shutdown. This presents a problem for monitoring some middleware or proxies as there are many legitimate reasons to want track client disconnections. For example nginx logs these as the non-standard http status code 499. This can be helpful for identifying mismatched timeouts between client and server or services that aren't performing to the expectations or clients or other issues between the client and server.
Since a "normal" tracer exit will close the connection before receiving a response (and again this is happening even if the tracer is quiescent because there is almost always a request in flight, even if it is empty of spans), this can create a lot of noise in the metrics and logs for this scenario (client closing the connection before the response is sent) that may mask "real" client side timeouts that need to be investigated and resolved.
Not also this isn't just a matter of "well if your network/middleware/etc is fast enough the response will be sent back quickly and you won't see this", I have replicated this with a pretty simple "fake satellite" and the example stream
program running on locahost: Example.
It seems like some variant of Flush that would set a flag in OnReadable
to close the connection (just FreeSocket
instead of Reconnect
) could let folks opt-into a clean exit. The graceful shutdown timer could still apply, so even for those folks it wouldn't be an indefinite shutdown, it just wouldn't be a "timeout of 0" like it seems to be today. Another option could be to not initiate new connections if there were no spans to send for some period of time. Again this would a tunable time period and folks could just stop generating spans (likely to happen on the shutdown path anyway). I have not tried to implement either of these, and there may be a nuance or nuances that I am missing.
Filing this here to document a change request for Envoy.
LightStep engineering has observed small reports from new Envoy installations at several customers. These reports are created in the LightStep C++ tracer library, although it is an Envoy configuration default which controls this setting.:
The LightStep C++ tracer itself uses a default of 2000 for the maximum buffer size. As Envoy is using a per-CPU tracer, we recommend a default of 2000/std::thread::hardware_concurrency()
if the tracing.lightstep.min_flush_spans
variable is not set.
I tried to send the log using the sample code
auto tracer = lightstep::MakeLightStepTracer(std::move(options));
opentracing::Tracer::InitGlobal(tracer);
auto span = opentracing::Tracer::Global()->StartSpan("demo");
span->SetTag("key", "value");
span->Log({{"logkey", "logval"},
{"number", 1}});
span->Finish();
opentracing::Tracer::Global()->Close();
Getting following error :
Error: Report RPC failed: Socket closed
Support for use of tags in tracer configuration [1] for lightstep tracer C++ library that nginx ingress controller using dynamic loading.
As we are dynamically loading the module, I think the options are limited to the JSON configuration that this schema has and this is what is lacking support for tags . I do see tags being in the cpp library.
This would allow us to add a version tag and hostname tag that is listed under instrumentation quality for our nginx ingress service [2]
Links:
[1] https://github.com/lightstep/lightstep-tracer-cpp/blob/master/lightstep-tracer-configuration/tracer_configuration.schema.json
[2] https://app.lightstep.com/UnderArmour_Production/service-directory/nutrition-ingress-external/instrumentation-quality
Files generated by protoc doesn't compatible with the newest protobuf.
Can the build script be changed to generate the protobuf artifacts at build time instead of having them check in as source code?
As outlined in this Slack chat on envoy-dev, when lightstep-tracer-cpp is built within Envoy, it does not respect the CC
and CXX
environment variables. When building on a system where CC=clang
and CXX=clang++
, the build defaults back to gcc
and gcc++
.
I did some testing and it looks like this may be resolved by passing -DCMAKE_C_COMPILER
and -DCMAKE_CXX_COMPILER
to the cmake
command. I couldn't get Bazel to resolve CXX
but the following works:
diff --git a/BUILD.bazel b/BUILD.bazel
index 9bbb466..f0ebedf 100644
--- a/BUILD.bazel
+++ b/BUILD.bazel
@@ -42,6 +42,8 @@ genrule(
LIGHTSTEP_ROOT=$$(dirname $${PWD}/$(location :CMakeLists.txt))
cd $$TEMP_DIR
cmake \\
+ -DCMAKE_C_COMPILER=$$(which $(CC)) \\
+ -DCMAKE_CXX_COMPILER=$$(which $(CC)++) \\
-DWITH_GRPC=OFF \\
-DHEADERS_ONLY=ON \\
-L \\
With the current behavior the Envoy binary is built with two different compilers.
readelf --string-dump .comment bazel-bin/source/exe/envoy-static
String dump of section '.comment':
[ 0] GCC: (Ubuntu 7.3.0-16ubuntu3) 7.3.0
[ 24] clang version 7.0.0-svn341916-1~exp1~20180911115939.26 (branches/release_70)
[ 72] Linker: LLD 7.0.0
com_lightstep_tracer_cpp is being evoked as part of the bazel build for envoyproxy/envoy.
Errror:-
ERROR=1' '-std=gnu++0x' -Iexternal/com_lightstep_tracer_cpp/include -Ibazel-out/ppc-opt/binexternal/com_lightstep_tracer_cpp//include -Iexternal/com_lightstep_tracer_cpp/src -Ibazel-out/ppc-opt/binexternal/com_lightstep_tracer_cpp//src -Iexternal/com_lightstep_tracer_cpp/test -Ibazel-out/ppc-opt/binexternal/com_lightstep_tracer_cpp//test -isystem external/com_lightstep_tracer_cpp/3rd_party/base64/include -isystem bazel-out/ppc-opt/binexternal/com_lightstep_tracer_cpp//3rd_party/base64/include -Wall -Wextra -Werror -Wnon-virtual-dtor -Woverloaded-virtual -Wold-style-cast -Wno-overloaded-virtual -Wvla '-std=c++11' -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE_="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/com_lightstep_tracer_cpp/src/common/utility.cpp -o bazel-out/ppc-opt/bin/external/com_lightstep_tracer_cpp/src/common/_objs/utility_lib/utility.o)
external/com_lightstep_tracer_cpp/src/common/utility.cpp: In function 'void lightstep::WriteEscapedString(std::ostringstream&, opentracing::v2::string_view)':
external/com_lightstep_tracer_cpp/src/common/utility.cpp:75:20: error: comparison is always true due to limited range of data type [-Werror=type-limits]
75 | if ('\x00' <= c && c <= '\x1f') {
| ~~~~~~~^~~~
cc1plus: all warnings being treated as errors
Target //source/exe:envoy-static failed to build
INFO: Elapsed time: 8.142s, Critical Path: 7.69s
INFO: 9 processes: 9 processwrapper-sandbox.
FAILED: Build did NOT complete successfully
[root@d7f1b4d1bfca envoy]#
When calling Close()
on the C++ tracer, there should be an option to flush on exit immediately (how our other tracers currently work). Currently the behavior is to call Close()
and then it waits til the end of the current reporting interval to flush, which isn't optimal.
Related ticket is LS-8887
Please add B3 header support :)
Please see this doc for the problem statement and specifics: https://docs.google.com/document/d/1ukvnSoXkavcojlVXpCRzep8PS_guXly_NbbhHIDgbFY/edit
steady_clock cannot go backwards. the system_clock can move with NTP, leap second, etc. durations observed by comparing timestamps from the system_clock will be incorrect if they span a system clock change.
in reality the library will need to use both clocks. start_timestamp should come from the system_clock and duration_micros should be measured using steady_clock.
Dockerfile:
FROM centos:7.2.1511
# prep system
RUN yum update -y
RUN yum group install -y 'Development Tools'
RUN yum install -y epel-release
RUN yum install -y cmake3
WORKDIR /tmp
RUN curl -O https://cbs.centos.org/kojifiles/packages/protobuf/3.6.1/4.el7/x86_64/protobuf-3.6.1-4.el7.x86_64.rpm && \
curl -O https://cbs.centos.org/kojifiles/packages/protobuf/3.6.1/4.el7/x86_64/protobuf-compiler-3.6.1-4.el7.x86_64.rpm && \
curl -O https://cbs.centos.org/kojifiles/packages/protobuf/3.6.1/4.el7/x86_64/protobuf-devel-3.6.1-4.el7.x86_64.rpm && \
yum -y install protobuf-3.6.1-4.el7.x86_64.rpm protobuf-compiler-3.6.1-4.el7.x86_64.rpm protobuf-devel-3.6.1-4.el7.x86_64.rpm c-ares-devel libevent-devel && \
rm protobuf-3.6.1-4.el7.x86_64.rpm protobuf-compiler-3.6.1-4.el7.x86_64.rpm protobuf-devel-3.6.1-4.el7.x86_64.rpm
# OpenTracing API for C++
RUN git clone -b v1.5.1 https://github.com/opentracing/opentracing-cpp.git && \
mkdir opentracing-cpp/.build && \
cd opentracing-cpp/.build && \
cmake3 .. && \
make && \
make install && \
cd ../.. && \
rm -rf opentracing-cpp
# LightStep distributed tracing library for C++.
RUN git clone https://github.com/lightstep/lightstep-tracer-cpp.git && \
mkdir lightstep-tracer-cpp/.build && \
cd lightstep-tracer-cpp/.build && \
cmake3 -DWITH_LIBEVENT=ON -DWITH_CARES=ON -DWITH_GRPC=OFF -DWITH_DYNAMIC_LOAD=OFF .. && \
make && \
make install && \
cd ../.. && \
rm -rf lightstep-tracer-cpp
output/error:
-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.8.5
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find Protobuf (missing: Protobuf_DIR)
-- Found Protobuf: /usr/lib64/libprotobuf.so;-lpthread (found version "3.6.1")
-- Found libevent: /usr/lib64/libevent_core.so
-- Found c-ares: /usr/lib64/libcares.so
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/lightstep-tracer-cpp/.build
Scanning dependencies of target lightstep_3rd_party
[ 0%] Building CXX object 3rd_party/CMakeFiles/lightstep_3rd_party.dir/base64/src/base64.cpp.o
[ 0%] Built target lightstep_3rd_party
[ 1%] Generating generated/lightstep-tracer-configuration/tracer_configuration.pb.cc, generated/lightstep-tracer-configuration/tracer_configuration.pb.h
Scanning dependencies of target lightstep_tracer_configuration
[ 1%] Building CXX object CMakeFiles/lightstep_tracer_configuration.dir/generated/lightstep-tracer-configuration/tracer_configuration.pb.cc.o
[ 1%] Built target lightstep_tracer_configuration
[ 2%] Generating generated/lightstep-tracer-common/google/api/http.pb.h, generated/lightstep-tracer-common/google/api/http.pb.cc, generated/lightstep-tracer-common/google/api/annotations.pb.h, generated/lightstep-tracer-common/google/api/annotations.pb.cc, generated/lightstep-tracer-common/collector.pb.h, generated/lightstep-tracer-common/collector.pb.cc, generated/lightstep-tracer-common/lightstep_carrier.pb.h, generated/lightstep-tracer-common/lightstep_carrier.pb.cc
Scanning dependencies of target lightstep_tracer_common
[ 2%] Building CXX object CMakeFiles/lightstep_tracer_common.dir/generated/lightstep-tracer-common/google/api/http.pb.cc.o
[ 3%] Building CXX object CMakeFiles/lightstep_tracer_common.dir/generated/lightstep-tracer-common/google/api/annotations.pb.cc.o
[ 3%] Building CXX object CMakeFiles/lightstep_tracer_common.dir/generated/lightstep-tracer-common/collector.pb.cc.o
[ 4%] Building CXX object CMakeFiles/lightstep_tracer_common.dir/generated/lightstep-tracer-common/lightstep_carrier.pb.cc.o
[ 4%] Built target lightstep_tracer_common
Scanning dependencies of target lightstep_tracer
[ 5%] Building CXX object CMakeFiles/lightstep_tracer.dir/src/common/utility.cpp.o
In file included from /tmp/lightstep-tracer-cpp/src/common/utility.cpp:1:0:
/tmp/lightstep-tracer-cpp/src/common/utility.h: In function 'std::tuple<long unsigned int, unsigned int> lightstep::ProtobufFormatTimestamp(const time_point&)':
/tmp/lightstep-tracer-cpp/src/common/utility.h:31:53: error: converting to 'std::tuple<long unsigned int, unsigned int>' from initializer list would use explicit constructor 'constexpr std::tuple<_T1, _T2>::tuple(_U1&&, _U2&&) [with _U1 = long unsigned int; _U2 = unsigned int; <template-parameter-2-3> = void; _T1 = long unsigned int; _T2 = unsigned int]'
static_cast<uint32_t>(nanos % nanosPerSec)};
^
make[2]: *** [CMakeFiles/lightstep_tracer.dir/src/common/utility.cpp.o] Error 1
make[1]: *** [CMakeFiles/lightstep_tracer.dir/all] Error 2
make: *** [all] Error 2
When using the python bindings, there is a memory leak when creating spans with this incorrect pattern:
with tracer.start_span(operation_name='make_some_request') as client_span:
tracer.scope_manager.activate(client_span, True)
pass
This issue is documented more extensively here:
lightstep/lightstep-tracer-python#74
Hello maintainers of lightstep-tracer-cpp ๐,
The Bazel team is in progress of migrating the native Protobuf rules to Starlark. As a first step towards this goal, starting with Bazel 3.0, all Protobuf rules will require explicit load statements.
You can use the following buildifier command to apply most of the required migrations for you:
buildifier --lint=fix --warnings=native-proto $(find . -name "BUILD" -o -name "BUILD.bazel" -o -name "*.bzl")
For further information, see bazelbuild/bazel#8922 or ping me.
Thanks,
Yannic
when we use the MakeSpanContext function
Currently posix specific libraries are in use.
Having the lib available statically or as a dll would solve this and allow the tracer to be used on Windows
Relevant LS ticket is PRD-494
Folks,
It will be great if we could make this library available within vcpkg. I have opened an issue for this here. Seems like all the dependencies are already available in vcpkg.
Potential (however small) of one component in the system blocking for 24 hours. That could cascade into all sorts of failures. Timeouts in the range milliseconds are more appropriate and give the option to override.
The document is really limited about how to pass along the trace_id, trace_parent, span_id etc when I use a textmapcarrier,
For example, you want to create a carrier on your own as a text map, then you do something like
m_textMapCarrier.Set("traceid", m_trace_parent.get_trace_id());// I got this id somewhere in my system
m_textMapCarrier.Set("spanid", m_trace_parent.get_parent_id());// I got this id somewhere in my system
Then you use this m_textMapCarrier somewhere to create a span.
const auto carrier_reader = get_trace_text_map_carrier();
const auto context = tracer->Extract(carrier_reader);
What information does the carrier want? I saw different versions of spanid, Span_ID, Span-ID, x-b3-traceid(not in the current stable release)
, tried all of them , not seeing the tracer to pick up the parent when I
I was using the LightStepTracer::MakeSpanContext
to pass the ids directly to it, but I was told that is not a good practice as I wasn't be using the proper standard opentracing propagation API. With this method. I am able to set the parent with my own ID
Here is my Dockerfile:
FROM ubuntu:20.04
# prep system
RUN ln -fs /usr/share/zoneinfo/America/Los_Angeles /etc/localtime
RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y libpcre3-dev libssl-dev zlib1g-dev libprotobuf-dev protobuf-compiler perl make cmake build-essential curl unzip autoconf libtool pkg-config git
WORKDIR /tmp
# OpenTracing API for C++
RUN git clone -b v1.5.1 https://github.com/opentracing/opentracing-cpp.git && \
mkdir opentracing-cpp/.build && \
cd opentracing-cpp/.build && \
cmake .. && \
make && \
make install && \
cd ../.. && \
rm -rf opentracing-cpp
# gRPC
RUN git clone -b v1.28.1 --recursive https://github.com/grpc/grpc && \
mkdir -p grpc/cmake/build && \
cd grpc/cmake/build && \
cmake ../.. && \
make && \
make install && \
cd ../../.. && \
rm -rf grpc
# LightStep distributed tracing library for C++.
RUN git clone https://github.com/lightstep/lightstep-tracer-cpp.git && \
mkdir lightstep-tracer-cpp/.build && \
cd lightstep-tracer-cpp/.build && \
cmake -DWITH_GRPC=ON -DWITH_DYNAMIC_LOAD=OFF .. && \
make && \
make install && \
cd ../.. && \
rm -rf lightstep-tracer-cpp
and here are the errors I get:
Cloning into 'lightstep-tracer-cpp'...
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found Protobuf: /usr/local/bin/protoc-3.11.2.0 (found version "3.11.2.0")
-- Found Protobuf: /usr/local/lib/libprotobuf.a (found version "3.11.2")
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/lightstep-tracer-cpp/.build
Scanning dependencies of target lightstep_3rd_party
...
[100%] Linking CXX executable tutorial
/usr/bin/ld: ../../liblightstep_tracer.so.0: undefined reference to `lightstep::MetricsTracker::MetricsTracker(lightstep::MetricsObserver&)'
/usr/bin/ld: ../../liblightstep_tracer.so.0: undefined reference to `lightstep::MetricsTracker::OnFlush()'
/usr/bin/ld: ../../liblightstep_tracer.so.0: undefined reference to `lightstep::MetricsTracker::OnSpansSent(int)'
/usr/bin/ld: ../../liblightstep_tracer.so.0: undefined reference to `lightstep::MetricsTracker::ConsumeDroppedSpans()'
/usr/bin/ld: ../../liblightstep_tracer.so.0: undefined reference to `lightstep::MetricsTracker::UnconsumeDroppedSpans(int)'
collect2: error: ld returned 1 exit status
make[2]: *** [example/tutorial/CMakeFiles/tutorial.dir/build.make:109: example/tutorial/tutorial] Error 1
make[1]: *** [CMakeFiles/Makefile2:1135: example/tutorial/CMakeFiles/tutorial.dir/all] Error 2
make: *** [Makefile:163: all] Error 2
Help appreciated.
When the wrong library version for opentracing-cpp is installed, the build fails in difficult-to-understand ways. We should do something to protect against this. I think checking for a compatible release and printing an error about getting the proper version will suffice.
Please add trace-context header support :)
Please see this doc for the problem statement and specifics: https://docs.google.com/document/d/1ukvnSoXkavcojlVXpCRzep8PS_guXly_NbbhHIDgbFY/edit
We are currently building GRPC with cmake because it affords some additional flexibility with respect to managing grpc's dependencies.
It would be convenient for us (and possibly others) if LightStep's cmake build could perhaps optionally find GRPC via find_package / cmake config (e.g. lib/cmake/grpc/*.cmake
and friends) in addition to its current pkg config based support.
Thanks!
The API always requires an access token, even when access token is not needed because private satellite pool is used.
It would be clearer if access token is set as not required, that way private satellite pools can be used as intended, and more descriptive error messages can be returned when private satellite pools are not used.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.