grafana/pyroscope-dotnet

Dotnet profiler

Pyroscope .NET Agent

The .NET profiling agent for Pyroscope.io. Based on dd-trace-dotnet with some key improvements:

• support for dynamic tags
• export to pyroscope

Documentation.

Copyright

Copyright (c) 2017 Datadog https://www.datadoghq.com

Copyright (c) 2022 Pyroscope

License

See license information.

Contact us

Security Vulnerabilities

If you have found a security issue, please contact the security team directly at [email protected].

pyroscope-dotnet's Issues

When running Pyroscope on Docker getting LoadDynamicLibrary: Error loading dynamic library '/dotnet/Datadog.Trace.ClrProfiler.Native.so': /dotnet/Datadog.Trace.ClrProfiler.Native.so: cannot open shared object file: No such file or directory

Started a container with configuration as such one:

WORKDIR /dotnet

RUN apt-get update && apt-get install -y wget tar
RUN wget -qO- https://github.com/grafana/pyroscope-dotnet/releases/download/v0.8.8-pyroscope/pyroscope.0.8.8-glibc-x86_64.tar.gz
| tar xvz

ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={BD1A650D-AC5D-4896-B64F-D6FA25D6B26A}
ENV CORECLR_PROFILER_PATH=/dotnet/Pyroscope.Profiler.Native.so
ENV LD_PRELOAD=/dotnet/Pyroscope.Linux.ApiWrapper.x64.so

ENV PYROSCOPE_APPLICATION_NAME=APPNAME
ENV PYROSCOPE_SERVER_ADDRESS=URLHERE
ENV PYROSCOPE_LOG_LEVEL=debug
ENV PYROSCOPE_PROFILING_ENABLED=1
ENV PYROSCOPE_PROFILING_ALLOCATION_ENABLED=true
ENV PYROSCOPE_PROFILING_CONTENTION_ENABLED=true
ENV PYROSCOPE_PROFILING_EXCEPTION_ENABLED=true

And when running container (where also detail on how to run application is included) I'm getting Pyroscope logs but with reference to DataDog file:

[2023-08-22 10:37:38.471 | info | PId: 1 | TId: 1] Profiler DLL loaded.
[2023-08-22 10:37:38.471 | info | PId: 1 | TId: 1] Pointer size: 64 bits.
[2023-08-22 10:37:38.471 | info | PId: 1 | TId: 1] Value "1" for "DD_PROFILING_ENABLED" environment variable. Enable = 1
[2023-08-22 10:37:38.471 | info | PId: 1 | TId: 1] DllGetClassObject(): Returning an instance of CorProfilerCallbackFactory (hr=0x0)
[2023-08-22 10:37:38.472 | info | PId: 1 | TId: 1] CorProfilerCallback is initializing.
[2023-08-22 10:37:38.472 | info | PId: 1 | TId: 1] No "DD_TRACE_DEBUG" environment variable has been found. Enable debug log = 0 (default).
[2023-08-22 10:37:38.472 | info | PId: 1 | TId: 1] Environment variables:
[2023-08-22 10:37:38.472 | info | PId: 1 | TId: 1] ICorProfilerInfo13 available. Profiling API compatibility: .NET Core 7.0 or later.
[2023-08-22 10:37:38.472 | info | PId: 1 | TId: 1] Initializing the Profiler: Reported runtime version : { clrInstanceId: 0, runtimeType:CORE_CLR, majorVersion: 7, minorVersion: 0, buildNumber: 10, qfeVersion: 0 }.
[2023-08-22 10:37:38.472 | info | PId: 1 | TId: 1] No "DD_INTERNAL_OPERATIONAL_METRICS_ENABLED" environment variable has been found. Default: Operational metrics disabled.
[2023-08-22 10:37:38.472 | info | PId: 1 | TId: 1] ProfilerSignalManager::SetupSignalHandler: Successfully setup signal handler for SIGUSR1 signal.
[2023-08-22 10:37:38.472 | info | PId: 1 | TId: 1] ThreadsCpuManager started successfully.
[2023-08-22 10:37:38.472 | info | PId: 1 | TId: 1] ManagedThreadList started successfully.
[2023-08-22 10:37:38.472 | info | PId: 1 | TId: 1] ManagedThreadList started successfully.
**[2023-08-22 10:37:38.472 | warning | PId: 1 | TId: 1] LoadDynamicLibrary: Error loading dynamic library '/dotnet/Datadog.Trace.ClrProfiler.Native.so': /dotnet/Datadog.Trace.ClrProfiler.Native.so: cannot open shared object file: No such file or directory
[2023-08-22 10:37:38.472 | warning | PId: 1 | TId: 1] The RuntimeID store service couldn't load the native proxy.**
[2023-08-22 10:37:38.472 | info | PId: 1 | TId: 1] RuntimeID Store started successfully.
[2023-08-22 10:37:38.472 | info | PId: 1 | TId: 1] CpuTimeProvider started successfully.
[2023-08-22 10:37:38.472 | info | PId: 1 | TId: 1] ExceptionsProvider started successfully.
[2023-08-22 10:37:38.472 | info | PId: 1 | TId: 1] AllocationsProvider started successfully.
[2023-08-22 10:37:38.472 | info | PId: 1 | TId: 1] ContentionProvider started successfully.
[2023-08-22 10:37:38.472 | info | PId: 1 | TId: 1] Processor cores = 20
[2023-08-22 10:37:38.472 | info | PId: 1 | TId: 1] CPU and wall time sampling period = 9 ms

Tracing integration should set `span_name` pprof label

By convention, pyroscope SDKs also propagate span name as a pprof label, which allows building aggregated span profiles. pyroscope-dotnet should do the same

How to profile async code?

If we check documentation
https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/dotnet/

now we can profile only sync code

var labels = Pyroscope.LabelSet.Empty.BuildUpon()
    .Add("key1", "value1")
    .Build();
Pyroscope.LabelsWrapper.Do(labels, () =>
{
  SlowCode();
});

Do you have plans for add DoAsync version?

Client applications should be able to use non-bearer authorization

Currently it seems that client applications are forced to have their tokens interpreted as a Bearer token. This causes problems where other authentication schemes are required.

An option that allows the client to set the entire value of the Authorization header would be the fix for my use case.

https://github.com/pyroscope-io/pyroscope-dotnet/blob/a3682736d2bc6e0b8bf043b69a03265794d8f677/profiler/src/ProfilerEngine/Datadog.Profiler.Native/PyroscopePprofSink.cpp#L14

Try to update libunwind, maybe it will help with arm64

libunwind/libunwind#382

Error about Datadog.Trace.ClrProfiler.Native.so

Hello,

I was checking the log of Pyroscope and this is what I found:

[2023-06-22 21:15:17.725 | info | PId: 1 | TId: 1] Profiler DLL loaded.
[2023-06-22 21:15:17.725 | info | PId: 1 | TId: 1] Pointer size: 64 bits.
[2023-06-22 21:15:17.726 | info | PId: 1 | TId: 1] Value "1" for "DD_PROFILING_ENABLED" environment variable. Enable = 1
[2023-06-22 21:15:17.726 | info | PId: 1 | TId: 1] DllGetClassObject(): Returning an instance of CorProfilerCallbackFactory (hr=0x0)
[2023-06-22 21:15:17.726 | info | PId: 1 | TId: 1] CorProfilerCallback is initializing.
[2023-06-22 21:15:17.726 | info | PId: 1 | TId: 1] No "DD_TRACE_DEBUG" environment variable has been found. Enable debug log = 0 (default).
[2023-06-22 21:15:17.727 | info | PId: 1 | TId: 1] Environment variables:
[2023-06-22 21:15:17.727 | info | PId: 1 | TId: 1] ICorProfilerInfo13 available. Profiling API compatibility: .NET Core 7.0 or later.
[2023-06-22 21:15:17.727 | info | PId: 1 | TId: 1] Initializing the Profiler: Reported runtime version : { clrInstanceId: 0, runtimeType:CORE_CLR, majorVersion: 7, minorVersion: 0, buildNumber: 8, qfeVersion: 0 }.
[2023-06-22 21:15:17.727 | info | PId: 1 | TId: 1] No "DD_INTERNAL_OPERATIONAL_METRICS_ENABLED" environment variable has been found. Default: Operational metrics disabled.
[2023-06-22 21:15:17.727 | info | PId: 1 | TId: 1] ProfilerSignalManager::SetupSignalHandler: Successfully setup signal handler for SIGUSR1 signal.
[2023-06-22 21:15:17.728 | info | PId: 1 | TId: 1] ThreadsCpuManager started successfully.
[2023-06-22 21:15:17.728 | info | PId: 1 | TId: 1] ManagedThreadList started successfully.
[2023-06-22 21:15:17.728 | info | PId: 1 | TId: 1] ManagedThreadList started successfully.
[2023-06-22 21:15:17.728 | warning | PId: 1 | TId: 1] LoadDynamicLibrary: Error loading dynamic library '/lib/Datadog.Trace.ClrProfiler.Native.so': Error loading shared library /lib/Datadog.Trace.ClrProfiler.Native.so: No such file or directory
[2023-06-22 21:15:17.728 | warning | PId: 1 | TId: 1] The RuntimeID store service couldn't load the native proxy.
[2023-06-22 21:15:17.728 | info | PId: 1 | TId: 1] RuntimeID Store started successfully.
[2023-06-22 21:15:17.728 | info | PId: 1 | TId: 1] CpuTimeProvider started successfully.
[2023-06-22 21:15:17.728 | info | PId: 1 | TId: 1] Processor cores = 4
[2023-06-22 21:15:17.728 | info | PId: 1 | TId: 1] CPU and wall time sampling period = 9 ms
[2023-06-22 21:15:17.728 | info | PId: 1 | TId: 1] Wall time sampled threads = 5
[2023-06-22 21:15:17.728 | info | PId: 1 | TId: 1] Max CodeHotspots sampled threads = 10
[2023-06-22 21:15:17.728 | info | PId: 1 | TId: 1] Max CPU sampled threads = 64
[2023-06-22 21:15:17.728 | info | PId: 1 | TId: 1] StackSamplerLoopManager started successfully.
[2023-06-22 21:15:17.728 | info | PId: 1 | TId: 1] ApplicationStore started successfully.
[2023-06-22 21:15:17.728 | info | PId: 1 | TId: 1] Starting the samples collector
[2023-06-22 21:15:17.728 | info | PId: 1 | TId: 16] StackSamplerLoopManager::WatcherLoop started.
[2023-06-22 21:15:17.728 | info | PId: 1 | TId: 1] SamplesCollector started successfully.
[2023-06-22 21:15:20.352 | info | PId: 1 | TId: 14] PyroscopePprofSink 200
[2023-06-22 21:15:30.156 | info | PId: 1 | TId: 14] PyroscopePprofSink 200
[2023-06-22 21:15:40.124 | info | PId: 1 | TId: 14] PyroscopePprofSink 200
[2023-06-22 21:15:50.071 | info | PId: 1 | TId: 14] PyroscopePprofSink 200
[2023-06-22 21:16:00.081 | info | PId: 1 | TId: 14] PyroscopePprofSink 200
[2023-06-22 21:16:10.075 | info | PId: 1 | TId: 14] PyroscopePprofSink 200
[2023-06-22 21:16:20.071 | info | PId: 1 | TId: 14] PyroscopePprofSink 200

I also currently see no data in Grafana cloud.
Here is my env vars:

CORECLR_ENABLE_PROFILING=1
CORECLR_PROFILER_PATH=/lib/Pyroscope.Profiler.Native.so
CORECLR_PROFILER={BD1A650D-AC5D-4896-B64F-D6FA25D6B26A}

PYROSCOPE_SERVER_ADDRESS=https://profiles-prod-010.grafana.net
PYROSCOPE_BASIC_AUTH_PASSWORD=MY_PUBLISH_METRIC_API_KEY
PYROSCOPE_PROFILING_LOG_DIR=/tmp/
PYROSCOPE_APPLICATION_NAME=AddictedProxy
PYROSCOPE_BASIC_AUTH_USER=MY_USER_IN_GRAFANA
PYROSCOPE_APPLICATION_TAGS=env:prod
PYROSCOPE_PROFILING_ENABLED=1

Content of /lib folder

/app # ls -al /lib
total 29512
drwx------    1 1001     ntp           4096 Jun 22 21:13 .
drwxr-xr-x    1 root     root          4096 Jun 22 21:15 ..
-rwxr-xr-x    1 1001     ntp          17160 May 11 09:00 Pyroscope.Linux.ApiWrapper.x64.so
-rwxr-xr-x    1 1001     ntp       24748752 May 11 09:00 Pyroscope.Profiler.Native.so
drwxr-xr-x    1 root     root          4096 Jun 14 14:28 apk
drwxr-xr-x    2 root     root          4096 Jun 14 14:28 firmware
-rwxr-xr-x    1 root     root        616992 May 19 15:29 ld-musl-x86_64.so.1
-rwxr-xr-x    1 root     root        184008 Oct 23  2022 libapk.so.3.12.0
lrwxrwxrwx    1 root     root            19 Jun 14 14:28 libc.musl-x86_64.so.1 -> ld-musl-x86_64.so.1
lrwxrwxrwx    1 root     root            17 Jun 15 00:15 libcom_err.so.2 -> libcom_err.so.2.1
-rwxr-xr-x    1 root     root         18184 Mar  5 17:04 libcom_err.so.2.1
-rwxr-xr-x    1 root     root       3882720 May 30 17:48 libcrypto.so.3
-rwxr-xr-x    1 root     root        602120 May 30 17:48 libssl.so.3
lrwxrwxrwx    1 root     root            14 Jun 14 14:28 libz.so.1 -> libz.so.1.2.13
-rwxr-xr-x    1 root     root        100264 Oct 13  2022 libz.so.1.2.13
drwxr-xr-x    2 root     root          4096 Jun 14 14:28 mdev
drwxr-xr-x    2 root     root          4096 Jun 14 14:28 modules-load.d
drwxr-xr-x    2 root     root          4096 Jun 14 14:28 sysctl.d

Pyroscope libraries resulting in segmentation fault

Following the instructions for the dotnet beta integration. I am trying to profile an ASPNET application with Pyroscope.

Dockerfile below - this is largely copied from a Microsoft example. When I build and run this, the container exits with a 139 code (seg fault).

Things I've tried:

• Downgrading Pyroscope libraries from 0.7.2 to 0.7.0
• Using Debian and Ubuntu images instead of Alpine
• Messing around with the libc6-compat library instead of gcompat

FROM mcr.microsoft.com/dotnet/sdk:6.0-alpine3.16 AS build
WORKDIR /sln

# Copy project file and restore
COPY "TestApp.Api.csproj" "."
RUN dotnet restore TestApp.Api.csproj

# Copy the actual source code
COPY "." "."

# Build and publish the app
RUN dotnet publish "TestApp.Api.csproj" -c Release -o /app/publish --self-contained false --no-restore

#FINAL image
FROM mcr.microsoft.com/dotnet/aspnet:6.0-alpine3.16
WORKDIR /app
COPY --from=build /app/publish .

# Set up Pyroscope profiling
RUN apk update && apk add gcompat
RUN wget -P /lib/ https://github.com/pyroscope-io/pyroscope-dotnet/releases/download/v0.7.0/Pyroscope.Linux.ApiWrapper.x64.so
RUN wget -P /lib/ https://github.com/pyroscope-io/pyroscope-dotnet/releases/download/v0.7.0/Pyroscope.Profiler.Native.so
ENV PYROSCOPE_APPLICATION_NAME=test.dotnet.app
ENV PYROSCOPE_SERVER_ADDRESS=http://pyroscope-route:80
ENV PYROSCOPE_PROFILING_ENABLED=1
ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={BD1A650D-AC5D-4896-B64F-D6FA25D6B26A}
ENV CORECLR_PROFILER_PATH=/lib/Pyroscope.Profiler.Native.so
ENV LD_PRELOAD=/lib/Pyroscope.Linux.ApiWrapper.x64.so

ENTRYPOINT ["dotnet", "TestApp.Api.dll"]

Improve pyroscope server configuration

• right now if the server ends with a trailing slash it fails to ingest data
• also it is now impossible to ingest to a url which is not index(for example is a pyroscope server is proxied from http://host:4040/ops/pyroscope/)

Dynamic tags/labels not working

Hello!

Before anything, thanks a lot for this library, despite this issue, I find this tool to be a complete game changer for any performance-sensitive dotnet service 👏

Context

• I use the Pyroscope service 1.2.0 in K8S to profile dotnet 6 web applications (also in K8S).
• I use the Pyroscope profiler 0.8.14
• I use the Pyroscope nuget package 0.8.14
• I use the current environment variables for my dotnet service:

CORECLR_ENABLE_PROFILING=1
CORECLR_PROFILER={BD1A650D-AC5D-4896-B64F-D6FA25D6B26A}
CORECLR_PROFILER_PATH=/app/Pyroscope.Profiler.Native.so
LD_PRELOAD=/app/Pyroscope.Linux.ApiWrapper.x64.so

PYROSCOPE_APPLICATION_NAME=ssbengine
PYROSCOPE_PROFILING_ENABLED=1
PYROSCOPE_PROFILING_CPU_ENABLED=1
PYROSCOPE_PROFILING_EXCEPTION_ENABLED=1
PYROSCOPE_PROFILING_ALLOCATION_ENABLED=1
PYROSCOPE_PROFILING_LOCK_ENABLED=1
PYROSCOPE_PROFILING_WALLTIME_ENABLED=0

PYROSCOPE_LABELS="version_env:123"

Issue(s)

I am setting dynamic tags using the C# library like this:

Profiler.Instance.ClearDynamicTags();
Profiler.Instance.SetDynamicTag("version_dynamic", "123");

I do not use the LabelsWrapper class.

Expected

I expect the version_dynamic tag to show up in Pyroscope.

Observed

I do not have version_dynamic tag showing up in Pyroscope.
I have however the version_env tag that I have set from the environment variable PYROSCOPE_LABELS.

A few other remarks

• On the documentation/developer experience side, Pyroscope uses the "Tag" terminology, while pyroscope-dotnet names a tag a "Label" instead. This can be confusing.
• The readme explains how to use the LabelWrapper with a delegate action to wrap a code path with specific tags. While this can be an interesting feature, in my opinion, the principal usecase is setting tags once (SetDynamicTag(s)). I believe it is very worthy documenting this part.

Managed helper not working in alpine

Setting tags and other helpers do not work

Automate release changes in Pyroscope/Pyroscope/Pyroscope.csproj

I often forget to update version for managed helper

Which leads to
https://github.com/grafana/pyroscope-dotnet/actions/runs/10403329606/job/28809596112

To skip already published packages, use the option --skip-duplicate
error: Response status code does not indicate success: 409 (A package with ID 'Pyroscope' and version '0.8.17' already exists and cannot be modified.).

split pprof into multiple pprofs based on profile types

currently we sedn 1 pprof with lall profile types
we need to split them so that it has memory cpu, etc separate files

log ingest error message body

Error messages like these are not very useful for troubleshooting especially if you do not own the pyroscope server.

[2024-08-12 23:36:40.305 | info | PId: 1 | TId: 16] PyroscopePprofSink 400
[2024-08-12 23:36:40.305 | info | PId: 1 | TId: 16] PyroscopePprofSink 422

The http response body should contain a useful error, like name: invalid tag key: service.namespace: character is not allowed: '.' or pushing IngestInput-pprof failed invalid_argument: profile with labels '{__delta__=\"false\", __name__=\"wall\", deployment_environment=\"dev\", pyroscope_spy=\"dotnetspy\", service_name=\"webmvc\", service_name=\"webmvc\", service_namespace=\"eshop\"}' has duplicate label name. Which we should include in the info log message to aid troubleshooting

protobuf parsing error when sending info to vector

Context

I'm developing a PoC about connecting pyroscope-dotnet with vector. I have this example app running in a simple pod within a k8s cluster.

PYROSCOPE_SERVER_ADDRESS env var is set and points to a vector http server using protobuf decoding like so:

pyroscope:
    type: http_server
    address: 0.0.0.0:4040
    path: /ingest
    decoding:
      codec: protobuf
      protobuf:
        desc_file: /vector-data-files/profile.desc
        message_type: perftools.profiles.Profile

As you might see, I'm referencing a profile.desc based on this .proto file and generating it using protoc CLI tool like this:

protoc --descriptor_set_out=profile.desc --include_imports profile.proto

profile.desc is being transformed into a configMap so the pod is able to consume it.

Actual Behavior

Seems that both actors are able to communicate but when checking the logs I see the following:

• Sample app is erroring as follows:
  [2024-01-27 00:30:20.158 | info | PId: 1 | TId: 13] PyroscopePprofSink 400

• vector logs shows:

{"error":"Error parsing protobuf: DecodeError { description: \"invalid wire type: ThirtyTwoBit (expected LengthDelimited)\", stack: [] }","error_code":"decoder_deserialize","error_type":"parser_failed","host":"aks-default-42715425-vmss000028","internal_log_rate_limit":true,"message":"Failed deserializing frame.","metadata":{"kind":"event","level":"ERROR","module_path":"vector::internal_events::codecs","target":"vector::internal_events::codecs"},"pid":1,"source_type":"internal_logs","stage":"processing","timestamp":"2024-01-27T00:30:20.158087899Z","vector":{"component_id":"pyroscope-ingest","component_kind":"source","component_type":"http_server"}}

Questions

I'm pretty new to all this stuff but I was just wondering if:

1. This is even possible
2. That's the right .proto file to use (I've noticed there are other .proto file in grafana/pyroscope repo and also in vector repo)

If all this make sense, what could be causing the parsing error and whats the right path to follow in order to fix it?

adjust make files to tag lastest docker image conditionally

sometimes we may want to publish an image without tagging it as latest.

• consider pushing -dev images without tagging them latest

Bump dd-trace-dotnet to the last version

Hi,

We are using pyroscope-dotnet to profile our C# applications with Pyroscope.

Unfortunately, the applications with profiling enabled are crashing every 1/2 days.

The crash seems very similar to these, related here:
#62

And it seems fixed with a newer version of dd-trace-dotnet:
DataDog/dd-trace-dotnet#4753

Would it be possible to bump to the last version of dd-trace-dotnet ?

Thanks a lot.

Disable logging to file by default

By default we are logging to a file like /var/log/pyroscope/dotnet/Pyroscope-DotNet-Profiler-Native-dotnet-7.log.

I haven't found a way to disable this using the environment variable PYROSCOPE_PROFILING_LOG_DIR (and I don't think there is one). I would argue by default we should not log to files and only if a value is given to PYROSCOPE_PROFILING_LOG_DIR, we should write into a file.

Esp. when the directory cannot be created this blocks the whole profiling session:

Logger::GetLogPath failed to create a parent directory: "/var/log/pyroscope/dotnet"
LoggerImpl Handler: Error creating native log file.

Can be recreated by testing the example as non-root:

--- a/examples/language-sdk-instrumentation/dotnet/rideshare/Dockerfile
+++ b/examples/language-sdk-instrumentation/dotnet/rideshare/Dockerfile
@@ -24,5 +24,7 @@ ENV PYROSCOPE_PROFILING_CONTENTION_ENABLED=true
 ENV PYROSCOPE_PROFILING_EXCEPTION_ENABLED=true
 ENV RIDESHARE_LISTEN_PORT=5000

+USER nobody
+

 CMD sh -c "ASPNETCORE_URLS=http://*:${RIDESHARE_LISTEN_PORT} exec dotnet /dotnet/example.dll"

.NET 8 support

Are there plans to support .NET 8 soon? The DataDog profiler has supported it since last November (v2.42.0).

Please add support for Windows as OS

Currently the profiler only supports running Dotnet applications on Linux but being able to also profile them while running on windows would be quite nice, since a lot of people probably run dotnet apps on Windows using IIS.

The README.md mentions that this profiler is based on https://github.com/DataDog/dd-trace-dotnet which seems to support Windows as OS. Source: https://docs.datadoghq.com/profiler/enabling/dotnet?tab=windows#requirements

Thanks a lot

Clarify purpose and use of environment variables in documentation

Hi folks. There's a discrepancy between the variables listed in the docs, and those which actually have an effect in practice.

For example PYROSCOPE_PROFILING_LOG_DIR is shown in the docs, but a search of this repo suggests that this variable does not appear in the code.

I'm currently attempting to configure the profiling-related logging for our app, but if there's a way to reduce the verbosity, I think can't find it.

The following variables do appear to have an effect on the location and verbosity, but they don't support minimum-level restrictions, and more importantly, they don't have any effect on the logs written to /dev/stdout.

export DD_TRACE_DEBUG="0"
export DD_TRACE_LOG_DIRECTORY="${PYROSCOPE_PATH}/logs/"

There appear to be many such DD_-prefixed variables, but it's unclear which of these ought to be configured by consumers of the nuget package versus package developers.

Any light you can shed on this would be very helpful, thank you.

Allow configuration methods other than env vars (i.e. implement IConfiguration support)

I would like to be able to configure all the settings from my application using the standard MSFT IConfiguration system.

Thanks!

Program terminated with signal SIGSEGV, Segmentation fault.

Sorry, my english is very poor
More than 100 docker containers have been started, and occasionally a container will be hung due to a segmentation fault in pyroscope.

os: debian 10
dotnet version: 8.0.201
pyroscope version: pyroscope.0.8.14-glibc-x86_64.tar.gz

enable optionals：
ENV PYROSCOPE_PROFILING_ENABLED=1
ENV PYROSCOPE_PROFILING_ALLOCATION_ENABLED=true
ENV PYROSCOPE_PROFILING_CONTENTION_ENABLED=true
ENV PYROSCOPE_PROFILING_EXCEPTION_ENABLED=true

#0 0x00007f3d00b17dc4 in _ULx86_64_tdep_trace () from Pyroscope.Profiler.Native.so
[Current thread is 1 (Thread 0x7f3d0403cc40 (LWP 7))]
(gdb) bt
#0 0x00007f3d00b17dc4 in _ULx86_64_tdep_trace () from Pyroscope.Profiler.Native.so
#1 0x00007f3d00b16963 in unw_backtrace2 () from Pyroscope.Profiler.Native.so
#2 0x00007f3d00a4582a in LinuxStackFramesCollector::CollectCallStackCurrentThread(void*) () from Pyroscope.Profiler.Native.so
#3 0x00007f3d00a44c42 in LinuxStackFramesCollector::CollectStackSampleSignalHandler(int, siginfo_t*, void*) () from Pyroscope.Profiler.Native.so
#4 0x00007f3d00a472f3 in ProfilerSignalManager::SignalHandler(int, siginfo_t*, void*) () from Pyroscope.Profiler.Native.so
#5
#6 0x00007f3d0413f8f7 in __GI_munmap () at ../sysdeps/unix/syscall-template.S:117
#7 0x00007f3d00b185fc in _ULx86_64_tdep_trace () from Pyroscope.Profiler.Native.so
#8 0x00007f3d00b1682f in unw_backtrace () from Pyroscope.Profiler.Native.so
#9 0x00007f3d00b1698d in unw_backtrace2 () from Pyroscope.Profiler.Native.so
#10 0x00007f3d00a4582a in LinuxStackFramesCollector::CollectCallStackCurrentThread(void*) () from Pyroscope.Profiler.Native.so
#11 0x00007f3d00a45532 in LinuxStackFramesCollector::CollectStackSampleImplementation(ManagedThreadInfo*, unsigned int*, bool) () from Pyroscope.Profiler.Native.so
#12 0x00007f3d00ae378f in StackFramesCollectorBase::CollectStackSample(ManagedThreadInfo*, unsigned int*) () from Pyroscope.Profiler.Native.so
#13 0x00007f3d00a4a79f in AllocationsProvider::OnAllocation(unsigned int, unsigned long, char16_t const*, unsigned long, unsigned long, unsigned long) ()
from Pyroscope.Profiler.Native.so
#14 0x00007f3d00a50ea7 in ClrEventsParser::ParseGcEvent(unsigned int, unsigned int, unsigned int, unsigned char const*) () from Pyroscope.Profiler.Native.so
#15 0x00007f3d00a62113 in CorProfilerCallback::EventPipeEventDelivered(unsigned long, unsigned int, unsigned int, unsigned int, unsigned char const*, unsigned int, unsigned char const*, _GUID const*, _GUID const*, unsigned long, unsigned int, unsigned long*) () from Pyroscope.Profiler.Native.so

why pyroscope crashed

.NET config is not working inside a container

Hello,

I've been attempting to set up Grafana Pyroscope as per the instructions provided in the Grafana Pyroscope Docs. However, despite following the steps closely, I haven't been able to get it running correctly.

Here's a brief rundown of my setup:

I'm utilizing the mcr.microsoft.com/dotnet/runtime-deps:6.0-alpine as my base image.
After setting up this base, I proceed to download and extract the necessary profiling files:

RUN wget https://github.com/grafana/pyroscope-dotnet/releases/download/v0.8.10-pyroscope/pyroscope.0.8.10-musl-x86_64.tar.gz && \
    tar xvzf pyroscope.0.8.10-musl-x86_64.tar.gz -C .

I've also configured the environment with the recommended default variables:

ENV PYROSCOPE_APPLICATION_NAME=web-api
ENV PYROSCOPE_SERVER_ADDRESS=http://pyroscope.profiling.svc.cluster.local:4040
ENV PYROSCOPE_PROFILING_ENABLED=1
ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={BD1A650D-AC5D-4896-B64F-D6FA25D6B26A}
ENV CORECLR_PROFILER_PATH=Pyroscope.Profiler.Native.so
ENV LD_PRELOAD=Pyroscope.Linux.ApiWrapper.x64.so

Additionally, I've opted to use musl over glibc in line with the guidance on the Pyroscope Dotnet Profiler Docker Hub page.

Is there something I might be overlooking in this process?

Thanks for your assistance.

Musl - Arm64

I would like to know if Pyroscope supports Alpine\Arm64.

I saw the docker image that is supported, but the first test didn't work: Img Docker

Trace to Profile in aspnetcore nested Activities

The PyroscopeSpanProcessor works like a charm with aspnetcore, I see the pyroscope.profile.id in the root span and the linked profile includes the cpu/mem intensive call stack

        [HttpGet]
        public async Task<string> Get()
        {
            ... cpu/mem intensive work ...
        }

But it does not work when manually defining child Activities. With the following example I see the pyroscope.profile.id in the root span but the linked profile does not include the cpu/mem intensive call stack and there is no pyroscope.profile.id on the MyCustomActivity child span.

        [HttpGet]
        public async Task<string> Get()
        {
                using (var activity = ActivitySource.StartActivity("MyCustomActivity", ActivityKind.Internal, default(ActivityContext)))
                {
                     ... cpu/mem intensive work ...
                }
        }

Environment

.Net 8
opentelemetry.instrumentation.aspnetcore 1.8.1
pyroscope-dotnet v0.2.0-opentelemetry
pyroscope.0.8.14-glibc-x86_64

See: https://github.com/grafana/pyroscope-dotnet/blob/main/Pyroscope/Pyroscope.OpenTelemetry/PyroscopeSpanProcessor.cs#L12

When will upgrade to support .net7 and Live heap profiling?

The datadog .NET Profiler is already supported this.