Windows Service memory grows indefinitely with Tracing and Profiling enabled.

[jamescrosswell]

Windows Service memory grows indefinitely with Tracing and Profiling enabled.

[jamescrosswell]

@jamescrosswell any chance you could verify the fix in #3382 ?

It works - thanks @vaind!🕺🏻

[vaind]

thank you!

for reference: this was the actual fix: microsoft/perfview@36d2e2c#diff-31ffbfdad96fc84f3aa7d25e1980cdca5a4d7870529c669ac9cc975cf24969c8R342

[jamescrosswell]

So there does appear to be an issue here. This is what I'm seeing when I run the service in a memory profiler:

Although the LOH and POH do get cleaned up occassionally (somewhat), over time these just keep growing and growing. Heap Gen 0, 1 and 2 all appears stable.

The only thing that I can think might be using the LOH or POH is profiling, which might need to pin objects when passing pointers to unsafe code. I see a number of instances where the PerView module pins objects (e.g. here).

That's consistent with the call tree, which indicates 99% of the memory is being used by instances of EventMarker, which is an internal class used by modules/perfview/src/TraceEvent/EventPipe/EventCache.cs:

The EventMarker constructor takes a PinnedBuffer as an argument.

It looks like the ProcessEventBlock method creates multiple EventMarker instances and never explicitly cleans any of these up.

I'll play around to see if I can explicitly free these up.

[jamescrosswell]

When I look at the generations I can see:

Pretty much all of that is accounted for by Microsoft.Diagnostics.Tracing.Etlx.TraceLog+EventsToStackIndex[]

I see events get added to that stack here but the only place I see them being removed is in the internal FlushRealtimeEvents method, which in turn gets called by TraceLogProcess, which we call here when starting the profiling:

sentry-dotnet/src/Sentry.Profiling/SampleProfilerSession.cs

Line 88 in ca3744b

Task.Factory.StartNew(eventSource.Process, TaskCreationOptions.LongRunning)

As far as I can tell then, that GrowableArray just gets bigger and bigger, and there are no public APIs that we can use to flush out old item.

@vaind any ideas?

[vaind]

@jamescrosswell any chance you could verify the fix in #3382 ?