Investigate Memory Leak

[awslabs]

Investigate Memory Leak

[jaredcnance]

If there is a leak, it does not appear to be in any Amazon.* types

dotnet gcdump report  20201119_071200_61049.gcdump  | grep Amazon
             64         2  Amazon.CloudWatch.EMF.Config.Configuration  [Amazon.CloudWatch.EMF.dll]
             64         1  System.Func<Amazon.CloudWatch.EMF.Model.DimensionSet,System.Boolean>  [System.Private.CoreLib.dll]
             64         1  System.Func<Amazon.CloudWatch.EMF.Model.DimensionSet,System.Collections.Generic.IEnumerable<System.String>>  [System.Private.CoreLib.dll]
             56         2  Amazon.CloudWatch.EMF.Environment.EnvironmentProvider  [Amazon.CloudWatch.EMF.dll]
             56         2  Amazon.CloudWatch.EMF.Model.MetricDirective  [Amazon.CloudWatch.EMF.dll]
             48         2  Amazon.CloudWatch.EMF.Logger.MetricsLogger  [Amazon.CloudWatch.EMF.dll]
             40         2  Amazon.CloudWatch.EMF.Model.MetaData  [Amazon.CloudWatch.EMF.dll]
             32         2  Amazon.CloudWatch.EMF.Environment.LocalEnvironment  [Amazon.CloudWatch.EMF.dll]
             32         2  Amazon.CloudWatch.EMF.Model.RootNode  [Amazon.CloudWatch.EMF.dll]
             32         2  Amazon.CloudWatch.EMF.Model.MetricsContext  [Amazon.CloudWatch.EMF.dll]
             32         2  System.Collections.Generic.List<Amazon.CloudWatch.EMF.Model.MetricDirective>  [System.Private.CoreLib.dll]
             32         2  System.Collections.Generic.List<Amazon.CloudWatch.EMF.Model.MetricDefinition>  [System.Private.CoreLib.dll]
             32         2  System.Collections.Generic.List<Amazon.CloudWatch.EMF.Model.DimensionSet>  [System.Private.CoreLib.dll]
             24         4  Amazon.CloudWatch.EMF.Model.DimensionSet  [Amazon.CloudWatch.EMF.dll]
             24         4  Amazon.CloudWatch.EMF.Model.DimensionSet[]  [Amazon.CloudWatch.EMF.dll]
             24         3  Amazon.CloudWatch.EMF.Model.MetricDirective[]  [Amazon.CloudWatch.EMF.dll]
             24         2  Amazon.CloudWatch.EMF.Environment.ResourceFetcher  [Amazon.CloudWatch.EMF.dll]
             24         2  Microsoft.Extensions.Logging.Logger<Amazon.CloudWatch.EMF.Environment.ResourceFetcher>  [Microsoft.Extensions.Logging.Abstractions.dll]
             24         2  Microsoft.Extensions.Logging.Logger<Amazon.CloudWatch.EMF.Logger.MetricsLogger>  [Microsoft.Extensions.Logging.Abstractions.dll]
             24         2  Amazon.CloudWatch.EMF.Sink.ConsoleSink  [Amazon.CloudWatch.EMF.dll]
             24         2  Microsoft.Extensions.Logging.Logger<Amazon.CloudWatch.EMF.Sink.ConsoleSink>  [Microsoft.Extensions.Logging.Abstractions.dll]
             24         1  []c  [Amazon.CloudWatch.EMF.Canary.dll]
             24         1  []c  [Amazon.CloudWatch.EMF.dll]
             24         1  Amazon.CloudWatch.EMF.Model.MetricDefinition[]  [Amazon.CloudWatch.EMF.dll]
             24         1  Amazon.CloudWatch.EMF.Model.RootNode[]  [Amazon.CloudWatch.EMF.dll]
             24         1  Amazon.CloudWatch.EMF.Serializer.UnixMillisecondDateTimeConverter  [Amazon.CloudWatch.EMF.dll]

[jaredcnance]

Largest diff by Object Bytes shows an Object[] that has grown by 5KB

dotnet gcdump report  20201119_071200_61049.gcdump  | grep "System.Object\[\]"
          2,368         8  System.Object[] (Bytes > 1K)  [System.Private.CoreLib.dll]
dotnet gcdump report  20201119_071956_61049.gcdump | grep "System.Object\[\]"
          7,376         8  System.Object[] (Bytes > 1K)  [System.Private.CoreLib.dll]

Largest diff by number of objects

Count 1	Count 2	Diff	Count
2,422	3,342	920	CallbackNode
2,418	3,338	920	System.Threading.CancellationTokenSource
2,416	3,336	920	CallbackPartition[]
2,416	3,336	920	CallbackPartition
2,414	3,334	920	System.Threading.SemaphoreSlim
2,414	3,334	920	System.Threading.ThreadStart
2,414	3,334	920	System.Runtime.CompilerServices.StrongBox<System.Boolean>
2,414	3,334	920	Microsoft.Extensions.Logging.Console.AnsiLogConsole
1,932	2,392	460	System.String
1,670	2,130	460	System.Object
1,290	1,750	460	System.Object[]
1,216	1,676	460	System.Diagnostics.ThreadInfo
1,208	1,668	460	System.Threading.Thread
1,207	1,667	460	Slot<Microsoft.Extensions.Logging.Console.LogMessageEntry>[]
1,207	1,667	460	System.Collections.Concurrent.ConcurrentQueueSegment<Microsoft.Extensions.Logging.Console.LogMessageEntry>
1,207	1,667	460	Linked1CancellationTokenSource
1,207	1,667	460	System.Collections.Concurrent.BlockingCollection<Microsoft.Extensions.Logging.Console.LogMessageEntry>
1,207	1,667	460	d__68<Microsoft.Extensions.Logging.Console.LogMessageEntry>
1,207	1,667	460	System.Threading.ThreadHelper
1,207	1,667	460	Microsoft.Extensions.Logging.Console.ConsoleLoggerProcessor
1,309	1,667	358	System.SByte[]
1,207	1,322	115	System.Collections.Concurrent.ConcurrentQueue<Microsoft.Extensions.Logging.Console.LogMessageEntry>

[jaredcnance]

Looking at the canary code, this issue is likely not with the library itself, but instead with the canary. We are creating a logger factory inside a while loop. There should be no need for multiple logger factories and based on the above heap analysis, this is a likely leak candidate. Additionally, Process is disposable and we're not currently disposing it.

Update: Memory leak resolved.