serilog/serilog-sinks-map

A Serilog sink wrapper that dispatches events based on a property value

serilog-sinks-map's Issues

MappedSink.Dispose doesn't dispose underlying sinks

MappedSink.Dispose doesn't dispose underlying sinks.\

Following test fails

[Fact]
public void SinksAreDisposedWithMapSinkDispose()
{
    var a = Some.LogEvent("Hello, {Name}!", "Alice");

    var sink = new DisposeTrackingSink();
    using (var log = new LoggerConfiguration()
        .WriteTo.Map("Name", (name, wt) => wt.Sink(sink))
        .CreateLogger())
    {
        log.Write(a);
    }

    Assert.True(sink.IsDisposed);
}

Hot-reload any Serilog sink sample not working

I was following code from Hot-reload any Serilog sink, the result had a few issues.

I started off with this:

// Program.cs
builder.Host.UseSerilog( ( context, loggerConfig ) =>
{
	loggerConfig.ReadFrom.Configuration( context.Configuration );
} );

And two appsettings.json files that merge together...

appsettings.json

"Serilog": {
	"MinimumLevel": {
		"Default": "Information",
		"Override": {
			"FastEndpoints": "Warning",
			"Microsoft": "Warning",
			"Microsoft.Hosting.Lifetime": "6",
			"System.Net.Http": "6",
			"Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware": "6", 
			"System": "Warning"
		}
	}
}

appsettings.Development.json

"Serilog": {
	"MinimumLevel": {
		"Override": {
			"Microsoft.Hosting.Lifetime": "Information",
			"Nexgen.Middleware.NexgenMiddleware": "Warning"
		}
	},
	"WriteTo": {
		"Console": { 
			"Name": "Console",
			"Args": {
				"outputTemplate": "[{Timestamp:HH:mm:ss} {Level:u3}] {Message:lj} <s:{SourceContext}>{NewLine}{Exception}"
			}
		}
	}
}

Running this as is inside VS Code debugging, I get the output I expect:

[15:23:43 INF] Starting Nexgen web host <s:Program>
[15:23:43 INF] Now listening on: https://localhost:7058 <s:Microsoft.Hosting.Lifetime>
[15:23:43 INF] Application started. Press Ctrl+C to shut down. <s:Microsoft.Hosting.Lifetime>
[15:23:43 INF] Hosting environment: Development <s:Microsoft.Hosting.Lifetime>
[15:23:43 INF] Content root path: C:\BTR\Camelot\Websites\Nexgen <s:Microsoft.Hosting.Lifetime>

However, if all I do is swap out my loggerConfig.ReadFrom.Configuration( context.Configuration ); with the code found in the blog post:

var configVersion = 0L;
builder.Host.UseSerilog( ( context, loggerConfig ) =>
{
	loggerConfig.WriteTo.Map(
		_ => Interlocked.Read( ref configVersion ),
		( _, writeTo ) => writeTo.Logger( sub => sub.ReadFrom.Configuration( context.Configuration ) ),
		sinkMapCountLimit: 5
	);
} );

// snipped

var app = builder.Build();

ChangeToken.OnChange<object?>(
	() => app.Configuration.GetReloadToken(),
	_ => Interlocked.Increment( ref configVersion ),
	null
);

A few things are happening:

1. Upon site start up, I get ~25 log entries in Console. With most of the extra ones coming from Microsoft.AspNetCore.* sources.
2. The "Nexgen.Middleware.NexgenMiddleware": "Warning" setting is not obeyed and Information items are still appearing.
3. After the site is running, if I go and toggle "Nexgen.Middleware.NexgenMiddleware": "Warning" to be Information and back to Warning, I'm still getting Information logs (note that the configVersion is incremented when I update my file).

Finally, maybe I'm over engineering, but I'm thinking I'd like the following features:

1. Ability to configure MinimumLevel and Override via appsettings.json (the final built IConfiguration from all sources) and change at run time (globally, this possible).
2. Ability to configure custom MinimumLevel and Override per sink (doesn't appear supported in Serilog.Settings.Configuration).
3. Ability to add new Override settings at runtime and have them obeyed (not supported in Serilog.Settings.Configuration).

The blog describing this package made it look like a promising direction for what I am hoping for, but maybe I'm misunderstanding/misusing its purpose?

Target .NET 4.5?

I am working on some legacy .NET 4.5 - 4.6.2 projects that cant be upgrade because they are plugins to another system, and although I can use this package as-is, it requires a dependency to NETStandard.Library which outputs a ton of dll's we have to deploy (manually ofcourse ;-)).

Would you be open for targeting .NET 4.5 like some of the other sinks or at least 4.6.2? I could make a pull request if you want 🙂

Limit the number of sinks held open in the map

More scenarios would be supported here if "old" sinks could be automatically garbage collected/removed from the map.

For example, to emulate rolling files, but across rolling directories:

WriteTo.Map(
    le => le.Timestamp.Date,
    (date, lc) => lc.WriteTo.File($"./logs/{date:yyyyMMdd}/log.txt"))

Via: https://stackoverflow.com/questions/44501377/serilog-rollingfile-sinks-create-folder-named-with-date

Can I configure by appsettiongs?

How do I configure the map through the appsettings?

LogFile name for each assembly

Hi, I have a project with a HostBuilder and more different class libraries.
Is it possible for each class library to write the registration into a separate file?
How can I configure appsettings.json or LoggerConfiguration to do it?

Serilog.Settings.Configuration compatibility?

Hey,

Aare you able to use this sink with the Serilog.Settings.Configuration package?

I'm checking out Serilog.Settings.Configuration source and it seems like they only support Action<LoggerConfiguration> and Action<LoggerSinkConfiguration>.

https://github.com/serilog/serilog-settings-configuration/blob/fae623c627496f00fcfab29f721cd49b2bbed989/src/Serilog.Settings.Configuration/Settings/Configuration/ConfigurationSectionArgumentValue.cs#L20

Does anyone have any ideas of how we can get around this?
ie. Conform to the supported actions, ask Serilog.Settings.Configuration to support Action<string, ...> (this seems to edge case for them though)

Not working sub-loggers for properties with specific char - dot

Hi!
This is logger fantastic! Thanks for beautiful logger ;)
Used next following code in asp net core application (of course an abstract example):

return WebHost.CreateDefaultBuilder(webHostArgs)
                .UseContentRoot(pathToContentRoot)
                .UseConfiguration(configuration)
                .UseIISIntegration()
                .UseStartup<Startup>()
                .UseSerilog(ConfigureSerilog())
                .Build();

private static Action<WebHostBuilderContext, LoggerConfiguration> ConfigureSerilog()
        {
	        return (hostingContext, loggerConfiguration) =>
	               {
		               var config = hostingContext.Configuration.Get<ApplicationConfig>();
		               var template = config.SerilogAdditionalParameters.OutputTemplate;
		               var limitDays = config.SerilogAdditionalParameters.RetainedFileCountLimit;
		               var basePath = config.SerilogAdditionalParameters.BasePath;

		               loggerConfiguration
			               .ReadFrom.Configuration(hostingContext.Configuration)
			               .Enrich.FromLogContext()
			               .Enrich.WithThreadId()
			               .WriteTo.File(Path.Combine(basePath, ".log"),
			                             outputTemplate: template,
			                             retainedFileCountLimit: limitDays,
			                             rollingInterval: RollingInterval.Day)
			               .WriteTo.File(Path.Combine(basePath, "errors", ".log"),
			                             outputTemplate: template,
			                             retainedFileCountLimit: limitDays,
			                             restrictedToMinimumLevel: LogEventLevel.Error,
			                             rollingInterval: RollingInterval.Day)
			               .WriteTo.Logger(x =>
			                               {
				                               x.Filter.ByIncludingOnly(Matching.WithProperty("A.SomeProperty1"))
				                                .WriteTo.Map("A.SomeProperty1", "[Null]",
				                                             (some1, wtp) =>
				                                             {
					                                             wtp.Logger(t =>
					                                                        {
						                                                        t.Filter.ByIncludingOnly(Matching.WithProperty("A.SomeProperty2"))
						                                                         .WriteTo.Map("A.SomeProperty2", "[Null]",
						                                                                      (some2, wtu) =>
						                                                                      {
							                                                                      wtu.File(Path.Combine(basePath, "by-SomeProperty1", some1, some2, ".log"),
							                                                                               outputTemplate: template,
							                                                                               retainedFileCountLimit: limitDays, rollingInterval: RollingInterval.Day);
						                                                                      });
					                                                        });
				                                             });
			                               });
	               };
        }

This setup need for next structure :
logs/
errors/log
main-log
by-SomeProperty1/SomeProperty1/SomeProperty2/log

In the code, the logger is used as follows:

using (logger.BeginScope("Test:{A.SomeProperty1} for {A.SomeProperty2}", "testing", "serilog"))
{
  logger.Debug("test log");
}

This is code work fine on versions 1.0.0-dev-00012/00017
If add write to file before filter for "A.SomeProperty2", than log is writing only for "A.SomeProperty1":

.WriteTo.Map("A.SomeProperty1", "[Null]",
        (some1, wtp) =>
		{
		    wtu.File(Path.Combine(basePath, "by-SomeProperty1", some1, ".log"),		
			outputTemplate: template,
			retainedFileCountLimit: limitDays, rollingInterval: RollingInterval.Day);
																 
			wtp.Logger(t =>
			{
			    t.Filter.ByIncludingOnly(Matching.WithProperty("A.SomeProperty2"))
				    .WriteTo.Map("A.SomeProperty2", "[Null]",
					        (some2, wtu) =>
						    {
							    wtu.File(Path.Combine(basePath, "by-SomeProperty1", some1, some2, ".log"),
							                outputTemplate: template,
							                retainedFileCountLimit: limitDays, rollingInterval: RollingInterval.Day);
						    });
			});
});

The below property "A.SomeProperty2" in next Map() ignored.

Why behavion was changed after version 1.0.0-dev-00012/00017? Maybe this is broken changes?

Getting started example not working

Hello,

I'm trying to re-create the example in the README.md

///Install-Package Serilog.Sinks.Map -Pre
Log.Logger = new LoggerConfiguration()
    .WriteTo.Map("Name", "Other", (name, wt) => wt.File($"./logs/log-{name}.txt"))
    .CreateLogger();

It gives error

Error CS1061 'LoggerSinkConfiguration' does not contain a definition for 'File' and no accessible extension method 'File' accepting a first argument of type 'LoggerSinkConfiguration' could be found (are you missing a using directive or an assembly reference?)

Nuget installed the following
Successfully installed 'Serilog 2.8.0' to netcore ConsoleApp
Successfully installed 'Serilog.Sinks.Map 1.0.1' to netcore ConsoleApp
Successfully installed 'System.Collections.NonGeneric 4.3.0' to netcore ConsoleApp
Executing nuget actions took 630.19 ms

How do I get this to work? The tests also do not appear to have this same example.

Update: It also requires a further dependency on Serilog.Sinks.File 4.1.0

Question: Is there a reason why this package is still Preview?

Could anybody have an update on when we can expect 1.0 released?

Memory leak when 'sinkMapCountLimit' is set in WriteTo.Map

Description
When sinkMapCountLimit is set lower than the possible amount of sinks being created, evicted sinks are still being referenced by LoggerConfiguration's _logEventSinks. The GC is unable to reclaim the resources of those sinks.

Reproduction

private const int FILE_COUNT = 5;

private static readonly Random _random = new Random();

private static readonly string _path = System.IO.Path.Combine(Environment.GetFolderPath(
	Environment.SpecialFolder.CommonApplicationData), "Serilog-Memleak-Test");

static async Task Main(string[] args)
{
	Log.Logger = new LoggerConfiguration()
		.WriteTo.Map("Number", "",
			     (number, wt) => wt.File($"{_path}/logs-{number}.txt"),
			     sinkMapCountLimit: FILE_COUNT - 1)
		.CreateLogger();

	do
	{
		while (!Console.KeyAvailable)
		{
			Log.Information("Test log entry for {Number}", _random.Next(0, FILE_COUNT).ToString());
			await Task.Delay(100).ConfigureAwait(false);
		}
	} while (Console.ReadKey(true).Key != ConsoleKey.Escape);
}

Expected behavior
When a wrapped sink is evicted from MappedSink, its resources are reclaimed by the GC.

Relevant package, tooling and runtime versions

First Discovered	Reproduced
.NET Framework 4.6.1	.NET Core 3.1
Serilog v2.9.0	Serilog v2.10.0
Sinks.Map v1.0.0	Sinks.Map v1.0.1
Sink.File v4.1.0	Sinks.File v4.1.0

System.OutOfMemoryException with Serilog.Sinks.Map

Description
The memory keeps growing until the memory is exhausted,memory is not released causing memory leaks.

Reproduction
see demo

Expected behavior
System.OutOfMemoryException

feat: Configure Sink via Serilog.Settings.Configuration

I just had another requirement where Map made a lot of sense, however this project is using appsettings to configure the logger.

Since I didn't want to have parts of the logger configured in code and other parts configured in appsettings, I tried a few different json formats to make the configuration but I assume the parser doesn't currently understand the Action<T1, T2> used on the default Map overload.

I was able to work around it slightly using a custom extension method that performed some of the logic but it ended up with a lot of code duplication:

        public static LoggerConfiguration MapToFile(
            this LoggerSinkConfiguration loggerSinkConfiguration,
            string keyPropertyName,
            string pathFormat,
            bool rollOnFileSizeLimit,
            int? retainedFileCountLimit,
            long? fileSizeLimitBytes)
        {
            return loggerSinkConfiguration.Map(
                keyPropertyName,
                (key, config) => config.File(
                    string.Format(pathFormat, key),
                    rollOnFileSizeLimit: rollOnFileSizeLimit,
                    retainedFileCountLimit: retainedFileCountLimit,
                    fileSizeLimitBytes: fileSizeLimitBytes));
        }

And then using this on appsettings:

    "WriteTo": [
      {
        "Name": "MapToFile",
        "Args": {
          "KeyPropertyName": "Operation",
          "FileSizeLimitBytes": 10485760,
          "PathFormat": "/Logs/WebApps/MyApp/{0}.log",
          "RetainedFileCountLimit": 100,
          "RollOnFileSizeLimit": true
        }
      }
    ]

This is obviously less than ideal, as I had to duplicate a bunch of arguments used in File.

Is it currently possible to configure the standard Map using json (in which case I probably missed something), or can we add support for it?

Does this support SQL Server?

feat: Customizable disposal policy wrt concurrent sinks limit

The current implementation only allows users to control cleanup by setting the maximum amount of concurrent sinks in the map.

In my scenario, I'd like to have more control over this behavior, perhaps by implementing policies like:

• Dispose the sink after a certain amount of time after it's created
• Dispose the sink after a certain amount of time after being inactive (no events received)
• Dispose the sink after a certain log event is received (arbitrary predicate on a given event)

Perhaps it would be possible to expose an extension point by introducing an abstraction, like a SinkPool, then allow consumers to implement their own versions with custom creation and disposal semantics.

Having this feature would allow me to have better control with my unbounded sinks. Today, I can't leave this option null because it would create a memory leak in the application, but controlling the number of open sinks doesn't match our business needs.

docs: Add cleanup policy overview

The main documentation for this sink does not properly explain how the sinks are cleaned up once the limit is reached. I spent some time looking at the code to see how it was disposing the sinks because this was important to my application from a performance standpoint.

Can we add more information related to the cleanup process:

• Is the process deterministic?
• In what order are sinks disposed once the limit is reached?
• Does the cleanup happen asynchronously, or synchronously?
• When exactly cleanup attempts are made? Before or after the next event is logged?

Having this information upfront would've saved me a few hours of testing and investigation, so I believe this could prove useful for other folks as well.

Map ignores retainedFileCountLimit

Hi,
I have a problem regarding limiting the number of log files.

I have retainedFileCountLimit 2, but map seems to ignore this rule. Unfortunately it creates more than 2 files.

 .WriteTo.Async(a =>
            {
                a.Map<string>(
                    e => DateTime.Now.ToString("dd_MM_yyyy"),
                    (v, wt) => wt.File($"{AppDomain.CurrentDomain.BaseDirectory}/logs/log_{v}.txt",
                                       retainedFileCountLimit: 2,
                                       outputTemplate: "[{Timestamp:dd/MM/yyyy HH:mm:ss} {Level:u3}] {Message:lj}{NewLine}{Exception}",
                                       buffered: true,
                                       fileSizeLimitBytes: 105000000),
                    sinkMapCountLimit: 1);
            }, bufferSize: 120000)

it should only have been 2 files, but I see 3 files:

log_15_05_2021.txt
log_16_05_2021.txt
log_17_05_2021.txt

Missing documentation on json configuration

Is it possible to configure sink maps in a appsettings.json file?
How does the syntax for that look?

Combined with dynamic creating SQL tables inside a transaction delivers EF distributed transaction error

• When you create dynamic tables in the database based on a log tag
• And a table is not yet existing
• And a TRANSACTION is started
• And you then log inside that transaction

You will run into "distributed transactions not supported"

E.g.:

.WriteTo.Map("Tag", configuration["Serilog:Sql:DefaultTable"], (name, wt) => wt.MSSqlServer(
connectionString: configuration.GetConnectionString(configuration["Serilog:Sql:ConnectionStringName"]),
sinkOptions: new Serilog.Sinks.MSSqlServer.MSSqlServerSinkOptions
{
TableName = configuration["Serilog:Sql:Table"] + name,
SchemaName = configuration["Serilog:Sql:Schema"],
BatchPeriod = new TimeSpan(0, 0, 5),
AutoCreateSqlTable = true,
BatchPostingLimit = 50,
EagerlyEmitFirstEvent = true
},
columnOptions: SeriLogColumns()
))

A workaround is logical: create all possible log tables that are touched during a transaction before-hand.
But this will require to mimic the exact same schema as the columns definition used in the Serilog Sql Server configuration which does it automagically. So the easiest is to check if a table exists, if not , write an error to it, then delete the error, so the table is created assuming the highest level someone would set is error.

I want to map a reference type，but it not working as set value

this is my code

        static void Main(string[] args)
        {
           var defaultModel = new MyModel
           {
               DateTime = DateTime.Now,
               BusinessType = "Normal"
           };

           var myLogger = new LoggerConfiguration()
              .WriteTo.Console()
              .WriteTo.Map("configInfo", defaultModel, (config, wt) =>
              {
                  Console.WriteLine(config.BusinessType);
                  Console.WriteLine(config.DateTime);
                  wt.File($"./logs/{config.BusinessType}/{config.DateTime.ToString("yyyy-MM")}/{config.DateTime.ToString("yyyy-MM-dd")}-log.txt");
              }).CreateLogger();

            myLogger.ForContext("configInfo", new MyModel
            {
                DateTime = DateTime.Now,
                BusinessType = "Pay"
            });

            myLogger.Information("Hello World1! ");

            myLogger.Warning("Hello World2! {configInfo}", new MyModel
            {
                DateTime = DateTime.Now,
                BusinessType = "Flow"
            });

            myLogger.Information("Hello World3! ");

            Console.ReadKey();
        }

this is the result

so i want to dynamic set reference type param to log different directory file
like ..\logs\Flow\2022-07\2022-07-15-log.txt and ..\logs\Pay\2022-07\2022-07-15-log.txt