Azure Functions Telemetry

The Application Insights integration for Azure Functions v3 and v4 suffers from a few quirks that can lead to a huge Application Insights bill:

Telemetry processors are not supported, preventing developers from discarding telemetry items
Each Function execution records a trace when starting and on completion
Exceptions are logged twice for the HTTP binding
Exceptions are logged three times for the Service Bus binding

The next issue has no impact on the cost of Application Insights but is related to the development experience. TelemetryConfiguration is not registered in the Inversion Of Control container when the Application Insights connection string is not set. Emitting a custom metric requires to inject the TelemetryConfiguration. Running locally without having configured the Application Insights connection string will then result in an exception.

The last issue is not related to Application Insights but also negatively affect developers' productivity. The custom Console logger provider used by the Azure Functions runtime does not include the stack trace when displaying an exception (for the HTTP binding at least).

If you are not familiar with some of the more advanced features of Application Insights, I suggest you go through the below references before reading the rest of this document:

In this repository I have tried to address all the quirks listed above.

🚨 The customisation is experimental. There may be undesired side effects around performance and missing telemetry. If you are experiencing issues such as missing telemetry, no exception being recorded when something does not work as expected I recommend disabling the custom integration and reproducing the problem without it.

Using the Application Insights customisation

The customisation supports both v3 and v4 runtime.

dotnet add package AzureFunctions.Better.ApplicationInsights

For the most basic integration, you need to provide:

{ApplicationName} used to set Application Insights' Cloud role name
{TypeFromEntryAssembly} typically would be typeof(Startup). I read the Assembly Informational Version of the entry assembly to set Application Insights' Application version (I use unknown as a fallback)

In your Startup class add the below snippet:

var appInsightsOptions = new CustomApplicationInsightsOptionsBuilder(
        "{ApplicationName}",
        {TypeFromEntryAssembly})
    .Build();

builder.Services
    .AddCustomApplicationInsights(appInsightsOptions)
    .AddCustomConsoleLogging();

What do I get?

Discarding Function execution traces

This is implemented by FunctionExecutionTracesFilter and always enabled.

Discarding duplicate exceptions

This is implemented by DuplicateExceptionsFilter and always enabled.

Discarding health requests

This is enabled by calling WithHealthRequestFilter and supplying the Function name (the argument provided to the FunctionNameAttribute). The telemetry processor used is HealthRequestFilter.

Better Service Bus binding "request"

The URL and response code are not being set on the service bus triggered "requests". The ServiceBusRequestInitializer will do this for you.

URL: I use the Function name
Response code: 200 in case of success, 500 in case of failure

The ServiceBusRequestInitializer is always enabled.

Discarding Service Bus trigger traces

This is recommended on high-volume services. This is done by calling WithServiceBusTriggerFilter. The telemetry processor used is ServiceBusTriggerFilter.

Replacing the Console logging provider

This is done by calling AddCustomConsoleLogging. You will then consistently get stack traces in the console.

Registering telemetry initializers

📝 The built-in integration supports telemetry initializers. The custom integration supports registering telemetry initializers in the same way than the built-in integration does.

Telemetry initializers can either be registered using TImplementation:

builder.Services.AddSingleton<ITelemetryInitializer, YourInitializer>();

Or an instance of the telemetry initializer:

// Use:
builder.Services.AddSingleton<ITelemetryInitializer>(new YourOtherInitializer("NiceValue"));
// Do not use, otherwise your telemetry initializer will not be called:
builder.Services.AddSingleton(new YourOtherInitializer("NiceValue"));

You can add as many telemetry initializers as you want.

Telemetry processors support

📝 The built-in integration does not support telemetry processors. I have added support so that you can use the same extension method than when registering a telemetry processor in ASP.NET Core:

builder.Services.AddApplicationInsightsTelemetryProcessor<YourTelemetryProcessor>();

You can add as many telemetry processors as you want.

Demo

A demo demonstrates all the customisation's features.

Q&A

Why so much code to support telemetry processors?

There is an opened GitHub issue about the lack of telemetry processors support in Azure Functions. The thread supplies a workaround to enable telemetry processors, but the telemetry processors added in this fashion will not be called for request telemetry items.

Appendix

Software versions

The latest version of the Azure Functions Core Tools I have been using is 4.0.4590.

NuGet packages:

Microsoft.NET.Sdk.Functions:
- v3: 3.1.1 (added automatically when creating the Function, updated later)
- v4: 4.1.0 (added automatically when creating the Function, updated later)
Microsoft.Azure.Functions.Extensions: 1.1.0 (added manually following Use dependency injection in .NET Azure Functions)
Microsoft.Extensions.DependencyInjection (added manually following Use dependency injection in .NET Azure Functions, updated later):
- v3: 3.1.29
- v4: 6.0.0
Microsoft.Azure.WebJobs.Logging.ApplicationInsights: 3.0.33 (added manually following Log custom telemetry in C# Azure Functions)

Supporting telemetry processors

The code in AddCustomApplicationInsights retrieves the configured built-in telemetry processors, adds them to a new telemetry processor chain and builds the chain. This gives me the opportunity to add our own processors to the chain.

The first built-in processor in the chain is OperationFilteringTelemetryProcessor, this processor discards all the dependencies considered internal to the Azure Functions runtime (such as access to blob storage for the distributed lock and the calls to Azure Service Bus for the Service Bus binding).

One of the side-effects of the approach I am using is that the Azure Functions runtime will reference the initial instance of OperationFilteringTelemetryProcessor and will call it directly when tracking requests manually. Normally the OperationFilteringTelemetryProcessor instance points to the second processor in the chain (QuickPulseTelemetryProcessor). One way for our processors to be called is to point the existing OperationFilteringTelemetryProcessor instance to our first processor and point our last processor to QuickPulseTelemetryProcessor. This is done through some pretty dodgy untested code, but it works ™️.

As I did not manage to cover my customisation with unit tests, I wrote integration tests instead.

fazer01 / azure-functions-telemetry Goto Github PK