Giter Site home page Giter Site logo

aws / aws-dotnet-extensions-configuration Goto Github PK

View Code? Open in Web Editor NEW
173.0 173.0 56.0 282 KB

This repository hosts various libraries that help developers configure .NET applications using AWS services.

Home Page: https://aws.amazon.com/developer/language/net/

License: Apache License 2.0

C# 100.00%

aws-dotnet-extensions-configuration's Introduction

.NET on AWS Banner

AWS .NET Configuration Extension for Systems Manager

nuget

Amazon.Extensions.Configuration.SystemsManager simplifies using AWS SSM's Parameter Store and AppConfig as a source for configuration information for .NET Core applications. This project was contributed by @KenHundley and @MichalGorski.

The library introduces the following dependencies:

Getting Started

Follow the examples below to see how the library can be integrated into your application. This extension adheres to the same practices and conventions of Configuration in ASP.NET Core.

ASP.NET Core Example

One of the common use cases for this library is to pull configuration from Parameter Store. You can easily add this functionality by adding 1 line of code.

public class Program
{
    public static void Main(string[] args)
    {
        CreateWebHostBuilder(args).Build().Run();
    }

    public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
        WebHost.CreateDefaultBuilder(args)
            .ConfigureAppConfiguration(builder =>
            {
                builder.AddSystemsManager("/my-application/");
            })
            .UseStartup<Startup>();
}

It is also possible to load AWS Secrets Manager secrets from Parameter Store parameters. When retrieving a Secrets Manager secret from Parameter Store, the name must begin with the following reserved path: /aws/reference/secretsmanager/{Secret-Id}. Below example demonstrates this use case:

public class Program
{
    public static void Main(string[] args)
    {
        CreateWebHostBuilder(args).Build().Run();
    }

    public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
        WebHost.CreateDefaultBuilder(args)
            .ConfigureAppConfiguration(builder =>
            {
                builder.AddSystemsManager("/aws/reference/secretsmanager/SomeSecret");
            })
            .UseStartup<Startup>();
}

For loading secrets, the library will use JsonParameterProcessor to load Key/Value pairs stored in the secret. These Key/Value pairs could be retrieved from the ConfigurationManager object. For more details, kindly refer Referencing AWS Secrets Manager secrets from Parameter Store parameters.

Another possibility is to pull configuration from AppConfig. You can easily add this functionality by adding 1 line of code.

public class Program
{
    public static void Main(string[] args)
    {
        CreateWebHostBuilder(args).Build().Run();
    }

    public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
        WebHost.CreateDefaultBuilder(args)
            .ConfigureAppConfiguration(builder =>
            {
                builder.AddAppConfig("AppConfigApplicationId", "AppConfigEnvironmentId", "AppConfigConfigurationProfileId", TimeSpan.FromSeconds(20));
            })
            .UseStartup<Startup>();
}

HostBuilder Example

Microsoft introduced .NET Generic Host to de-couple HTTP pipeline from the Web Host API. The Generic Host library allows you to write non-HTTP services using configuration, dependency injection, and logging features. The sample code below shows you how to use the AWS .NET Configuration Extension library:

namespace HostBuilderExample
{
    public static async Task Main(string[] args)
    {
        var host = new HostBuilder()
            .ConfigureAppConfiguration((hostingContext, config) =>
            {
                config.AddSystemsManager("/my-application/");
                config.AddAppConfig("AppConfigApplicationId", "AppConfigEnvironmentId", "AppConfigConfigurationProfileId", TimeSpan.FromSeconds(20));
            })
            .ConfigureServices((sc) => { ... })
            .Build();

        await host.RunAsync();
    }
}

AWS Lambda Example

For improved performance with AppConfig and Lambda it is recommended to use the AddAppConfigUsingLambdaExtension method and deploy the Lambda function with the AWS AppConfig Lambda extension. More information including the AppConfig Lambda extension layer arn can be found in the AWS AppConfig user guide.

var configurations = new ConfigurationBuilder()
                        .AddSystemsManager("/my-application/")
                        .AddAppConfigUsingLambdaExtension("AppConfigApplicationId", "AppConfigEnvironmentId", "AppConfigConfigurationProfileId")
                        .Build();

Config reloading

The reloadAfter parameter on AddSystemsManager() and AddAppConfig() enables automatic reloading of configuration data from Parameter Store or AppConfig as a background task. When using AddAppConfigUsingLambdaExtension reload is automatically configured.

Config reloading in AWS Lambda

In AWS Lambda, background tasks are paused after processing the AWS Lambda event. This could disrupt the provider from retrieving the latest configuration data from Parameter Store or AWS AppConfig. To ensure the reload is performed within the AWS Lambda event, we recommend calling the extension method WaitForSystemsManagerReloadToComplete from the IConfiguration object in the beginning of your AWS Lambda function handler. This method will immediately return unless a reload is currently being performed.

Remember to build IConfiguration in the AWS Lambda constructor! It is the only way to cache the configuration and reload it using reloadAfter parameter.

public class SampleLambda
{
    private readonly IConfiguration _configurations;
    
    public SampleLambda()
    {
        _configurations = new ConfigurationBuilder()
                            .AddSystemsManager("/my-application/")
                            .AddAppConfigForLambda("AppConfigApplicationId", "AppConfigEnvironmentId", "AppConfigConfigurationProfileId", TimeSpan.FromSeconds(20))
                            .Build();
    }

    protected void Invoke()
    {
        _configurations.WaitForSystemsManagerReloadToComplete(TimeSpan.FromSeconds(2));
    }
}

Samples

Custom ParameterProcessor Sample

Example of using a custom IParameterProcessor which provides a way to store and retrieve null values. Since AWS Parameter Store params are string literals, there is no way to store a null value by default.

namespace CustomParameterProcessorExample
{
    public class CustomParameterProcessor : DefaultParameterProcessor
    {
        const string NULL_STRING_LITERAL = "NULL";
        
        public override string GetValue(Parameter parameter, string path)
        {
            string value = base.GetValue(parameter, path);
            return value == NULL_STRING_LITERAL ? null : value;
        }
    }
    
    public class Program
    {
        public static void Main(string[] args)
        {
            CreateWebHostBuilder(args).Build().Run();
        }

        public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
            WebHost.CreateDefaultBuilder(args)
                .ConfigureAppConfiguration(builder =>
                {
                    builder.AddSystemsManager(config => {
                        config.Path = "/my-application/";
                        config.ParameterProcessor = new CustomParameterProcessor();
                    });
                })
                .UseStartup<Startup>();
    }
}

For more complete examples, take a look at sample projects available in samples directory.

Configuring Systems Manager Client

This extension is using AWSSDK.Extensions.NETCore.Setup to get AWSOptions from Configuration object and create AWS Systems Manager Client. You can edit and override the configuration by adding AWSOptions to your configuration providers such as appsettings.Development.json. Below is an example of a configuration provider:

{
  ...

  "AWS": {
    "Profile": "default",
    "Region": "us-east-1",
    "ResignRetries": true
  }

  ...
}

For more information and other configurable options please refer to Configuring the AWS SDK for .NET with .NET Core.

Permissions

Parameter Store

The AWS credentials used must have access to the ssm:GetParameters service operation from AWS System Manager. Below is an example IAM policy for this action.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SSMPermissionStatement",
            "Effect": "Allow",
            "Action": "ssm:GetParameters",
            "Resource": "arn:aws:ssm:${Region}:${Account}:parameter/${ParameterNamePrefix}*"
        }
    ]
}

The above policy gives user access to get and use parameters which begin with the specified prefix.

For more details, refer Restricting access to Systems Manager parameters using IAM policies.

Additionally, for referencing secrets from AWS Secrets Manager from Paramater Store parameters, AWS credentials used must have permissions to access the secret.

AppConfig

If the application reads configuration values from AWS Systems Manager AppConfig, the AWS credentials used must have access to appconfig:StartConfigurationSession and appconfig:GetLatestConfiguration service operations. Below is an example IAM policy for this action.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "appconfig:StartConfigurationSession",
        "appconfig:GetLatestConfiguration",
      ],
      "Resource": "arn:${Partition}:appconfig:${Region}:${Account}:application/${ApplicationId}/environment/${EnvironmentId}/configuration/${ConfigurationProfileId}"
    }
  ]
}

For more details, refer Configuring permissions for AWS AppConfig and Actions, resources, and condition keys for AWS AppConfig.

Getting Help

We use the GitHub issues for tracking bugs and feature requests and have limited bandwidth to address them.

If you think you may have found a bug, please open an issue.

Contributing

We welcome community contributions and pull requests. See CONTRIBUTING.md for information on how to set up a development environment and submit code.

Additional Resources

AWS .NET GitHub Home Page
GitHub home for .NET development on AWS. You'll find libraries, tools, and resources to help you build .NET applications and services on AWS.

AWS Developer Center - Explore .NET on AWS
Find all the .NET code samples, step-by-step guides, videos, blog content, tools, and information about live events that you need in one place.

AWS Developer Blog - .NET
Come see what .NET developers at AWS are up to! Learn about new .NET software announcements, guides, and how-to's.

@dotnetonaws Follow us on twitter!

License

Libraries in this repository are licensed under the Apache 2.0 License.

See LICENSE and NOTICE for more information.

aws-dotnet-extensions-configuration's People

Contributors

96malhar avatar adilson-tkrw avatar artemmelnychenko avatar ashishdhingra avatar ashovlin avatar bashatah avatar connorjbishop avatar doug-ferris-mondo avatar joshongithub avatar kellertk avatar kenhundley avatar mgorski-mg avatar ngl321 avatar normj avatar peterrsongg avatar philasmar avatar somayab avatar srutig avatar sstevenkang avatar transmax-open-source avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aws-dotnet-extensions-configuration's Issues

Resource Temporarily Unavailable

We've been using this library for a few months without any issues in aws govcloud. We are planning to move one of our environments out of GovCloud into us-east-2 and we're running into the following exception while attempting to access our configuration.

Unhandled Exception: System.Net.Http.HttpRequestException: Resource temporarily unavailable ---> System.Net.Sockets.SocketException: Resource temporarily unavailable
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask1.get_Result() at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Threading.Tasks.ValueTask1.get_Result()
at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask1 creationTask) at System.Threading.Tasks.ValueTask1.get_Result()
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken) at Amazon.Runtime.Internal.HttpHandler1.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Extensions.Configuration.SystemsManager.Internal.SystemsManagerProcessor.GetParametersByPathAsync()
at Amazon.Extensions.Configuration.SystemsManager.Internal.SystemsManagerProcessor.GetDataAsync()
at Amazon.Extensions.Configuration.SystemsManager.SystemsManagerConfigurationProvider.LoadAsync(Boolean reload)
at Amazon.Extensions.Configuration.SystemsManager.SystemsManagerConfigurationProvider.Load()
at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
at CoverMe.Notes.HostedApi.Program.Main(String[] args) in /src/CoverMe.Notes.HostedApi/Program.cs:line 18

I've added an appropriate IAM role to the ec2 instance to access SSM. I am able to access SSM with the aws cli on the ec2 instance.

We're connecting to SSM using the code defined below

return builder.ConfigureAppConfiguration((wb, cb) =>
{
    var env = wb.HostingEnvironment;
    cb
        .SetBasePath(env.ContentRootPath)
        .AddJsonFile("appsettings.json", optional: true,
                     reloadOnChange: true)
        .AddJsonFile($"appsettings.{env.EnvironmentName}.json",
                     optional: true, reloadOnChange: true)
        .AddEnvironmentVariables();

    string awsPath = Environment.GetEnvironmentVariable("AWS_CONFIG_PATH");

    if (!string.IsNullOrWhiteSpace(awsPath))
    {
        cb.AddSystemsManager(awsPath);
    }
});

Am I missing something really basic here? I just can't figure out why this is not working.

Unable to retrieve ConnectionStrings

The Question

I have followed this guide.

AWS Systems Manager parameters are
/philinq/ConnectionStrings/OAuthConnection
/philinq/ConnectionStrings/PhilinqConnection

In my Program.cs I have the following

Host.CreateDefaultBuilder(args)
.UseSerilog()
.ConfigureAppConfiguration((context, builder) =>
{
   builder.AddSystemsManager("/philinq");
})
 .ConfigureWebHostDefaults(webBuilder =>
{
   webBuilder.UseStartup<Startup>();
});

As far as credentials go I believe it is supposed to use the ones have configured for the aws cli. In the aws cli I am able to access both the parameters above.

In my Startup.cs I have the normal logic to get the ConnectionString from Configuration

var philinqConnectionString = Configuration.GetConnectionString("PhilinqConnection");
var oauthConnectionString = Configuration.GetConnectionString("OAuthConnection");

Since GetConnectionString is shorthand for GetSection("ConnectionStrings")[name]

However I keep getting null for both connection strings. Am I missing something?

When I debug Configuration I do see the provider
Screen Shot 2020-09-20 at 2 32 01 PM

Environment

  • Build Version: 1.2.0
  • OS Info: macOS Catalina
  • Build Environment: VSCode + Terminal dotnet
  • Targeted .NET Platform: .Net Core 3.1

This is a โ“ general question

How do you use this in Lamba only?

This is nice for ASP.NET Core but how do I actually use this in a simple lambda that needs to access it's connection string, for example?

Add filter by Parameter Label

Hi,
I couldn't find a way to use labels to filter parameters, with the Path, a Label could be used to retrieve a specific labeled version of parameters set, for example a set of parameters labeled with a specific release version.

Suggested change:

  1. Add Label property to SystemsManagerConfigurationSource .
  2. Use the Label if it is not null as a ParameterFiler in GetParametersByPathRequest
    Something like this
    var response = await client.GetParametersByPathAsync(new GetParametersByPathRequest { Path = Source.Path, Recursive = true, WithDecryption = true, NextToken = nextToken, ParameterFilters = CreateLabelFilter(Source.Label) }).ConfigureAwait(false);

public static List<ParameterStringFilter> CreateLabelFilter(string label) { if (string.IsNullOrWhiteSpace(label)) return null; return new List<ParameterStringFilter> { new ParameterStringFilter { Key="Label", Option="Equals", Values = new List<string>{label} }}; }

Usage:
config.AddSystemsManager(configurationSource=> { configurationSource.Path = "mypath"; configurationSource.Optional = false; configurationSource.ReloadAfter = TimeSpan.FromMinutes(10); configurationSource.Label="release_v1.2.3" });

Update
Maybe it's more helpful to add ParameterFilters property to SystemsManagerConfigurationSource ParameterFilters would of type List<ParameterStringFilter> , to keep it more open to the user?

  • if any of these suggested changes is accepted I can create a PR.

ReloadAfter doesn't work well in Lambda (2)

I'd like to reopen this issue: #17

I've been testing with this again and it still doesn't work. I've downloaded the code from the SDK and put tracing in, but the function doesn't reload it's configuration after the indicated time. I've downloaded the code and put tracing in and could confirm that configuration isn't reloaded as expected. The freeze of the lambda function means the reload timer (cancellation token) freezes as well. Eventually the config is reloaded, but it isn't predictable when.
I wouldn't mind waiting synchronously for the configuration to reload, if that's easier to build.

ReloadAfter doesn't work in Lambda

The ReloadAfter functionality in SystemsManagerConfigurationProvider doesn't work if the code runs in AWS Lambda. Parameter values remain the same during the lifetime of the Lambda instance

See also #17

This is a Bug not a Feature-Request

Add logging

Add some Verbose/Debug level logging to the provider. Probably would have to provide an ILogger via the extension method.

Examples: Maybe the key/values pairs being added to the combined Configuration, TTL? Interactions around parameter store.

Use IAM Role credentials instead of profile

Describe the Feature

When the application is deployed to the server, it is good security practice to use IAM Role when deployed to EC2 instance, rather than user profile. I assume that this is not possible.

Is your Feature Request related to a problem?

Keeping Credentials in the profile on the server is bad security approach, and AWS recommends against it. Additionally, the keys need to updated when they are rotated.

Proposed Solution

Probably, extend AWSOptions class to specify that the permissions are derived from IAM Role, and not Porfile

Describe alternatives you've considered

I used AWSSDK.SimpleSystemsManagement package instead; but it didn't work for me. I needed Role credentials to retrieve database credentials from SSM Parameter Store. I was able to accomplish the result in DBContext:OnConfiguring, but I need to do that in the Startup:ConfigureServices while setting up Dependency Injection

Additional Context

At this time we are going to store DB credentials in appsettings.json, which is not a good security practice.

Environment

ASP.NET Core application


This is a ๐Ÿš€ Feature Request

Configuring permissions to access the Property Store

I have been battling this error and I can't seem to figure it out. My dotnet core application deployed on an EC2 instance just crashes in a loop.

So I tried to launch it directly to get some data and when I run

/usr/bin/dotnet /home/ubuntu/aspnetcoreapp/MySite.dll

I get this error

Unhandled Exception: Amazon.SimpleSystemsManagement.AmazonSimpleSystemsManagementException: User: arn:aws:sts::012345678901:assumed-role/CodeStarWorker-mysite-WebApp/i-0123456789badcafe is not authorized to perform: ssm:GetParametersByPath on resource: arn:aws:ssm:us-east-1:012345678901:parameter/PropertyStoreRootName ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception)
   at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Extensions.Configuration.SystemsManager.Internal.SystemsManagerProcessor.GetParametersByPathAsync()

I went in IAM and tried to add the following policy on the aws-codestar-service-role role, but that didn't work.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ssm:DescribeParameters",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameterHistory",
                "ssm:GetParametersByPath",
                "ssm:GetParameters",
                "ssm:GetParameter"
            ],
            "Resource": "arn:aws:ssm:us-east-1:012345678901:parameter/*"
        }
    ]
}

That didn't work either.

My code looks pretty much like the samples...

public class Program
{
    public static void Main(string[] args)
    {
        CreateWebHostBuilder(args).Build().Run();
    }

    private static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
        WebHost.CreateDefaultBuilder(args)
            .ConfigureAppConfiguration(builder =>
            {
                if (Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") != "Development")
                {
                    builder.AddSystemsManager(source =>
                    {
                        source.Path = "/MySite";
                        source.ReloadAfter = TimeSpan.FromMinutes(5);
                    }); 
                }
            })
            .UseStartup<Startup>();
}

012345678901 is what I understand to be my unique user ID
0123456789badcafe is a bunch of hexa that doesn't seem to match anything I know of, but I anonymized it just in case.

What I want to store in the Property Store are the AWS key and secret, but I feel like I need those already in the app to retrieve properties.

Am I trying to drive in screws with a hammer here? Or is there something else I'm missing entirely?

Allowing Arrays through SecretsManager

The default Json conflguration parser (src/Amazon.Extensions.Configuration.SystemsManager/Internal/JsonConfigurationParser.cs) expects the root item of the SecretsManager value to be an object.

By changing lines 61-63 from

var jsonConfig = JObject.Load(_reader);	
VisitJObject(jsonConfig);

to

var jsonConfig = JToken.Load(_reader);
VisitToken(jsonConfig);

I was able to use a custom build of this project to load both arrays or objects at the root level.

is this a change that would be welcomed through a PR or are there concerns with this change that I am not aware of?

Amazon.Extensions.Configuration.SystemsManager Doesnt Fetch Parameter

When using Amazon.Extensions.Configuration.SystemsManager, parameters are unable to be fetched from SSM. However, using Amazon.SimpleSystemsManagement works perfectly.

Code Snippets:

Amazon.Extensions.Configuration.SystemsManager

using Amazon.Extensions.Configuration.SystemsManager;
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using System;

namespace TestParameterStore
{
	class Program
	{
		public static Action<SystemsManagerExceptionContext> OnLoadException { get; set; }
		static void Main(string[] args) => CreateHostBuilder(args).Build().Run();

		public static IWebHostBuilder CreateHostBuilder(string[] args)
		{
			return WebHost.CreateDefaultBuilder(args).ConfigureAppConfiguration(builder =>
			{
				builder.AddSystemsManager(configureSource =>
				{
					configureSource.Path = "/xyz/abc/";
					configureSource.Optional = true;
					configureSource.OnLoadException += OnLoadException;

				});
			})
			.UseStartup<Startup>();
		}
	}
}

Amazon.SimpleSystemsManagement

using System;
using System.Threading.Tasks;
using Amazon.SimpleSystemsManagement;
using Amazon.SimpleSystemsManagement.Model;

namespace TestParameterStore
{
    class Program1
    {
        //static void Main(string[] args) => GetConfiguration().Wait();

        static async Task GetConfiguration()
        {
            // NOTE: set the region here to match the region used when you created
            // the parameter
            var region = Amazon.RegionEndpoint.APSoutheast2;
            var request = new GetParameterRequest()
            {
                Name = "/app/abc/shared/rabbitmq/host"
            };

            using (var client = new AmazonSimpleSystemsManagementClient(region))
            {
                try
                {
                    var response = await client.GetParameterAsync(request);
                    Console.WriteLine($"Parameter {request.Name} value is: {response.Parameter.Value}");
                }
                catch (Exception ex)
                {
                    Console.Error.WriteLine($"Error occurred: {ex.Message}");
                }
            }
        }
    }
}

In addition, the Amazon.Extensions.Configuration.SystemsManager works fine in one environment and fails in another. There is no obvious reason as to why it would fail. On the other hand, Amazon.SimpleSystemsManagement works in both environments perfectly.

Release Date?

Any word on a release date... this has just come along at exactly the right time, we're just looking at moving a lot of stuff to the Code* suite, and this will be really important to make that work.

Fetching a single parameter with full path does not work

We are working on multiple projects in AWS and have multiple parameters. Some projects need only one parameter from a hierarchy. When adding a full path to the source's path property no parameters are fetched.

I tried with the SimpleSystemsManagement client and noticed that the method used (GetParametersByPathAsync) does not return any parameters with a full parameter path. There should be a method for getting single parameter(s) and optionally it could have the configuration key(s) as an parameter.

Remove unnececssary NuGet dependency

The project is declaring Microsoft.CodeQuality.Analyzers and Microsoft.NetCore.Analyzers as dependencies in the csproj causing them to be listed ass NuGet dependencies when they are only used for development purposes.

The dependencies can cause problems for other users.

I believe we need to add PrivateAssets>all</PrivateAssets> to the <PackageReference> to have them not be included in the nuspec.

Need some help with "optional" issue

I realized today that get-parameters-by-path does not throw an exception, even if the root does not exist. It just returns an empty parameter array
image

This means that currently, even if you put optional: false you will not get an error if the path does not exist. I see a couple ways to solve this.

  1. Leave it as is and optional: false will only "trigger" if there is an exception while calling the aws api
  2. Check the returned parameter count and if it's 0 and optional is false throw an exception

How do we want to solve this?

Does this library support pulling in AWS region from the default profile?

Just as the title asks, does this library support pulling in AWS region from the default profile?

I have the default AWS profile set up on my machine for my user (via .aws/config file) and this library did not seem to recognize that my region was configured there. I ended up having to add configuration for it in my appsettings.json file--which isn't ideal.

Thanks in advance.

List required permissions in the README

We were deploying our site to an EC2 instance, and it was failing to start.
After some digging around, we realised we needed to increase the permissions list for our EC2 role.

Would it be possible to list these requirements in the README, or link to a page that's up to date & easy to understand?
It was only discovered via here : https://aws.amazon.com/blogs/mt/the-right-way-to-store-secrets-using-parameter-store/

This would save the next person a lot of issues

warning NU5125: The 'licenseUrl' element will be deprecated. Consider using the 'license' element instead.

I noticed this warning today when testing Issue #26.
warning NU5125: The 'licenseUrl' element will be deprecated. Consider using the 'license' element instead.
https://docs.microsoft.com/en-us/nuget/reference/errors-and-warnings/nu5125

It appears that Microsoft is depricating license url because it's mutable after the package is created, and they are switching a different model where you'll directly imbed the license into the package or link to license text hosted by Nuget.
https://github.com/NuGet/Home/wiki/Packaging-License-within-the-nupkg

Nuget Apache 2.0 license: https://licenses.nuget.org/Apache-2.0

It seems like an easy modification to resolve this warning, but since this is licensing related I'm sure you prefer to handle the resolution.

No parameters are returned

I am not able to retrive any parameters. This is an aspnet core 2 program. I've added and verified my credentials are correct in my launchsettings (environment vars).

In program cs ...

        public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
            WebHost.CreateDefaultBuilder(args)
        .ConfigureAppConfiguration(builder =>
        {

            var awsOptions = new AWSOptions();
            awsOptions.Credentials = new EnvironmentVariablesAWSCredentials();
             builder.AddJsonFile("appsettings.json");
            builder.AddSystemsManager("/config", awsOptions, false, TimeSpan.FromMinutes(5));
        }).UseStartup<Startup>();
    }

In Startup.cs

        public void ConfigureServices(IServiceCollection services)
        {
            services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
            var awsOptions =new AWSOptions();
            awsOptions.Credentials = new EnvironmentVariablesAWSCredentials();
            services.AddDefaultAWSOptions(awsOptions);
        
            var myconfig = Configuration.GetSection("test");
         }

myconfig comes back with a value of null - my parameter is /config/test/something.

The bottom line is that i want to pass in my environment vars, and i know it works because when i dont i get an error that i dont have access. Why am i getting null essentially? It doesnt look to be pulling from the parameter store. Thank you!

Update dependent AWS libraries to support AWSSDK.Core >= 3.5.0

AWSSDK.Core is updated to v3.5.1, AWSSDK.SimpleSystemsManagement & AWSSDK.Extensions.NETCore.Setup library dependencies in this library are ~3.3.100.8 & ~3.3.100.1 (respectively) but those versions depend on AWSSDK.Core < 3.4.0 which means when you upgrade OTHER AWS libraries to use v3.5.1 of the AWSSDK.Core you get a warning in Visual Studio that you've got libraries depending on different versions of AWSSDK.Core.

Describe the Feature

Update dependencies to new(est/er) versions which link to AWSSDK.Core v3.5+

Is your Feature Request related to a problem?

Visual Studio warns:

Detected package version outside of dependency constraint: AWSSDK.SimpleSystemsManagement 3.3.100.8 requires AWSSDK.Core (>= 3.3.100.7 && < 3.4.0) but version AWSSDK.Core 3.5.1 was resolved.
Detected package version outside of dependency constraint: AWSSDK.Extensions.NETCore.Setup 3.3.100.1 requires AWSSDK.Core (>= 3.3.100 && < 3.4.0) but version AWSSDK.Core 3.5.1 was resolved.

Proposed Solution

Update library dependency versions.

Describe alternatives you've considered

Additional Context

Environment

I'm seeing this on Windows in Visual Studio but I presume it's not specific to my environment/platform.

  • ๐Ÿ‘‹ I may be able to implement this feature request
  • โš ๏ธ This feature might incur a breaking change

This is a ๐Ÿš€ Feature Request

Lambda Dotnet Core Serverless App

Can you provide an example of how to use this with a Lambda Dotnet Core Serverless App please.

Do we add it in to the ConfigureServices?

Sample Lambda custom runtime integration

I've scoured the interwebs for an example of integrating the .Net Generic Host with a lambda using a 3.1 custom runtime. I haven't found one - all the examples assume that MVC is being used.

As the lambda bootstrap doesn't seem to work natively with the generic host interface, I'm using Host.CreateDefaultBuilder to build the appropriate host and then setting the IHost.Services up as a static field to access in the Lambda Handler. This all seems awful and if there's a better way to do it it's not documented anywhere.

Is there a better way to get Config into a simple lambda without MVC?

Saving configuration, loading on demand

Is there a way that configuration can be saved/updated using this library? The Data dictionary is updated but there is no call to update or create the parameter.

The second one is for lambda loading the entire configuration is quite time consuming when you may not need to load anything for that invocation. Is there a way to do load on demand rather than load all at start-up?

Expected Behaviour

  1. configuration["myFeature"] = "false";
    Makes a call to PutParameter and saves the change to Parameter Store

  2. var config = configuration["myFeature"]
    Gets parameter on demand. Rather than a background task to get the parameters.

Current Behavior

Change is made locally, and the value stored in the ConfigurationProvider data.

Possible Solution

  1. Override void Set(string key, string value) in Configuration Provider, check if the key exists, update in parameter store. Keep in mind that all sources are called with Set() so a check must be made to whether we should save it.
    We run an installer to set values rather than using CDK so a switch to create the parameter would be useful. Don't really wish to use a different library to get configuration than to set it.

  2. On TryGet() either populate the Data in one go or preferably just get all parameters with this prefix. Keeping track of what's been downloaded already would be difficult but for lambda the runtime is crucial.

Your Environment

AWSSDK.Core version: 3.3.0.0+
Amazon.Extensions.Configuration.SystemsManager: 2.0

.NET Core Info

SDK 3.1.300

"no such host is known" when configuring startup in .net core app

I'm using Amazon.Extensions.Configuration.SystemsManager v 1.2.0 with VisualStudio 2019
I created a new project from the AWS Serverless Application template for NET Core, which defaulted to the 3.1 runtime but i've tried the 2.2 runtime as well and I see the same error.

The error description just says "No such host is known". I used Wireshark to determine the lib is trying to access ssm.defaultregion.amazonaws.com .

I did the "set-credentials" command from Powershell, I removed the RegisteredAccounts.json from this location per a similar issue elsewhere C:\Users\ladmin\AppData\Local\AWSToolkit\ I tried manually setting the AWS_REGION via powershell, and none of these resolve the issue.

Even stranger, I have another project that uses the exact same startup.cs but the 1.0.1 version of the SystemsManager lib, and it works correctly. Edit: I just tried downgrading to this version and i'm still seeing the issue.

The problem occurs when calling the .Build() function
51636d64-a6f4-42f8-bb6d-d1eba03941ee

I know I can set in code which region/creds I want to use, but this would break the deployment pipeline as I would have to keep the code files updated with the different configs vs just using the credentials files available on each environment.

It seems the failure is caused because the lib is using default values to try and connect to the AWS services, and is not reading the credentials correctly. Is there any documentation or way to determine why this process is failing?

Full stack trace:

   at System.Net.Http.ConnectHelper.<ConnectAsync>d__2.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.<CreateConnectionAsync>d__44.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.<WaitForCreatedConnectionAsync>d__49.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.<SendWithRetryAsync>d__39.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.RedirectHandler.<SendAsync>d__4.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsyncUnbuffered>d__63.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.HttpWebRequestMessage.<GetResponseAsync>d__20.MoveNext() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\_mobile\HttpRequestMessageFactory.cs:line 524
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.HttpHandler`1.<InvokeAsync>d__9`1.MoveNext() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\HttpHandler.cs:line 175
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.Unmarshaller.<InvokeAsync>d__3`1.MoveNext() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\Unmarshaller.cs:line 89
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__5`1.MoveNext() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\ErrorHandler\ErrorHandler.cs:line 121
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\CallbackHandler.cs:line 64
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.<InvokeAsync>d__2`1.MoveNext() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\EndpointDiscoveryHandler.cs:line 89
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.<InvokeAsync>d__2`1.MoveNext() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\EndpointDiscoveryHandler.cs:line 79
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.CredentialsRetriever.<InvokeAsync>d__7`1.MoveNext() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\CredentialsRetriever.cs:line 98
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\RetryHandler\RetryHandler.cs:line 154
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\RetryHandler\RetryHandler.cs:line 137
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\CallbackHandler.cs:line 64
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\CallbackHandler.cs:line 64
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__5`1.MoveNext() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\ErrorCallbackHandler.cs:line 60
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.MetricsHandler.<InvokeAsync>d__1`1.MoveNext() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\MetricsHandler.cs:line 73
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Amazon.Extensions.Configuration.SystemsManager.Internal.SystemsManagerProcessor.<GetParametersByPathAsync>d__6.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Extensions.Configuration.SystemsManager.Internal.SystemsManagerProcessor.<GetDataAsync>d__5.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Extensions.Configuration.SystemsManager.SystemsManagerConfigurationProvider.<LoadAsync>d__13.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Extensions.Configuration.SystemsManager.SystemsManagerConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at MYAPP.Startup..ctor(IConfiguration configuration, IHostingEnvironment env) in C:\Users\ladmin\Documents\events\Sites\HSS\AccessibilityPlayerFrontend\HSSAccessibility\Startup.cs:line 52

Using SSM parameter as ConfigurationSection

Is it possible to use SSM parameter as ConfigurationSection?

For example parameter with name /example/settings/Logging
with json value

{
"Level": "Information",
"SomethingElse": "Test"
}

And then use it like this?

public class Logging
{
    public string Level { get; set; }
    public string SomethingElse{ get; set; }
}

...

var configurationSection = configuration.GetSection(nameof(Logging));
services.Configure<Logging>(configurationSection);

AddSystemsManager causes LocalEntryPoint to hang

I created a default "AWS Serverless Application" for Blueprint "ASP.NET Core Web App", loaded the Amazon.Extensions.Configuration.SystemsManager nuget package
and modified the LocalEntryPoint.

public static IWebHost BuildWebHost(string[] args) => WebHost.CreateDefaultBuilder(args) .ConfigureAppConfiguration(builder => { builder.AddSystemsManager("/pf"); }) .UseStartup<Startup>() .Build();

I get this exception:
This exception was originally thrown at this call stack: System.Net.Http.HttpClient.HandleFinishSendAsyncError(System.Exception, System.Threading.CancellationTokenSource) System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(System.Threading.Tasks.Task<System.Net.Http.HttpResponseMessage>, System.Net.Http.HttpRequestMessage, System.Threading.CancellationTokenSource, bool) System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task) System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) System.Runtime.CompilerServices.ConfiguredTaskAwaitable<TResult>.ConfiguredTaskAwaiter.GetResult() System.Net.Http.HttpClient.GetStringAsyncCore(System.Threading.Tasks.Task<System.Net.Http.HttpResponseMessage>) System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task) System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) ... [Call Stack Truncated]

Would love to get the Parameter Store working but currently there is insufficient documentation for .NET Core 2.1 running in Lambda and Locally.

Tried https://github.com/aws/aws-dotnet-extensions-configuration but the page does not reflect the code we see in the template for "AWS Serverless Application" for Blueprint "ASP.NET Core Web App".

Also tried https://seroter.wordpress.com/2019/03/04/want-to-yank-configuration-values-from-your-net-core-apps-heres-how-to-store-and-access-them-in-azure-and-aws/ but this gives me the above exception.

Any help to get this going would be appreciated.

Cannot Retrieve Data From Parameter Store

We have stored our logging configuration in AWS parameter store. we use Amazon.Extensions.Configuration.SystemsManager (nuget package) for reading configuration and AWS.Logger.AspNetCore (nuget) to write into cloud watch via .NET core.

Here is the code where we have set up the system manager

public static IWebHostBuilder CreateWebHostBuilder(string[] args) => WebHost.CreateDefaultBuilder(args) .ConfigureAppConfiguration(config => { config.AddSystemsManager(g => { g.AwsOptions = new Amazon.Extensions.NETCore.Setup.AWSOptions(); g.AwsOptions.Region = RegionEndpoint.USEast1; g.Path = $"/Dev/FileService"; }); }) .UseStartup<Startup>();

Here is the code for the logging configuration

`//This line retreives the json data
var t = Configuration["AWS.Logging"];

//This returns a null
AWSLoggerConfigSection awsLoggingConfig = Configuration.GetAWSLoggingConfigSection();`

I have a parameter key "/Dev/FileService/AWS.Logging" defined in parameter store and the following JSON

{ "Region": "us-east-1", "LogStreamNameSuffix": "fileservice", "LogGroup": "imagerservice", "LogLevel": { "Default": "Error", "System": "Error", "Microsoft": "Error" } }

The problem is that I can retrieve this logging configuration if I do the following

var t = Configuration["AWS.Logging"];

But this code returns null all the time

AWSLoggerConfigSection awsLoggingConfig = Configuration.GetAWSLoggingConfigSection();

Any idea?

Add Json support to reduce need for so many key/value pairs

It would be nice if this library supported a string that contained JSON and would parse that string and create settings based on that string, exactly how the standard appsettings.json works.

Describe the Feature

Most developers in .net core are used to storing their settings in appsetting.json files in a json format, it would be nice if a single key could contain a full json string that could parsed into configuration keys in the same manner.

Proposed Solution

It looks like most the work is nearly done since there is already a json parser for using the secrets manager


This is a ๐Ÿš€ Feature Request

Testing via net472

Hi,

The integration tests I am working on right now is around an internal wrapper lib for your configuration extension and the tests are very similar to yours (in this repos' tests folder here).

As we are migrating our old net472 code to dotnet standard/core gradually, we have been targeting our test projects to both net472 and netcoreapp2.0 like this:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>netcoreapp2.2;net472</TargetFrameworks>

Although integration test that retrieves values from aws are successful for netcoreapp2 target, the same test fails for net472.

Would you have any insights, what am I missing? I should be able to link your lib in a net472 lib, right?

Cannot retrieve SSM StringList as list of string

Hi,

We have created a parameter named /Staging.Commom/Throttling/IpWhiteList as a StringList.
Current value is 1.1.1.1,0.0.0.0,2.2.2.2

The code below always return null:

Configuration.GetValue<IEnumerable<string>>("Throttling:IpWhiteList");
Configuration.GetValue<ICollection<string>>("Throttling:IpWhiteList");
Configuration.GetValue<IList<string>>("Throttling:IpWhiteList");
Configuration.GetValue<string[]>("Throttling:IpWhiteList");
Configuration.GetValue<StringListParameterValue>("Throttling:IpWhiteList");

The code below works fine:

Configuration.GetValue<string>("Throttling:IpWhiteList");

Since it's a string list in AWS SSM, we were expecting to get a collection of strings. Did we do something wrong?

Regards,
Seb

Using SecretsManager with a PlainText value causes the JObject Parse to fail

  • Create a secret in Secrets Manager as PlainText
  • Try and reference secret using the path below in progam.cs for example:

config.AddSystemsManager($"/aws/reference/secretsmanager/settings/SecretKey");

  • Following exception occurs in the JsonConfigurationParser (Line 61) when trying to load the JObject:

    at Newtonsoft.Json.JsonTextReader.ParseValue()
    at Newtonsoft.Json.JsonTextReader.Read()
    at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader, JsonLoadSettings settings)
    at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader)
    at Amazon.Extensions.Configuration.SystemsManager.Internal.JsonConfigurationParser.ParseInput(TextReader input) in C:\Development\SmartGit\aws-dotnet-extensions-configuration\src\Amazon.Extensions.Configuration.SystemsManager\Internal\JsonConfigurationParser.cs:line 61
    at Amazon.Extensions.Configuration.SystemsManager.Internal.JsonConfigurationParser.Parse(String input) in C:\Development\SmartGit\aws-dotnet-extensions-configuration\src\Amazon.Extensions.Configuration.SystemsManager\Internal\JsonConfigurationParser.cs:line 52
    at Amazon.Extensions.Configuration.SystemsManager.Internal.SystemsManagerProcessor.GetParameterAsync() in C:\Development\SmartGit\aws-dotnet-extensions-configuration\src\Amazon.Extensions.Configuration.SystemsManager\Internal\SystemsManagerProcessor.cs:line 88
    at Amazon.Extensions.Configuration.SystemsManager.Internal.SystemsManagerProcessor.GetDataAsync() in C:\Development\SmartGit\aws-dotnet-extensions-configuration\src\Amazon.Extensions.Configuration.SystemsManager\Internal\SystemsManagerProcessor.cs:line 47
    at Amazon.Extensions.Configuration.SystemsManager.SystemsManagerConfigurationProvider.LoadAsync(Boolean reload) in C:\Development\SmartGit\aws-dotnet-extensions-configuration\src\Amazon.Extensions.Configuration.SystemsManager\SystemsManagerConfigurationProvider.cs:line 109

'IConfigurationBuilder' does not contain a definition for 'AddSystemsManager'

I'm getting the following error on .NET Core 2.2 after adding this library as a NuGet package:

'''
'IConfigurationBuilder' does not contain a definition for 'AddSystemsManager' and no accessible extension method 'AddSystemsManager' accepting a first argument of type 'IConfigurationBuilder' could be found (are you missing a using directive or an assembly reference?)
'''

Snag_52c7fe6

Calling Build on IConfigurationBuilder Will Break the Configuration Process When Streams Are Involved

Description

On line 211 of SystemsManagerExtensions Build() is called on the IConfigurationBuilder that the subject of the extensions methods, this IConfigurationBuilder builder. By doing this any other code that adds a stream through AddJsonStream via Microsoft.Extensions.Configuration.Json or Microsoft.Extensions.Configuration.NewtonsoftJson will fail when IHostBuilder.Build() is executed. This is because the streams are closed after the first call to Build() on the IConfigurationBuilder

Reproduction Steps

This is reproducible only when the configuration is read from the IConfigurationBuilder. Meaning, that if the overload to AddSystemsManger which accepts AWSOptions as an object is used this error does not happen. However, this defeats the purpose of storing the AWS configuration within the .NET Core configuration framework.

Host.CreateDefaultBuilder(args)
  .ConfigureAppConfiguration((configurationBuilder) => {
    configurationBuilder.AddJsonStream(new MemoryStream(Encoding.UTF8.GetBytes("{ \"SomeKey\": \"SomeValue\" }")));
    configurationBuilder.AddSystemsManager($"/test", true, TimeSpan.FromMinutes(15));
  })
  .ConfigureWebHostDefaults((webBuilder) => {
    webBuilder.UseStartup<Startup>();
  })
  .Build()
  .Run();

Logs

System.ArgumentException: 'Stream was not readable.'

Environment

  • Build Version: 2.1.0
  • OS Info: Win10 (Suspected to not be env specific)
  • Build Environment: VS2019
  • Targeted .NET Platform: .NET Core 3.1

Resolution

  • ๐Ÿ‘‹ I can/would-like-to implement a fix for this problem myself

Two issues have been raised on the root cause in the .NET projects:

The first references the second

Per the team:

In general using Build() twice on configuration builder is not in the realm of recommended workflows and how we expect callers to use it.

I'm not familiar with the structure behind configuration providers but after looking at the implementation I can make a guess why this is done. My assumption is that the dependency injection framework isn't available to configuration providers because the host hasn't been built. This is the idea around the SystemsManagerConfigurationSource, correct?

Perhaps attempting to read the AWSOptions values from the host configuration would work. The extension usage would instead focus on extending IHostBuilder. Using the fluent configuration a call to IHostBuilder.ConfigureHostConfiguration would allow for the addition of appsettings.json and appsettings.<env>.json. A subsequent call to IHostBuilder.ConfigureAppConfiguration could then use context.Configuration to read the AWSOptions and feed those through the current calls to AddSystemsManager that take the object.

At the very least some sort of documentation of this should be made. I've been chasing this issue down for a day. :)

This is a ๐Ÿ› bug-report

appsettings.json element AWS:Region:DefaultRegion results in crash

Description

We are utilizing the library from AWS GovCloud. This behavior may not reproduce in AWS Commerical.

Reproduction Steps

Please reference: aws/aws-lambda-dotnet#832

But the short answer was that if the appsettings.json contained the default that apparently gets overwritten during a 'dotnet lambda package` command the AWS:Region:DefaultRegion element was causing a crash.

Example appsettings.json:

{
  "Lambda.Logging": {
    "IncludeCategory": false,
    "IncludeLogLevel": false,
    "IncludeNewline": true,
    "LogLevel": {
      "Default": "Debug",
      "Microsoft": "Information"
    }
  },
  "AWS": {
    "Region": "DefaultRegion"
  }
}

Logs

An exception was thrown when the constructor for type 'XXXXX.LambdaEntryPoint' was invoked. Check inner exception for more details.: LambdaException


   at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean wrapExceptions, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& hasNoDefaultCtor)
   at System.RuntimeType.CreateInstanceDefaultCtorSlow(Boolean publicOnly, Boolean wrapExceptions, Boolean fillCache)
   at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, Boolean wrapExceptions)
   at System.Activator.CreateInstance(Type type, Boolean nonPublic, Boolean wrapExceptions)
   at System.Activator.CreateInstance(Type type)
Name or service not known: HttpRequestException
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Extensions.Configuration.SystemsManager.Internal.SystemsManagerProcessor.GetParametersByPathAsync()
   at Amazon.Extensions.Configuration.SystemsManager.Internal.SystemsManagerProcessor.GetDataAsync()
   at Amazon.Extensions.Configuration.SystemsManager.SystemsManagerConfigurationProvider.LoadAsync(Boolean reload)
   at Amazon.Extensions.Configuration.SystemsManager.SystemsManagerConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at Amazon.Lambda.AspNetCoreServer.AbstractAspNetCoreFunction`2.Start()
   at Amazon.Lambda.AspNetCoreServer.AbstractAspNetCoreFunction`2..ctor(StartupMode startupMode)
   at Amazon.Lambda.AspNetCoreServer.AbstractAspNetCoreFunction`2..ctor()
   at Amazon.Lambda.AspNetCoreServer.APIGatewayProxyFunction..ctor()
   at XXXXX.LambdaEntryPoint..ctor()
Name or service not known: SocketException
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

[WARN] ([email protected]:331 errno: Address family not supported by protocol) run_dotnet(dotnet_path, &args) failed

Resolution

I was able to avoid the crash by manually overriding the DefaultRegion value with our actual region

Environment Variable:

Key: AWS__Region
Value:


This is a ๐Ÿ› bug-report

SSM does not work with AWS X-Ray

I added this project to solution in according to documentation.
I don't see X-Ray tracing for SSM client. And also I don't see SSM servive on X-Ray Service map. I suppose it is because IAmazonSimpleSystemsManagement client is created using reflection. I think there should be way to resolve IAmazonSimpleSystemsManagement using DI or implicit setting IAmazonSimpleSystemsManagement instance into the library.

AddSystemsManager causing Deadlock or .NET application to stop unexpectedly

Description

Calling .AddSystemsManager and then .Build() causes a 30 second lag.

Reproduction Steps

Using Amazon.Lambda.RuntimeSupport to run a .NET 5 container. Running latest lambda, CDK, packages and latest amazonlinux. Was working 3 weeks ago. Downgrading Lambda packages has not fixed the issue.

Logs

Cannot get any logs because I don't know of any easy way to access the lambda container.

Environment

  • Build Version 2.1.0
    • OS Info: AmazonLinux
  • Build Environment: Terminal dotnet publish inside .net 5 SDK container
    • Targeted .NET Platform: net5.0

What I need to debug the issue further

Need a better way to debug what is going on in the container, right now it's like testing a black box that takes 10 minutes to deploy.

What I've tried

  1. Downgrading packages,
  2. Moving Configuration build to before starting lambda bootstrapper.
  3. Specifying awsCredentials explicitly (this fixed local SSO with the latest awssdk)
  4. downgrading to Amazonlinux build from 2 months ago
  5. Tracing calls by submoduling aws-dotnet-extensions-configuration (traced issue to SystemsManagerProcessor:65)

This is a ๐Ÿ› bug-report

Integration tests should use IAsyncLifetime

The integration tests use shared setup and cleanup code within a fixture. The fixture follows a constructor/dispose pattern, which forces logic to be synchronous. This makes this fixture complicated and messy. The fixture could instead implement IAsyncLifetime and use async init/dispose.

ReloadAfter doesn't work in Lambda

The ReloadAfter functionality in SystemsManagerConfigurationProvider doesn't work if the code runs in AWS Lambda. Parameter values remain the same during the lifetime of the Lambda instance. It works fine if I run the code locally on my development machine.

We would very much like this to work, since you can't force a Lambda-function to restart and load the new configuration values.

How to query the parameter store without ASP.NET

Could you please provide a pure dotnet core example that doesn't depend on ASP.NET. I just have a console app and I don't want to take an unnecessary dependency on Microsoft.Extensions (or that way of doing things) just to read from the parameter store.

Cannot load parameters from empty string path

Expected: Setting the Path option to empty string loads all parameters.
Actual: Path cannot be null exception.

Setting path to the minimum "/" loads all parameters, but trims the first character from each of the keys.

Support for loading multiple paths

I would like to be able to load from two or more path prefixes. My use case is as follows:

I have two categories of configuration. There is global configuration that is applicable to many projects, and there is specific configuration that only applies to one service.

These are organized like this:
Global Configuration : /metaProjectName/global/dev/queueDynamoTableName
Specific Configuration : /metaProjectName/serviceXYZ/dev/abcApiRequestBatchSize

With the currently available functionality, I could access everything I need by using /metaProjectName as the base path, but I'm also then exposing not only other services' configuration but other environments to each other which I don't particularly like.

I would prefer if I could just specify /metaProjectName/global/dev as well as /metaProjectName/serviceXYZ/dev and have access to exactly what's necessary.

Should I add a Sample Project?

Would you guys like me to add a sample project with some examples of calling the extension? If so should I add some scripts to seed some example data in their parameter store?

Consider adding a roslyn ruleset specifically for tests

As discovered in #19, if async tasks do not do ConfigureAwait(), dotnet build will emit the following warning:

warning CA2007: Do not directly await a Task without calling ConfigureAwait 

It might be a good idea to suppress the warning by having a separate rosyln ruleset for test projects.

ReloadAfter does not work in deployed MVC Web API 3.1

System Manager ReloadAfter works fine running this executable locally on development machine.

Unfortunately, the ReloadAfter functionality in SystemsManagerConfiguration does not work in our .NET solution running/deployed in AWS when the application is running inside a Docker container (default Microsoft/Alpine base image mcr.microsoft.com/dotnet/core/aspnet:3.1-alpine) deployed to ECS/Fargate

In the AWS deployment Parameter values are loaded at startup but remain the same during the lifetime of the instance.

The Parameter values are loaded at started up and then not reload when deployed in AWS. Locally the Parameters values are reloaded every 2 minutes as specified by call to ReloadAfter


This is a ๐Ÿ› bug-report

Anything else?

Outside PR #12 is there anything else that you need or want from me for the project? Either way just let me know.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.