newrelic/newrelic-lambda-cli

A CLI to install the New Relic AWS Lambda integration and layers.

newrelic-lambda-cli's Issues

Problem running the script to setup lambda

Hello!

Context:

• trying to follow the steps outlined in the documentation
• aws cli is configured and works

Problem:

When running the script:

$ ./nr-lambda-onboarding-master/newrelic-cloud set-up-lambda-integration --nr-account-id "xxx" --linked-account-name "xxx" --nr-api-key "xxx" --nr-license-key "xxx" --regions "ap-southeast-2"

Getting the following output:

Failed actions:
  Function: None, Region: None, Error: Failed to set up lambda integration: HTTP Error 401: Unauthorized

ADd us-gov-* regions

boto3 defaults to aws partition for session.get_available_regions(). Include other partitions, such as aws-us-gov.

Add python3.8 Runtime Support

Cannot use function ARN for installing log subscription

Description

The documentation for newrelic-lambda subscriptions install says that the --function flag is "the AWS Lambda function name or ARN." However, passing the ARN always yields an error message.

newrelic-lambda subscriptions install --function arn:aws:lambda:us-east-1:{account id}:function:{function name} --no-aws-permissions-check

Error retrieving log subscription filters for 'arn:aws:lambda:us-east-1:{account id}:function:{function name}': An error occurred (InvalidParameterException) when calling the DescribeSubscriptionFilters operation: 1 validation error detected: Value '/aws/lambda/arn:aws:lambda:us-east-1:{account id}:function:{function name} at 'logGroupName' failed to satisfy constraint: Member must satisfy regular expression pattern: [.-_/#A-Za-z0-9]+

(Note: I have redacted our account id and function names from the above messages.)

Steps to Reproduce

See above.

Expected Behaviour

It should internally translate the function ARN to the function name when deriving the log group name, as implied by the documentation.

Relevant Logs / Console output

See above.

Your Environment

Running inside a docker container using version 0.1.22 of this CLI and version 1.18.16 of the AWS CLI.

Support CloudFormation stack tagging when installing Lambda integration

Our AWS accounts have automation enabled to delete untagged resources, including CloudFormation stacks. The current behavior of the Lambda CLI creates untagged CloudFormation stacks. Unlike the API for Lambda, S3, etc., the AWS CLI does not support tagging CloudFormation resources as an independent operation. You can specify them in an update-stack request, but will also need to provide parameter names, etc.

Since using this CLI is the recommended approach from New Relic technical support, it would be a welcome addition to support CloudFormation resource tagging.

Summary

See above.

Desired Behaviour

Users can specify tags to associate with their CloudFormation stack.

Possible Solution

newrelic-lambda integrations install --nr-account-id $nr_account_id --linked-account-name $aws_account_name --nr-api-key ${log_ingestion_api_key} --no-aws-permissions-check --cf-stack-tags Key1=Value1,Key2=Value2,etc.

Additional context

See above: untagged CloudFormation resources in our AWS accounts are automatically deleted within 24 hours and there's no independent way to add tags to a CloudFormation stack.

Failed to set up lambda integration: 'service'

Open New Relic One's Modern platforms> Lamda menu
The script downloaded from "Download Setup Script" was executed.

An error message was displayed, but it doesn't matter what caused the error, which is not listed in troubleshooting.

EC2 (Cloud9) is used for the script execution environment, and AdministraotrAccess is assigned to the IAM role.

NR account is used with Admin's authority.

$./newrelic-cloud set-up-lambda-integration --nr-account-id xxxxxx --linked-account-name "xxxxxxx" --nr-api-key aaaaaaaa --regions "ap-northeast-1"
Validating New Relic credentials and retrieving integration license key
Checking for a preexisting link between NR and AWS, and validating the AWS CLI credentials
Creating the AWS role for the New Relic integration
Linking your New Relic account to your AWS account
Cloud integrations account [xxxxxx] already exists in New Relic account [yyyyyy].
Enabling Lambda integration on the link between New Relic and AWS

Failed actions:
Function: None, Region: None, Error: Failed to set up lambda integration: 'service'

newrelic-lambda integrations uninstall fails with botocore.exceptions.WaiterError

newrelic-lambda integrations uninstall fails with botocore.exceptions.WaiterError: Waiter StackDeleteComplete failed: Waiter encountered a terminal failure state

Description

Updated newrelic-lambda-cli today 11/13/2020
Attempted to deinstall integration that was set up and verified 11/11/20

Steps to Reproduce

newrelic-lambda integrations uninstall for a working integration

Expected Behaviour

Script to complete without error

Relevant Logs / Console output

his will uninstall the New Relic AWS Lambda log ingestion function and role. Are you sure you want to proceed? [y/N]: y
No New Relic AWS Lambda log ingestion found in region us-east-1, skipping
This will uninstall the New Relic License Key managed secret, and IAM Policy. Are you sure you want to proceed? [y/N]: y
Deleting stack 'NewRelicLicenseKeySecret'
Waiting for stack deletion to complete, this may take a minute... Unexpected AWS error: Waiter StackDeleteComplete failed: Waiter encountered a terminal failure state
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/newrelic_lambda_cli/utils.py", line 27, in _boto_error_wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/newrelic_lambda_cli/cli/init.py", line 27, in main
cli()
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/newrelic_lambda_cli/cli/integrations.py", line 217, in uninstall
integrations.remove_license_key(session)
File "/usr/local/lib/python3.9/site-packages/newrelic_lambda_cli/integrations.py", line 609, in remove_license_key
client.get_waiter("stack_delete_complete").wait(StackName=LICENSE_KEY_STACK_NAME)
File "/usr/local/lib/python3.9/site-packages/botocore/waiter.py", line 53, in wait
Waiter.wait(self, **kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/waiter.py", line 320, in wait
raise WaiterError(
botocore.exceptions.WaiterError: Waiter StackDeleteComplete failed: Waiter encountered a terminal failure state

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/newrelic-lambda", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/newrelic_lambda_cli/utils.py", line 35, in _boto_error_wrapper
error("Unexpected AWS error: %s" % e)
File "/usr/local/lib/python3.9/site-packages/newrelic_lambda_cli/utils.py", line 77, in error
raise Exit(1)
click.exceptions.Exit: 1

Your Environment

macOs Catalina
python 3.9
aws-cli/2.0.62 Python/3.7.4 Darwin/19.6.0 exe/x86_64

Additional context

[layers install] Support installing layer for .NET Core and Java runtime.

Summary

Now New Relic Lambda extension can support .NET Core, Java and Go in addition to node.js and Python.
https://github.com/newrelic/newrelic-lambda-extension/tree/main/examples

It's good to support these runtimes.

Desired Behaviour

When the runtime is specified as one of dotnetcore3.1, java8.al2, java11, layers install command will add NewRelicLambdaExtension layer.
The layer endpoint returns an available layer.

https://ap-southeast-1.layers.newrelic-external.com/get-layers?CompatibleRuntime=dotnetcore3.1

It seems when code is written in Go, we need another approach since we should use provided runtime.

Possible Solution

How about adding the following key-value pairs to the RUNTIME_CONFIG

    "dotnetcore3.1": {"Handler": "NewRelicExampleDotnet::NewRelicExampleDotnet.Function::FunctionHandler"},
    "java8.al2": {"Handler": "com.newrelic.lambda.example.App::handleRequest"},
    "java11": {"Handler": "com.newrelic.lambda.example.App::handleRequest"},

Additional context

Wire up CI/CD

IndexError During Integration Install When Pre-Existing Install Present

The link_account method is too brittle. Additional checks are needed to verify that the integration was created by the CLI before attempting to link it. Also handling the case where no integrations are returned from the API.

~/tools/NewRelic $ newrelic-lambda integrations install --nr-account-id XXXXXX --linked-account-name XXXXXX --nr-api-key XXXXXXX
Validating New Relic credentials
Retrieving integration license key
Checking for a pre-existing link between New Relic and AWS
Creating the AWS role for the New Relic AWS Lambda Integration
✔️ New Relic AWS Lambda integration role 'NewRelicLambdaIntegrationRole_XXXXXX' already exists
Linking New Relic account to AWS account
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/bin/newrelic-lambda", line 11, in <module>
sys.exit(main())
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/newrelic_lambda_cli/utils.py", line 24, in _boto_error_wrapper
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/newrelic_lambda_cli/cli/__init__.py", line 25, in main
cli()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/newrelic_lambda_cli/cli/integrations.py", line 68, in install
gql_client, nr_account_id, linked_account_name, role
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/newrelic_lambda_cli/api.py", line 295, in create_integration_account
account = gql.link_account(role_arn, linked_account_name)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/newrelic_lambda_cli/api.py", line 134, in link_account
return res["cloudLinkAccount"]["linkedAccounts"][0]
IndexError: list index out of range

integrations install produces KeyError: 'service'

Description

Installing the new relic lambda integration fails at the Enabling Lambda integration on the link between New Relic and AWS step, with a KeyError.

Steps to Reproduce

Run CLI command to install the new relic lambda integration:

newrelic-lambda integrations install --nr-account-id XXX \
--linked-account-name XXX \
--nr-api-key XXX

Expected Behaviour

Everything should work.

Relevant Logs / Console output

Full console output:

Validating New Relic credentials
Retrieving integration license key
Checking for a pre-existing link between New Relic and AWS
Creating the AWS role for the New Relic AWS Lambda Integration
✔️ New Relic AWS Lambda integration role 'NewRelicLambdaIntegrationRole_XXX' already exists
Linking New Relic account to AWS account
✔️ Cloud integrations account [AWS Energetiq Dev] already exists in New Relic account [XXX].
Enabling Lambda integration on the link between New Relic and AWS
Traceback (most recent call last):
  File "/usr/local/bin/newrelic-lambda", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/newrelic_lambda_cli/utils.py", line 24, in _boto_error_wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/newrelic_lambda_cli/cli/__init__.py", line 26, in main
    cli()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/newrelic_lambda_cli/cli/integrations.py", line 73, in install
    gql_client, nr_account_id, linked_account_name
  File "/usr/local/lib/python3.7/site-packages/newrelic_lambda_cli/gql.py", line 321, in enable_lambda_integration
    is_lambda_enabled = gql.is_integration_enabled(account["id"], "lambda")
  File "/usr/local/lib/python3.7/site-packages/newrelic_lambda_cli/gql.py", line 199, in is_integration_enabled
    linked_account_id, service_slug
  File "/usr/local/lib/python3.7/site-packages/newrelic_lambda_cli/gql.py", line 194, in get_integration_by_service_slug
    (i for i in integrations if i["service"]["slug"] == service_slug), None
  File "/usr/local/lib/python3.7/site-packages/newrelic_lambda_cli/gql.py", line 194, in <genexpr>
    (i for i in integrations if i["service"]["slug"] == service_slug), None
KeyError: 'service'

Your Environment

macOS 10.14.6

Additional context

Attempted using these AWS credential modes, all resulting in the same error mentioned above:

• AWS_PROFILE environment variable
• AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables
• --aws-profile command line argument

It's possible that the new relic lambda integration has already been enabled in the past for this new relic account. After instrumenting a particular lambda function, I can actually see data/traces etc. coming through into New Relic One, except it's missing logs, and error reporting only works at the CloudWatch metrics level – I can see an error spike in the CloudWatch metrics section, but all traces report an error count of 0 and no errors appear in the 'Errors' section.

I assume that general data coming through but without logs and errors has something to do with this command not being successfully completed, possibly related to the CLI not being used when enabling the integration originally (I have no idea who would've done this – this account is a shared work account). Note that I have run the subscriptions install command successfully, but no logs appear for the instrumented lambda function.

UPDATE

Turns out that along with enabling the new relic lambda integration at some point, someone had also created a newrelic-log-ingestion lambda. I found the person who did it, and they said that they weren't following the instructions for this CLI, that they had followed some other guide a couple months ago.

After I removed the old lambda function (which wasn't in use) the integrations install command successfully completed.

I should note however that logs and errors are still not appearing – except for precisely 3 log entries which seem to have somehow come through (there should be at least an order of magnitude more than this). This was after following all of the instructions in this guide: https://docs.newrelic.com/docs/serverless-function-monitoring/aws-lambda-monitoring/get-started/enable-new-relic-monitoring-aws-lambda

But anyway, the fact that the logs and errors aren't coming through is a separate issue to this one I think, so I'll close this issue.

Readme should use pip3 for installing CLI

[Repolinter] Open Source Policy Issues

Repolinter Report

🤖This issue was automatically generated by repolinter-action, developed by the Open Source and Developer Advocacy team at New Relic. This issue will be automatically updated or closed when changes are pushed. If you have any problems with this tool, please feel free to open a GitHub issue or give us a ping in #help-opensource.

This Repolinter run generated the following results:

❗ Error	❌ Fail	⚠️ Warn	✅ Pass	Ignored	Total
0	4	0	3	0	7

• Fail
  • ❌ readme-starts-with-community-plus-header
  • ❌ readme-contains-link-to-security-policy
  • ❌ readme-contains-forum-topic
  • ❌ code-of-conduct-should-not-exist-here
• Passed
  • ✅ license-file-exists
  • ✅ readme-file-exists
  • ✅ third-party-notices-file-exists

Fail #

❌ readme-starts-with-community-plus-header #

The README of a community plus project should have a community plus header at the start of the README. If you already have a community plus header and this rule is failing, your header may be out of date, and you should update your header with the suggested one below. For more information please visit https://opensource.newrelic.com/oss-category/. Below is a list of files or patterns that failed:

• README.md: The first 5 lines do not contain the pattern(s): Open source Community Plus header (see https://opensource.newrelic.com/oss-category).
  • 🔨 Suggested Fix: prepend the latest code snippet found at https://github.com/newrelic/opensource-website/wiki/Open-Source-Category-Snippets#code-snippet-2 to file

❌ readme-contains-link-to-security-policy #

Doesn't contain a link to the security policy for this repository (README.md). New Relic recommends putting a link to the open source security policy for your project (https://github.com/newrelic/<repo-name>/security/policy or ../../security/policy) in the README. For an example of this, please see the "a note about vulnerabilities" section of the Open By Default repository. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

❌ readme-contains-forum-topic #

Doesn't contain a link to the appropriate forum.newrelic.com topic (README.md). New Relic recommends directly linking the your appropriate forum.newrelic.com topic in the README, allowing developer an alternate method of getting support. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

❌ code-of-conduct-should-not-exist-here #

New Relic has moved the CODE_OF_CONDUCT file to a centralized location where it is referenced automatically by every repository in the New Relic organization. Because of this change, any other CODE_OF_CONDUCT file in a repository is now redundant and should be removed. Note that you will need to adjust any links to the local CODE_OF_CONDUCT file in your documentation to point to the central file (README and CONTRIBUTING will probably have links that need updating). For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view. Found files. Below is a list of files or patterns that failed:

• CODE_OF_CONDUCT.md
  • 🔨 Suggested Fix: Remove file

Passed #

Click to see rules

✅ license-file-exists #

Found file (LICENSE). New Relic requires that all open source projects have an associated license contained within the project. This license must be permissive (e.g. non-viral or copyleft), and we recommend Apache 2.0 for most use cases. For more information please visit https://docs.google.com/document/d/1vML4aY_czsY0URu2yiP3xLAKYufNrKsc7o4kjuegpDw/edit.

✅ readme-file-exists #

Found file (README.md). New Relic requires a README file in all projects. This README should give a general overview of the project, and should point to additional resources (security, contributing, etc.) where developers and users can learn further. For more information please visit https://github.com/newrelic/open-by-default.

✅ third-party-notices-file-exists #

Found file (THIRD_PARTY_NOTICES.md). A THIRD_PARTY_NOTICES.md file can be present in your repository to grant attribution to all dependencies being used by this project. This document is necessary if you are using third-party source code in your project, with the exception of code referenced outside the project's compiled/bundled binary (ex. some Java projects require modules to be pre-installed in the classpath, outside the project binary and therefore outside the scope of the THIRD_PARTY_NOTICES). Please review your project's dependencies and create a THIRD_PARTY_NOTICES.md file if necessary. For JavaScript projects, you can generate this file using the oss-cli. For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view.

No option `--disable-license-key-secret` as shown in the setup instructions

Description

When running the command to install the layer into lambda, the instructions list an option that is not supported.

newrelic-lambda layers install \
 -f all  \
 --nr-account-id XXXX --disable-extension  --disable-license-key-secret

The instructions are the page titled Setup AWS Lambda monitoring. I unfortunately can't link directly to it without exposing my account numbers and such.

Steps to Reproduce

Install newrelic-lambda-cli==0.5.2 (0.5.3 is not functioning, see #151).
Run the setup commands to setup logging for a lambda.

Expected Behaviour

That the instructions would either not have the flag listed, or the flag would work.

Relevant Logs / Console output

Your Environment

Python 3.9, macOS Catalina.

Invalid or incomplete schema, unknown type: AttributeMap

Description

When performing the lambda integration installation in a new virtual environment, the "client schema" fails to generate. This used to work in the past.

Steps to Reproduce

1. Set up a virtual environment virtualenv venv
2. Activate the virtual environment source ./venv/binactivate
3. Install the NR CLI pip install newrelic-lambda-cli
4. Configure AWS credentials aws configure
5. Attempt to install the integration newrelic-lambda integrations install -n <label> -a <account_id> -k <personal_api_token>
6. Fail on building a client schema

Expected Behaviour

Expect the command to install the new relic log ingestion lambda onto my aws account.

Relevant Logs / Console output

Validating New Relic credentials
Traceback (most recent call last):
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/bin/newrelic-lambda", line 8, in <module>
    sys.exit(main())
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/newrelic_lambda_cli/utils.py", line 25, in _boto_error_wrapper
    return func(*args, **kwargs)
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/newrelic_lambda_cli/cli/__init__.py", line 26, in main
    cli()
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/newrelic_lambda_cli/cli/integrations.py", line 53, in install
    gql_client = gql.validate_gql_credentials(nr_account_id, nr_api_key, nr_region)
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/newrelic_lambda_cli/gql.py", line 265, in validate_gql_credentials
    return NewRelicGQL(nr_account_id, nr_api_key, nr_region)
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/newrelic_lambda_cli/gql.py", line 45, in __init__
    self.client = Client(transport=transport, fetch_schema_from_transport=True)
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/gql/client.py", line 26, in __init__
    schema = build_client_schema(introspection)
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/graphql/utils/build_client_schema.py", line 315, in build_client_schema
    types=types,
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/graphql/type/schema.py", line 104, in __init__
    self._type_map = GraphQLTypeMap(initial_types)  # type: GraphQLTypeMap
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/graphql/type/typemap.py", line 28, in __init__
    self.update(reduce(self.reducer, types, OrderedDict()))  # type: ignore
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/graphql/type/typemap.py", line 139, in reducer
    reduced_map = cls.reducer(reduced_map, getattr(field, "type", None))
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/graphql/type/typemap.py", line 139, in reducer
    reduced_map = cls.reducer(reduced_map, getattr(field, "type", None))
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/graphql/type/typemap.py", line 139, in reducer
    reduced_map = cls.reducer(reduced_map, getattr(field, "type", None))
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/graphql/type/typemap.py", line 82, in reducer
    return cls.reducer(map, type.of_type)
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/graphql/type/typemap.py", line 106, in reducer
    field_map = type.fields
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/graphql/pyutils/cached_property.py", line 22, in __get__
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/graphql/type/definition.py", line 226, in fields
    return define_field_map(self, self._fields)
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/graphql/type/definition.py", line 240, in define_field_map
    field_map = field_map()
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/graphql/utils/build_client_schema.py", line 163, in <lambda>
    fields=lambda: build_field_def_map(object_introspection),
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/graphql/utils/build_client_schema.py", line 235, in build_field_def_map
    for f in type_introspection.get("fields", [])
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/graphql/utils/build_client_schema.py", line 235, in <listcomp>
    for f in type_introspection.get("fields", [])
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/graphql/utils/build_client_schema.py", line 115, in get_output_type
    output_type = get_type(type_ref)
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/graphql/utils/build_client_schema.py", line 91, in get_type
    return get_named_type(type_ref["name"])
  File "/Users/joliva/Desktop/work/nr-lambda-test/venv/lib/python3.6/site-packages/graphql/utils/build_client_schema.py", line 101, in get_named_type
    "is used in order to build a client schema.".format(type_name)
Exception: Invalid or incomplete schema, unknown type: AttributeMap. Ensure that a full introspection query is used in order to build a client schema.

Your Environment

• Python 3.6.5
• macOS High Sierra 10.13.6

Additional context

This exact same command, with my exact same credentials, used to work when I first tested this roughly two weeks ago. I've also tried using different AWS credentials and a different NR personal token, but am still hitting the same issue.

I've tried running the uninstall command to ensure that there isn't already existing integration. I've also ensure that all resources that were created last time I did this in my AWS account were removed.

InvalidClientTokenId

On my local machine I am trying to set up linking my AWS account to NR Cloud integrations. But I am running into the following error:

Failed actions:
Function: None, Region: None, Error: Failed to set up lambda integration:
An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid.

I have my AWS credentials set up and set my region but I can't seem to resolve this. Any suggestions?

Users assuming roles should not have to use flag to bypass permssions check

Users who have to assume roles to get access to AWS should not have to use the --no-aws-permissions-check to use this solution.

Description

I am trying to setup this app and am running into this exception while using an assumed role.

newrelic-lambda-cli/newrelic_lambda_cli/permissions.py

Line 57 in e4815b3

"Error simulating IAM policies, try passing --no-aws-permissions-check to "

Many large AWS customers also use assumed roles to access their resources.

Steps to Reproduce

1. Login to AWS using your assumed role (ours are Admin with admin perms in AWS)
2. Execute the app newrelic-lambda integrations install --nr-account-id XXXXXXX --linked-account-name newrelic-testing --nr-api-key NRAA-XXXXXXXXXXXXXXXXX
3. receive the error Error: Error simulating IAM policies, try passing --no-aws-permissions-check to override.

Expected Behaviour

The simulation of policies works.

Relevant Logs / Console output

Your Environment

• Operating System and version: macOS Cataline v10.15.4
• Python: 3.7.6

Additional context

From my research about this error when running code snippits locally I found the true error is botocore.errorfactory.InvalidInputException: An error occurred (InvalidInput) when calling the SimulatePrincipalPolicy operation: Invalid Entity Arn: arn:aws:sts::XXXXXXXXXXXX:assumed-role/Administrator/xxxxxxxxxxx does not clearly define entity type and name. I found that SimulatePrinicipalPolicy does not handle assumed accounts. I found a few stackoverflow posts were others were having same error and ended up doing policy simulation a different way.

Maybe this should be an issue for BotoCore but I look forward to hearing maintaining teams input.

AttributeError: 'NewRelicGQL' object has no attribute 'get_linked_account_by_name'

Description

When running the "newrelic-lambda integrations install" command, I get the following error:
AttributeError: 'NewRelicGQL' object has no attribute 'get_linked_account_by_name'

Steps to Reproduce

newrelic-lambda integrations install --nr-account-id 123 --linked-account-name "Some AWS Account" --nr-api-key abc123

Expected Behaviour

No error is thrown.

Relevant Logs / Console output

Validating New Relic credentials
Retrieving integration license key
Checking for a pre-existing link between New Relic and AWS
Traceback (most recent call last):
  File "c:\program files\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\program files\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\dbruk\AppData\Roaming\Python\Python39\Scripts\newrelic-lambda.exe\__main__.py", line 7, in <module>
  File "C:\Users\dbruk\AppData\Roaming\Python\Python39\site-packages\newrelic_lambda_cli\utils.py", line 27, in _boto_error_wrapper
    return func(*args, **kwargs)
  File "C:\Users\dbruk\AppData\Roaming\Python\Python39\site-packages\newrelic_lambda_cli\cli\__init__.py", line 27, in main
    cli()
  File "C:\Users\dbruk\AppData\Roaming\Python\Python39\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\dbruk\AppData\Roaming\Python\Python39\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Users\dbruk\AppData\Roaming\Python\Python39\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\dbruk\AppData\Roaming\Python\Python39\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\dbruk\AppData\Roaming\Python\Python39\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\dbruk\AppData\Roaming\Python\Python39\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\dbruk\AppData\Roaming\Python\Python39\site-packages\click\decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "C:\Users\dbruk\AppData\Roaming\Python\Python39\site-packages\newrelic_lambda_cli\cli\integrations.py", line 132, in install
    integrations.validate_linked_account(gql_client, input)
  File "C:\Users\dbruk\AppData\Roaming\Python\Python39\site-packages\newrelic_lambda_cli\integrations.py", line 482, in validate_linked_account
    account = gql.get_linked_account_by_name(input.linked_account_name)
AttributeError: 'NewRelicGQL' object has no attribute 'get_linked_account_by_name'

Your Environment

Windows 10
Python 3.9.1
pip 20.2.3
newrelic-lambda-cli 0.5.3

Additional context

newrelic-lambda integrations uninstall confirmation prompt should be skippable via a flag

Summary

newrelic-lambda integrations uninstall shows a Y/N confirmation prompt when being used:

This will uninstall the New Relic AWS Lambda log ingestion. Are you sure you want to proceed? [y/N]:

Desired Behaviour

Adding a flag to skip this prompt would help when running this command quickly or through a script.

Possible Solution

Skipping this with a -y or --force flag to the command.

Additional context

I'm running this command in a script, and although I can combine a few other commands, like echo y | newrelic-lambda integrations uninstall, it would be nice if I could leave out that pipe command and leave things as simple as possible.

pip does not install.

pip install --upgrade newrelic-lambda-cli
results in:

Collecting newrelic-lambda-cli
  ERROR: Could not find a version that satisfies the requirement newrelic-lambda-cli (from versions: none)
ERROR: No matching distribution found for newrelic-lambda-cli

pip3 works fine, however.
Mac OS 10.14.6

Errors while setting up nr-lambda

Hey there,

Thanks for the interesting tool. I was hoping to play around with this over the weekend but I ran into a few issues I was unable to address.

Based on your documented example args, I was able to get the CloudFormation stacks applied. I see the following behavior though:

❯ ./newrelic-cloud set-up-lambda-integration --nr-account-id blah --regions us-west-2 \
--linked-account-name "blah" \
--nr-api-key blah \
--nr-license-key blah
Created role [NewRelicLambdaIntegrationRole] with policy [None] in your default AWS account.


Failed actions:
  Function: None, Region: None, Error: Failed to set up lambda integration: 'NoneType' object is not subscriptable

Simply running it a second time immediately afterward, it works.

❯ ./newrelic-cloud set-up-lambda-integration --nr-account-id blah --regions us-west-2 \
--linked-account-name "blah" \
--nr-api-key blah \
--nr-license-key blah
Cloud integrations account [myaccount] already exists in New Relic account [blah].
The AWS Lambda integration is already enabled in Cloud integrations account [myaccount] of New Relic account [blah].
Setting up "newrelic-log-ingestion" function in regions: ['us-west-2']


Completed actions:
  Function: newrelic-log-ingestion, Region: us-west-2

It's unclear to me why it fails the first time. After the successful execution, I can load up the Lambda function in the AWS console and I receive the following error at the top of the page:

An error occurred while listing cloudwatch-logs relations: 2 validation errors detected: Value '' at 'logGroupName' failed to satisfy constraint: Member must have length greater than or equal to 1; Value '' at 'logGroupName' failed to satisfy constraint: Member must satisfy regular expression pattern: [.-_/#A-Za-z0-9]+ (Service: AWSLogs; Status Code: 400; Error Code: InvalidParameterException; Request ID: 84a2692d-7f2e-11e9-bd9f-f916ced906c2)

I have not had a chance to look into it more yet. It's unclear to me if that's preventing the integration from working. When I try to verify the AWS account connection (using the instructions from https://one.newrelic.com/launcher/infra-nerdlets.launcher-lambda-onboarding), it says that my linked account has failed to connect with New Relic.

Let me know if I can provide any extra info to help troubleshoot. For what it's worth, I have no other Lambda functions in the region I'm testing. I reviewed the CloudFormation stack application and there are no errors.

Cheers,
Daniel

Update Default Subscription Filter Pattern

In order to catch timeout log messages the default log subscription filter pattern should be updated to the following:

?REPORT ?NR_LAMBDA_MONITORING ?"Task timed out"

Alert user about competing subscription filter

ImportError: cannot import name 'introspection_query' from 'graphql'

Make exception handling more defensive in a --no-aws-permissions-check context

The CLI by default assumes the AWS user has certain permissions after checking. But with the introduction of --no-aws-permissions-check in #37 this is no longer the case. So we will want to inspect the exception and raise a usage error if we get a status we don't expect to handle.

Make docker image available on DockerHub.

Summary

It would be very convenient to have the built image of the dockerfile available on DockerHub.

Desired Behaviour

I would like to be able to use the docker image without having to build it myself. Hosting the image on dockerhub would enable usage like

docker run -v $HOME/.aws:/home/newrelic-lambda-cli/.aws "newrelic/newrelic-lambda-cli" functions list

Possible Solution

Push the docker image to Dockerhub every time you make a new release.

Additional context

Maybe this image is already hosted somewhere? The instructions for using docker in the readme indicate that the whole repo must be cloned first. I also looked around a bit and couldn't find it hosted in any of the obvious places.

'NewRelicGQL' object has no attribute 'get_linked_account_by_name'

Description

When running basic installation command I'm getting an exception about missing method.

Steps to Reproduce

Try to install new integration:
newrelic-lambda integrations install --nr-account-id <id> --nr-api-key <key>

Expected Behaviour

Should install the integration.

Relevant Logs / Console output

Validating New Relic credentials
Retrieving integration license key
Checking for a pre-existing link between New Relic and AWS
Traceback (most recent call last):
  File "/usr/local/bin/newrelic-lambda", line 10, in <module>
    sys.exit(main())
  File "/Library/Python/3.7/site-packages/newrelic_lambda_cli/utils.py", line 27, in _boto_error_wrapper
    return func(*args, **kwargs)
  File "/Library/Python/3.7/site-packages/newrelic_lambda_cli/cli/__init__.py", line 27, in main
    cli()
  File "/Library/Python/3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Library/Python/3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Library/Python/3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Library/Python/3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Library/Python/3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Library/Python/3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Library/Python/3.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Library/Python/3.7/site-packages/newrelic_lambda_cli/cli/integrations.py", line 132, in install
    integrations.validate_linked_account(gql_client, input)
  File "/Library/Python/3.7/site-packages/newrelic_lambda_cli/integrations.py", line 482, in validate_linked_account
    account = gql.get_linked_account_by_name(input.linked_account_name)
AttributeError: 'NewRelicGQL' object has no attribute 'get_linked_account_by_name'

Your Environment

mac OS Catalina 10.15.7
python: Python 3.7.3

Additional context

Seems like that method get_linked_account_by_name does not exist in NewRelicGQL.

Set PermissionsBoundary on created IAM roles

Summary

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-role.html#cfn-iam-role-permissionsboundary

Desired Behaviour

Permission boundary can be set on created IAM roles.

Possible Solution

Add --aws-permissions-boundary param to provide PermissionBoundary to CloudFormation resources.

Additional context

Corporate policy prevents us creating IAM roles without permission boundary set.

pkg_resources.DistributionNotFound: The 'botocore<1.14.0,>=1.13.23' distribution was not found and is required by boto3

It appears boto3 has released a version that depends on an (as of yet unreleased) version of botocore. We may need to pin boto3 until this is fixed.

Incorporate AWS Region in License Key Secret Stack/Policy Names to Avoid Conflicts

Right now, --enable-license-key-secret creates a CloudFormation stack and IAM policy that is not unique across regions. As such, if a user attempts to create a license key secret in multiple regions they will run into a naming conflict. Such names where conflicts can occur should include the AWS region name as a part of the stack/policy name so that users can create license key secrets in multiple AWS regions for the same AWS account.

Disable AWS Permissions Check By Default

More users than not lack the IAM simulation permissions necessary to perform a permissions check. As such, this feature should be disabled by default.

Refactor Arguments Into Discreet Named Tuples or Data Classes

We have functions that accept a dozen or more arguments. Instead of mapping arguments from one function to the next we should refactor these argument sets into a named tuple or data class.

Integration between AWS and NewRelic failed.

Description

When trying to integrate my lambdas and my new relic account, I am getting some exceptions.

Steps to Reproduce

First Command:

newrelic-lambda integrations install --nr-account-id 1*** --nr-api-key NRAK-*** --linked-account-name "Dev"

Second Command:

newrelic-lambda integrations install --nr-account-id 1*** --nr-api-key NRAK-*** --linked-account-name "Dev" -integration-arn arn:aws:iam::******:role/NewRelic

Relevant Logs / Console output

First command logs:

Validating New Relic credentials
Retrieving integration license key
Checking for a pre-existing link between New Relic and AWS
Creating the AWS role for the New Relic AWS Lambda Integration
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/bin/newrelic-lambda", line 8, in <module>
    sys.exit(main())
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/newrelic_lambda_cli/utils.py", line 27, in _boto_error_wrapper
    return func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/newrelic_lambda_cli/cli/__init__.py", line 27, in main
    cli()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/newrelic_lambda_cli/cli/integrations.py", line 131, in install
    role = integrations.create_integration_role(input)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/newrelic_lambda_cli/integrations.py", line 407, in create_integration_role
    _create_role(input)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/newrelic_lambda_cli/integrations.py", line 69, in _create_role
    role_policy_name = "" if input.role_policy is None else input.role_policy
AttributeError: 'IntegrationInstall' object has no attribute 'role_policy'

Second command logs:

Validating New Relic credentials
Retrieving integration license key
Checking for a pre-existing link between New Relic and AWS
Creating the AWS role for the New Relic AWS Lambda Integration
✔ Found existing AWS IAM role 'arn:aws:iam::********:role/NewRelic', using it with the New Relic Lambda integration
Linking New Relic account to AWS account
✔ Cloud integrations account [Dev] already exists in New Relic account [1****].
Enabling Lambda integration on the link between New Relic and AWS
✔ Integration [id=680417, name=Lambda] has been enabled in Cloud integrations account [Dev] of New Relic account [1****].
Creating the managed secret for the New Relic License Key
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/bin/newrelic-lambda", line 8, in <module>
    sys.exit(main())
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/newrelic_lambda_cli/utils.py", line 27, in _boto_error_wrapper
    return func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/newrelic_lambda_cli/cli/__init__.py", line 27, in main
    cli()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/newrelic_lambda_cli/cli/integrations.py", line 145, in install
    res = integrations.install_license_key(input, nr_license_key)
AttributeError: module 'newrelic_lambda_cli.integrations' has no attribute 'install_license_key'

My Environment

• Python Version: 3.9
• Operating System and version: macOS BigSur 11.0.1
• Aws Cli version: 2.1.10
• New Relic Lambda CLI version: 0.4.9

How to specify the version of the lambda?

Hi we are wanting to add a version only but not latest. If this possible using the newrelic-lambda tool?

$ newrelic-lambda layers install -f <lambda-name>

get_cf_stack_status() does not account for 400 errors

Description

The call that makes this in get_cf_stack_status() expects a 404 from stacks that do not yet exist, although I receive a 400 response for some reason. I would also expect boto3 to return with a 404 in a case where a a stack does not exist, but 400 seems to be the one that is appearing.

Running the same command in version 0.1.3 works and successfully created the lambda role, suggesting that the info/keys I'm providing is valid.

Steps to Reproduce

run newrelic-lambda integrations install in v0.1.6 where the cloudformation stacks for New Relic Log Ingest do not yet exist

Expected Behaviour

Error catching should also catch 400 errors.

Relevant Logs / Console output

newrelic-lambda integrations install --nr-account-id <nrAccountId>--linked-account-name <linkedAccountName> --nr-api-key <nrAPIKey>
Validating New Relic credentials
Retrieving integration license key
Checking for a pre-existing link between New Relic and AWS
Creating the AWS role for the New Relic AWS Lambda Integration
Usage: newrelic-lambda integrations install [OPTIONS]

Error: An error occurred (ValidationError) when calling the DescribeStacks operation: Stack with id NewRelicLambdaIntegrationRole-<nrAccountId> does not exist

Your Environment

Python 3.6.5
macOS High Sierra 10.13.6

Additional context

import botocore into permissions.py

[NOTE]: # import botocore in order to catch errors

Description

[NOTE]: If you are having errors whit permissions the botocore expection doesn't work.
[TIP]: Import botocore

Steps to Reproduce

[NOTE]: try to fail in the execution of newrelic-lambda-cli

Expected Behaviour

[NOTE]: Catch the botocore error instead of NameError: name 'botocore' is not defined

Fix typos in readme and output

filter_pattern instead of filter-pattern in ReadMe; capitalization tweak in permissions output.

[layers install] Add `NEW_RELIC_LAMBDA_EXTENSION_ENABLED=true` environment variable

Summary

New Relic Lambda Layer required NEW_RELIC_LAMBDA_EXTENSION_ENABLED=true environment variable. We have to add this variable manually because laryers install command doesn't add it.

https://docs.newrelic.co.jp/docs/serverless-function-monitoring/aws-lambda-monitoring/enable-lambda-monitoring/enable-serverless-monitoring-using-lambda-layer

Desired Behaviour

The layers install command will add NEW_RELIC_LAMBDA_EXTENSION_ENABLED=true.

Possible Solution

How about add env var here?

Additional context

add note to readme that aws cli output must be json

as title, spent a bit of time digging to find out the cryptic error that the exception causes was due to output being text

AttributeError: module 'botocore.errorfactory' has no attribute 'InvalidInputException'

$ newrelic-lambda functions list --filter not-installed
Traceback (most recent call last):
  File "/home/vagrant/.local/lib/python3.6/site-packages/newrelic_lambda_cli/permissions.py", line 51, in check_permissions
    ContextEntries=context_entries,
  File "/home/vagrant/.local/lib/python3.6/site-packages/botocore/client.py", line 276, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/vagrant/.local/lib/python3.6/site-packages/botocore/client.py", line 586, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (Throttling) when calling the SimulatePrincipalPolicy operation (reached max retries: 4): Rate exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vagrant/.local/bin/newrelic-lambda", line 11, in <module>
    sys.exit(main())
  File "/home/vagrant/.local/lib/python3.6/site-packages/newrelic_lambda_cli/utils.py", line 24, in _boto_error_wrapper
    return func(*args, **kwargs)
  File "/home/vagrant/.local/lib/python3.6/site-packages/newrelic_lambda_cli/cli/__init__.py", line 25, in main
    cli()
  File "/home/vagrant/.local/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/vagrant/.local/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/vagrant/.local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/vagrant/.local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/vagrant/.local/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/vagrant/.local/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/vagrant/.local/lib/python3.6/site-packages/newrelic_lambda_cli/cli/functions.py", line 36, in list
    permissions.ensure_lambda_list_permissions(session)
  File "/home/vagrant/.local/lib/python3.6/site-packages/newrelic_lambda_cli/permissions.py", line 198, in ensure_lambda_list_permissions
    needed_permissions = check_permissions(session, actions=["lambda:ListFunctions"])
  File "/home/vagrant/.local/lib/python3.6/site-packages/newrelic_lambda_cli/permissions.py", line 53, in check_permissions
    except botocore.errorfactory.InvalidInputException:
AttributeError: module 'botocore.errorfactory' has no attribute 'InvalidInputException'

Users must be able to manage custom policies with CloudFormation

The creation of custom policies in a CloudFormation stack means users are unable to use CloudFormation to manage that custom policy themselves.

Summary

Since the custom policy is created in the nr-lambda-integration-role.yaml template, CloudFormation will not allow creation or maintenance of that role using CloudFormation created by the user. If a user wants to manage a policy, they likely are in a fairly regimented environment and have elected to exempt themselves from a part of the newrelic-lambda-cli process knowingly. It would be better to provide guidance/requirements for that custom policy and step out of the way. The user has already decided to decline the nicety of having the newrelic-lambda-cli provision it for them.

Desired Behaviour

• I should be able to manage the custom integration role for lambda integration through my own custom CloudFormation, and should be able to supply the name of the already extant role to the newrelic-lambda-cli

Possible Solution

• create an additional conditional in the template, which will use a customer-managed, customer-supplied role instead of creating one
• the existing code/template can still be used, if desired, to support users who have availed themselves of the current process

Additional context

Users who are in a position to create custom policies are likely there because they are subject to heightened security oversight/review and are prohibited from providing the blanket ReadOnly policy which is used by default. My gut feeling is that users with a heightened security review requirement, are also likely required to manage AWS resources through an Infrastructure as Code (IaC) solution such as CloudFormation. This is my situation, anyway, and I doubt I am unique in this.

Support Alternative to AWS Secrets Manager

Not all users have the permissions necessary to use AWS Secrets Manager. As such, the CLI should support an alternative method for configuring license keys when using the Lambda extension.

subscription install fails without explanation

[2020-03-11T19:56:45.606Z] + newrelic-lambda subscriptions install --no-aws-permissions-check --function arn:aws:lambda:us-east-1:xxxxx:function:yyyyy --function arn:aws:lambda:us-east-1:xxxxx:function:zzzzz
[2020-03-11T19:56:46.620Z] Found log subscription for 'arn:aws:lambda:us-east-1:xxxxx:function:yyyy', verifying configuration
[2020-03-11T19:56:46.620Z] Found log subscription for 'arn:aws:lambda:us-east-1:xxxxx:function:zzzzz', verifying configuration
[2020-03-11T19:56:46.620Z] ✖️ Install Incomplete. See messages above for details.

The installation fails without any clear explanation. Is it failing because the subscription already exists?

New Relic CLI v0.1.24
AWS CLI v1.18.19

Grant Secret Manager Permissions For Functions Using Lambda Extensions

Related to #109

The newrelic-lambda layers install command should have a flag that updates the function to grant permissions to secrets manager. By default the New Relic Lambda Extension uses secrets manager to retrieve the New Relic license key.

Will need a fallback to setting an NEW_RELIC_LICENSE_KEY environment variable in the event the user is not using secrets manager or doesn't have the permissions necessary to configure it.

Push docker image to public repo

[NOTE]: # Push docker image to public repo

Summary

Since I'm using newrelic-lambda-cli as docker I need to clone repo and build the image.
So, would be useful to have this image in a public registry like newrelic/cli or include newrelic-lambda-cli as command in newrelic/cli .

Provide exit code != 0 on error

Currently the exit code of the application is always 0. This makes it very hard to detect an error when using automation to setup an AWS region.

Expectation:

• the exit code of the application is not 0 when an error occurred

➜  nr-lambda-onboarding git:(master) ./newrelic-cloud set-up-lambda-integration --nr-account-id xxxxxxxx \
 --linked-account-name "xxxxxxxxxxxx" \
 --nr-api-key xxxxxxxxxxxxxxxxxxxxxxxxx \
 --regions "ap-northeast-1"
Validating New Relic credentials and retrieving integration license key
Checking for a preexisting link between NR and AWS, and validating the AWS CLI credentials
Creating the AWS role for the New Relic integration
Linking your New Relic account to your AWS account
Cloud integrations account [xxxxxxxxxxx] already exists in New Relic account [xxxxxxxxx].
Enabling Lambda integration on the link between New Relic and AWS
The AWS Lambda integration is already enabled in Cloud integrations account [xxxxxxxxxx] of New Relic account [xxxxxxxxxx].
Creating newrelic-log-ingestion Lambda function in your AWS account
Setting up "newrelic-log-ingestion" function in region: ap-northeast-1

Lambda integration is set up.

Next, you'll need to instrument your Lambda functions, and use 
	./newrelic-cloud stream-lambda-logs \
		--functions <function_name>... \
		--regions <aws_region>...

to connect your instrumented Lambda functions to New Relic


Failed actions:
  Function: newrelic-log-ingestion, Region: ap-northeast-1, Error: Failed to create "newrelic-log-ingestion" function: 
Waiter StackCreateComplete failed: Waiter encountered a terminal failure state
.
➜  nr-lambda-onboarding git:(master) echo $?
0

not-installed filter doesn't seem to work

The "installed" filter is working fine, along with the list functions command, but the -f not-installed command is just blank even though there are functions that are not yet instrumented.

Increase visibility into New Relic Lambda Layer function changes

Summary

Currently, the recommended way to install the New Relic Lambda Layer is to run the New Relic CLI command:

newrelic-lambda layers install --function <function name> --nr-account-id <new relic account ID>

This configures the latest New Relic Lambda Function as a layer to the Lambda function we specified. But we aren't able to see what New Relic Function versions exist and what changes actually occurred between them.

Desired Behaviour

The above is great but ideally, what I would like to also be able to see is a way for us to see all New Relic Lambda Function versions and a changelog of sorts that shows us the changes made whenever a new version is published.

Something that elaborates on what's commited here: https://github.com/newrelic/newrelic-lambda-layers

Possible Solution

Ideally, it would be nice to have it built into the CLI itself as a means of listing New Relic Lambda function versions as well as being able to see relevant changes corresponding to the Lambda function versions.

Alternatively, I'd be okay with even just having the newrelic-lambda-layers repository having some type of mapping that would allow us to see which New Relic Lambda Function maps to a specific release. That would at least give us an idea of whether we're using the latest version and what changes occurred between the version we were using.

Additional context

This would allow people to know exactly what they're updating to and for. Additionally, this would make it so we wouldn't have to guess on which version to use, why we should use it, and whether a version even still exists.

AWS credentials possibilities are restricted to small use cases

Description

Firstly apologies if the following isnt relevant and I've misinterpreted the code and docs and you do do what I am asking.

Looking through the code and docs, I believe you specify an aws profile or key/secret key as the primary ways of authenticating your boto3 session with aws. If I've read this correctly, this is far too restrictive and undermines the fundamental good practice of using a default credentials provider and letting a user configure credentials outside your tool according to their needs according to the default chain of 5 (not 2) possible credentials locations (and in boto the default 8 lookup locations) - see links below - which is default aws sdk behaviour and much preferred because it would allow newrelic-lambda-api to run on aws codepipeline, aws codebuild, ecs, ec2 etc. etc. without the need for managing profiles artificially.

In essence our fundamental problem is running newrelic-lambda-api as a codebuild step in a codepipeline so once lambdas are deployed I can use newrelic-lambda-api to attach newrelic integration to them as part of a codepipeline build step

Steps to Reproduce

See the above - set up your own codepipeline and codebuild, and then in a codebuild step (which is configured to use an IAM role) try and use the tool to use the default IAM credentials passed through the http://169.254.169.254 locallink aws address and autowired up as part of the standard AWS ecosystem. newrelic-lambda-api expects a profile on the box or an accesskey/secretaccesskey but the only options available are using an accesskey/secretaccesskey/sessiontoken extracted from the 169.254.169.254 address, or artificially setting up a profile on the box to do this

Expected Behaviour

I would expect like all tools to support the default credentials provider in the boto/python aws sdk (like all sdks) which allows the external user the ability to configure which one of 5/8 standard credentials locations to use, and use them to run the tool - so that it can be "autowired up" with the right credentials if running on AWS components such as codepipeline or many many other AWS services.

Relevant Logs / Console output

Your Environment

AWS Codepipeline, AWS Codebuild

Additional context

See https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default

See https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html for boto's 8 lookup paths. 8 is the one I want to work.

newrelic-lambda integrations uninstall should also uninstall the integration role

Summary

newrelic-lambda integrations uninstall only uninstalls the NewRelicLogIngestion cloudformation stack.

Desired Behaviour

newrelic-lambda integrations install will install both:

• NewRelicLambdaIntegrationRole
• NewRelicLogIngestion

newrelic-lambda integrations uninstall should do the opposite and uninstall both. It seems to only uninstall NewRelicLogIngestion and leaves NewRelicLambdaIntegrationRole active.

Possible Solution

In https://github.com/newrelic/newrelic-lambda-cli/blob/master/newrelic_lambda_cli/cli/integrations.py#L89 also use the remove_integration_role() function.

Additional context

My team is creating a demo that spins up a serverless application on demand. This serverless application also includes a demo for New Relic Lambda monitoring so that other teams can quickly spin up a test space for it, play with it, and see how it works.

To do this, we've created a script that spins up all resources automatically and another that tears them all down without any manual intervention. We can use boto3 to also tear down the integration role, but it would be nice if the NR CLI did it for us.