aws-samples/aws-health-aware

AHA is an incident management & communication framework to provide real-time alert customers when there are active AWS event(s). For customers with AWS Organizations, customers can get aggregated active account level events of all the accounts in the Organization. Customers not using AWS Organizations still benefit alerting at the account level.

Table of Contents

• Introduction
• What's New
• Architecture
  • Single Region
  • Multi Region
  • Created AWS Resources
• Configuring an Endpoint
  • Creating a Amazon Chime Webhook URL
  • Creating a Slack Webhook URL
  • Creating a Microsoft Teams Webhook URL
  • Configuring an Email
  • Creating a Amazon EventBridge Ingestion ARN
• Deployment Options
  • CloudFormation
  • AHA Without AWS Organizations using CloudFormation
    • Prerequisites
    • Deployment
  • AHA With AWS Organizations on Management Account using CloudFormation
    • Prerequisites
    • Deployment
  • AHA With AWS Organizations on Member Account using CloudFormation
    • Prerequisites
    • Deployment
  • Terraform
  • AHA Without AWS Organizations using Terraform
    • Prerequisites
    • Deployment - Terraform
  • AHA WITH AWS Organizations on Management Account using Terraform
    • Deployment - Terraform
  • AHA WITH AWS Organizations on Member Account using Terraform
    • Deployment - Terraform
• Updating using CloudFormation
• Updating using Terraform
• New Features
• Troubleshooting

Introduction

AWS Health Aware (AHA) is an automated notification tool for sending well-formatted AWS Health Alerts to Amazon Chime, Slack, Microsoft Teams, E-mail or an AWS Eventbridge compatible endpoint as long as you have Business or Enterprise Support.

What's New

Release 2.2 introduces an updated schema for Health events delivered to an EventBridge bus. This allows simplified matching of events which you can then consume with other AWS services or SaaS solutions. Read more about the new feature and how to filter events using EventBridge.

Architecture

Single Region

Multi Region

Created AWS Resources

Resource	Description
DynamoDBTable	DynamoDB Table used to store Event ARNs, updates and TTL
ChimeChannelSecret	Webhook URL for Amazon Chime stored in AWS Secrets Manager
EventBusNameSecret	EventBus ARN for Amazon EventBridge stored in AWS Secrets Manager
LambdaExecutionRole	IAM role used for LambdaFunction
LambdaFunction	Main Lambda function that reads from AWS Health API, sends to endpoints and writes to DynamoDB
LambdaSchedule	Amazon EventBridge rule that runs every min to invoke LambdaFunction
LambdaSchedulePermission	IAM Role used for LambdaSchedule
MicrosoftChannelSecret	Webhook URL for Microsoft Teams stored in AWS Secrets Manager
SlackChannelSecret	Webhook URL for Slack stored in AWS Secrets Manager

Configuring an Endpoint

AHA can send to multiple endpoints (webhook URLs, Email or EventBridge). To use any of these you'll need to set it up before-hand as some of these are done on 3rd party websites. We'll go over some of the common ones here.

Creating a Amazon Chime Webhook URL

You will need to have access to create a Amazon Chime room and manage webhooks.

1. Create a new chat room for events (i.e. aws_events).
2. In the chat room created in step 1, click on the gear icon and click manage webhooks and bots.
3. Click Add webhook.
4. Type a name for the bot (e.g. AWS Health Bot) and click Create.
5. Click Copy URL, we will need it for the deployment.

Creating a Slack Webhook URL

You will need to have access to add a new channel and app to your Slack Workspace.

Webhook

1. Create a new channel for events (i.e. aws_events)
2. In your browser go to: workspace-name.slack.com/apps where workspace-name is the name of your Slack Workspace.
3. In the search bar, search for: Incoming Webhooks and click on it.
4. Click on Add to Slack.
5. From the dropdown click on the channel your created in step 1 and click Add Incoming Webhooks integration.
6. From this page you can change the name of the webhook (i.e. AWS Bot), the icon/emoji to use, etc.
7. For the deployment we will need the Webhook URL.

Workflow

1. Create a new channel for events (i.e. aws_events)
2. Within Slack click on your workspace name drop down arrow in the upper left. click on Tools > Workflow Builder
3. Click Create in the upper right hand corner of the Workflow Builder and give your workflow a name click next.
4. Click on select next to Webhook and then click add variable add the following variables one at a time in the Key section. All data type will be text:
   -text
   -accounts
   -resources
   -service
   -region
   -status
   -start_time
   -event_arn
   -updates
5. When done you should have 9 variables, double check them as they are case sensitive and will be referenced. When checked click on done and next.
6. Click on add step and then on the add a workflow step click add next to send a message.
7. Under send this message to: select the channel you created in Step 1 in message text you can should recreate this following:
   
8. Click save and the click publish
9. For the deployment we will need the Webhook URL.

Creating a Microsoft Teams Webhook URL

You will need to have access to add a new channel and app to your Microsoft Teams channel.

1. Create a new channel for events (i.e. aws_events)
2. Within your Microsoft Team go to Apps
3. In the search bar, search for: Incoming Webhook and click on it.
4. Click on Add to team.
5. Type in the name of your on the channel your created in step 1 and click Set up a connector.
6. From this page you can change the name of the webhook (i.e. AWS Bot), the icon/emoji to use, etc. Click Create when done.
7. For the deployment we will need the webhook URL that is presented.

Configuring an Email

1. You'll be able to send email alerts to one or many addresses. However, you must first verify the email(s) in the Simple Email Service (SES) console.
2. AHA utilizes Amazon SES so all you need is to enter in a To: address and a From: address.
3. You may have to allow a rule in your environment so that the emails don't get labeled as SPAM. This will be something you have to congfigure on your own.

Creating a Amazon EventBridge Ingestion ARN

Only required if you are going to be using EventBridge, you can create new with the instructions below or use an existing one.

1. In the AWS Console, search for Amazon EventBridge.
2. On the left hand side, click Event buses.
3. Under Custom event bus click Create event bus
4. Give your Event bus a name and click Create.
5. For the deployment we will need the Name of the Event bus (not the ARN, e.g. aha-eb01).

Using AWS Health Delegated Administrator with AHA

On 2023-07-27, AWS Health released the Delegated Admin feature. Using this feature, you can deploy AHA in a Member Account without added permissions in the Org Management account.

To enable this feature:

1. Know the AWS Account ID of the Member Account you want to enable as a delegated administrator for AWS Health (e.g. 123456789012)
2. In the Org Management Account, run the command aws organizations register-delegated-administrator --account-id ACCOUNT_ID --service-principal health.amazonaws.com replacing ACCOUNT_ID with the ID of your Member Account
3. Deploy AHA in the Member Account using the steps for
4. AHA for users who ARE using AWS Organizations (CloudFormation)
5. AHA for users who ARE using AWS Organizations (Terraform)

Read more: https://docs.aws.amazon.com/health/latest/ug/delegated-administrator-organizational-view.html

Deployment Options

CloudFormation

There are 3 available ways to deploy AHA, all are done via the same CloudFormation template to make deployment as easy as possible.

The 3 deployment methods for AHA are:

1. AHA for users WITHOUT AWS Organizations: Users NOT using AWS Organizations.
2. AHA for users WITH AWS Organizations (Management Account): Users who ARE using AWS Organizations and deploying in the top-level management account.
3. AHA for users WITH AWS Organizations (Member Account): Users who ARE using AWS Organizations and deploying in a member account in the organization to assume a role in the top-level management account.

AHA Without AWS Organizations using CloudFormation

Prerequisites

1. Have at least 1 endpoint configured (you can have multiple)
2. Have access to deploy Cloudformation Templates with the following resources: AWS IAM policies, Amazon DynamoDB Tables, AWS Lambda, Amazon EventBridge and AWS Secrets Manager.
3. If using Multi-Region, you must deploy the following 2 CloudFormation templates to allow the Stackset deployment to deploy resources even if you have full administrator privileges, you still need to follow these steps.

• In CloudFormation Console create a stack with new resources from the following S3 URL: https://s3.amazonaws.com/cloudformation-stackset-sample-templates-us-east-1/AWSCloudFormationStackSetAdministrationRole.yml - this will allows CFT Stacksets to launch AHA in another region
• Launch the stack.
• In CloudFormation Console create a stack with new resources from the following S3 URL: https://s3.amazonaws.com/cloudformation-stackset-sample-templates-us-east-1/AWSCloudFormationStackSetExecutionRole.yml) - In AdministratorAccountId type in the 12 digit account number you're running the solution in (e.g. 000123456789)
• Launch the stack.

Deployment

1. Clone the AHA package that from this repository. If you're not familiar with the process, here is some documentation. The URL to clone is in the upper right-hand corner labeled Clone uri
2. In the root of this package you'll have two files; handler.py and messagegenerator.py. Use your tool of choice to zip them both up and name them with a unique name (e.g. aha-v1.8.zip). Note: Putting the version number in the name will make upgrading AHA seamless.
3. Upload the .zip you created in Step 1 to an S3 in the same region you plan to deploy this in.
4. In your AWS console go to CloudFormation.
5. In the CloudFormation console click Create stack > With new resources (standard).
6. Under Template Source click Upload a template file and click Choose file and select CFN_DEPLOY_AHA.yml Click Next.

• In Stack name type a stack name (i.e. AHA-Deployment).
• In AWSOrganizationsEnabled leave it set to default which is No. If you do have AWS Organizations enabled and you want to aggregate across all your accounts, you should be following the steps for AHA for users who ARE using AWS Organizations (Management Account) or AHA for users WITH AWS Organizations (Member Account)
• In AWSHealthEventType select whether you want to receive all event types or only issues.
• In S3Bucket type just the bucket name of the S3 bucket used in step 3 (e.g. my-aha-bucket).
• In S3Key type just the name of the .zip file you created in Step 2 (e.g. aha-v1.8.zip).
• In the Communications Channels section enter the URLs, Emails and/or ARN of the endpoints you configured previously.
• In the Email Setup section enter the From and To Email addresses as well as the Email subject. If you aren't configuring email, just leave it as is.
• In EventSearchBack enter in the amount of hours you want to search back for events. Default is 1 hour.
• In Regions enter in the regions you want to search for events in. Default is all regions. You can filter for up to 10, comma separated (e.g. us-east-1, us-east-2).
• In ARN of the AWS Organizations Management Account assume role leave it set to default None as this is only for customers using AWS Organizations.
• In Deploy in secondary region? select another region to deploy AHA in. Otherwise leave to default No.

7. Scroll to the bottom and click Next.
8. Scroll to the bottom and click Next again.
9. Scroll to the bottom and click the checkbox and click Create stack.
10. Wait until Status changes to CREATE_COMPLETE (roughly 2-4 minutes or if deploying in a secondary region, it can take up to 30 minutes).

AHA With AWS Organizations on Management Account using CloudFormation

Prerequisites

1. Enable Health Organizational View from the console, so that you can aggregate all Personal Health Dashboard (PHD) events for all accounts in your AWS Organization.
2. Have at least 1 endpoint configured (you can have multiple)
3. Have access to deploy Cloudformation Templates with the following resources: AWS IAM policies, Amazon DynamoDB Tables, AWS Lambda, Amazon EventBridge and AWS Secrets Manager in the AWS Organizations Master Account.
4. If using Multi-Region, you must deploy the following 2 CloudFormation templates to allow the Stackset deployment to deploy resources even if you have full administrator privileges, you still need to follow these steps.

• In CloudFormation Console create a stack with new resources from the following S3 URL: https://s3.amazonaws.com/cloudformation-stackset-sample-templates-us-east-1/AWSCloudFormationStackSetAdministrationRole.yml - this will allows CFT Stacksets to launch AHA in another region
• Launch the stack.
• In CloudFormation Console create a stack with new resources from the following S3 URL: https://s3.amazonaws.com/cloudformation-stackset-sample-templates-us-east-1/AWSCloudFormationStackSetExecutionRole.yml) - In AdministratorAccountId type in the 12 digit account number you're running the solution in (e.g. 000123456789)
• Launch the stack.

Deployment

1. Clone the AHA package that from this repository. If you're not familiar with the process, here is some documentation. The URL to clone is in the upper right-hand corner labeled Clone uri
2. In the root of this package you'll have two files; handler.py and messagegenerator.py. Use your tool of choice to zip them both up and name them with a unique name (e.g. aha-v1.8.zip). Note: Putting the version number in the name will make upgrading AHA seamless.
3. Upload the .zip you created in Step 1 to an S3 in the same region you plan to deploy this in.
4. In your AWS console go to CloudFormation.
5. In the CloudFormation console click Create stack > With new resources (standard).
6. Under Template Source click Upload a template file and click Choose file and select CFN_DEPLOY_AHA.yml Click Next.

• In Stack name type a stack name (i.e. AHA-Deployment).
• In AWSOrganizationsEnabled change the dropdown to Yes. If you do NOT have AWS Organizations enabled you should be following the steps for AHA for users who are NOT using AWS Organizations
• In AWSHealthEventType select whether you want to receive all event types or only issues.
• In S3Bucket type just the bucket name of the S3 bucket used in step 3 (e.g. my-aha-bucket).
• In S3Key type just the name of the .zip file you created in Step 2 (e.g. aha-v1.8.zip).
• In the Communications Channels section enter the URLs, Emails and/or ARN of the endpoints you configured previously.
• In the Email Setup section enter the From and To Email addresses as well as the Email subject. If you aren't configuring email, just leave it as is.
• In EventSearchBack enter in the amount of hours you want to search back for events. Default is 1 hour.
• In Regions enter in the regions you want to search for events in. Default is all regions. You can filter for up to 10, comma separated with (e.g. us-east-1, us-east-2).
• In ARN of the AWS Organizations Management Account assume role leave it set to default None.
• In Deploy in secondary region? select another region to deploy AHA in. Otherwise leave to default No.

7. Scroll to the bottom and click Next.
8. Scroll to the bottom and click Next again.
9. Scroll to the bottom and click the checkbox and click Create stack.
10. Wait until Status changes to CREATE_COMPLETE (roughly 2-4 minutes or if deploying in a secondary region, it can take up to 30 minutes).

AHA With AWS Organizations on Member Account using CloudFormation

Note: On 2023-07-27, AWS Health released the Delegated Admin feature which enables AHA deployments in member accounts without the extra steps below. See: Using AWS Health Delegated Administrator with AHA

Prerequisites

1. Enable Health Organizational View from the console, so that you can aggregate all Personal Health Dashboard (PHD) events for all accounts in your AWS Organization.
2. Have at least 1 endpoint configured (you can have multiple)
3. Have access to deploy Cloudformation Templates with the following resource: AWS IAM policies in the AWS Organizations Master Account.
4. If using Multi-Region, you must deploy the following 2 CloudFormation templates in the Member Account to allow the Stackset deployment to deploy resources even if you have full administrator privileges, you still need to follow these steps.

• In CloudFormation Console create a stack with new resources from the following S3 URL: https://s3.amazonaws.com/cloudformation-stackset-sample-templates-us-east-1/AWSCloudFormationStackSetAdministrationRole.yml - this will allows CFT Stacksets to launch AHA in another region
• Launch the stack.
• In CloudFormation Console create a stack with new resources from the following S3 URL: https://s3.amazonaws.com/cloudformation-stackset-sample-templates-us-east-1/AWSCloudFormationStackSetExecutionRole.yml) - In AdministratorAccountId type in the 12 digit account number you're running the solution in (e.g. 000123456789)
• Launch the stack.

Deployment

1. Clone the AHA package that from this repository. If you're not familiar with the process, here is some documentation. The URL to clone is in the upper right-hand corner labeled Clone uri
2. In your top-level management account AWS console go to CloudFormation
3. In the CloudFormation console click Create stack > With new resources (standard).
4. Under Template Source click Upload a template file and click Choose file and select CFN_MGMT_ROLE.yml Click Next.

• In Stack name type a stack name (i.e. aha-assume-role).
• In OrgMemberAccountId put in the account id of the member account you plan to run AHA in (e.g. 000123456789).

5. Scroll to the bottom and click Next.
6. Scroll to the bottom and click Next again.
7. Scroll to the bottom and click the checkbox and click Create stack.
8. Wait until Status changes to CREATE_COMPLETE (roughly 1-2 minutes). This will create an IAM role with the necessary AWS Organizations and AWS Health API permissions for the member account to assume.
9. In the Outputs tab, there will be a value for AWSHealthAwareRoleForPHDEventsArn (e.g. arn:aws:iam::000123456789:role/aha-org-role-AWSHealthAwareRoleForPHDEvents-ABCSDE12201), copy that down as you will need it for step 16.
10. Back In the root of the package you downloaded/cloned you'll have two files; handler.py and messagegenerator.py. Use your tool of choice to zip them both up and name them with a unique name (e.g. aha-v1.8.zip). Note: Putting the version number in the name will make upgrading AHA seamless.
11. Upload the .zip you created in Step 11 to an S3 in the same region you plan to deploy this in.
12. Login to the member account you plan to deploy this in and in your AWS console go to CloudFormation.
13. In the CloudFormation console click Create stack > With new resources (standard).
14. Under Template Source click Upload a template file and click Choose file and select CFN_DEPLOY_AHA.yml Click Next.

• In Stack name type a stack name (i.e. AHA-Deployment).
• In AWSOrganizationsEnabled change the dropdown to Yes. If you do NOT have AWS Organizations enabled you should be following the steps for AHA for users who are NOT using AWS Organizations
• In AWSHealthEventType select whether you want to receive all event types or only issues.
• In S3Bucket type just the bucket name of the S3 bucket used in step 12 (e.g. my-aha-bucket).
• In S3Key type just the name of the .zip file you created in Step 11 (e.g. aha-v1.8.zip).
• In the Communications Channels section enter the URLs, Emails and/or ARN of the endpoints you configured previously.
• In the Email Setup section enter the From and To Email addresses as well as the Email subject. If you aren't configuring email, just leave it as is.
• In EventSearchBack enter in the amount of hours you want to search back for events. Default is 1 hour.
• In Regions enter in the regions you want to search for events in. Default is all regions. You can filter for up to 10, comma separated with (e.g. us-east-1, us-east-2).
• In ManagementAccountRoleArn enter in the full IAM arn from step 10 (e.g. arn:aws:iam::000123456789:role/aha-org-role-AWSHealthAwareRoleForPHDEvents-ABCSDE12201)
• In Deploy in secondary region? select another region to deploy AHA in. Otherwise leave to default No.

15. Scroll to the bottom and click Next.
16. Scroll to the bottom and click Next again.
17. Scroll to the bottom and click the checkbox and click Create stack.
18. Wait until Status changes to CREATE_COMPLETE (roughly 2-4 minutes or if deploying in a secondary region, it can take up to 30 minutes).

Terraform

There are 3 available ways to deploy AHA, all are done via the same Terraform template to make deployment as easy as possible.

**NOTE: ** AHA code is tested with Terraform version v1.0.9, please make sure to have minimum terraform verson of v1.0.9 installed.

The 3 deployment methods for AHA are:

1. AHA for users NOT using AWS Organizations using Terraform: Users NOT using AWS Organizations.
2. AHA for users WITH AWS Organizations using Terraform (Management Account): Users who ARE using AWS Organizations and deploying in the top-level management account.
3. AHA for users WITH AWS Organizations using Terraform (Member Account): Users who ARE using AWS Organizations and deploying in a member account in the organization to assume a role in the top-level management account.

AHA Without AWS Organizations using Terraform

Prerequisites

1. Have at least 1 endpoint configured (you can have multiple)
2. Have access to deploy Terraform Templates with the following resources: AWS IAM policies, Amazon DynamoDB Tables, AWS Lambda, Amazon EventBridge and AWS Secrets Manager.

**NOTE: ** For Multi region deployment, DynamoDB table will be created with PAY_PER_REQUEST billing mode insted of PROVISIONED due to limitation with terraform.

Deployment - Terraform

1. Clone the AHA package that from this repository. If you're not familiar with the process, here is some documentation. The URL to clone is in the upper right-hand corner labeled Clone uri

$ git clone https://github.com/aws-samples/aws-health-aware.git
$ cd aws-health-aware/terraform/Terraform_DEPLOY_AHA

2. Update parameters file terraform.tfvars as below

• aha_primary_region - change to region where you want to deploy AHA solution
• aha_secondary_region - Required if needed to deploy in AHA solution in multiple regions, change to another region (Secondary) where you want to deploy AHA solution, Otherwise leave to default empty value.
• AWSOrganizationsEnabled - Leave it to default which is No. If you do have AWS Organizations enabled and you want to aggregate across all your accounts, you should be following the steps for AHA for users who ARE using AWS Organizations (Management Account)] or AHA for users WITH AWS Organizations (Member Account)
• AWSHealthEventType - select whether you want to receive all event types or only issues.
• Communications Channels section - enter the URLs, Emails and/or ARN of the endpoints you configured previously.
• Email Setup section - enter the From and To Email addresses as well as the Email subject. If you aren't configuring email, just leave it as is.
• EventSearchBack - enter in the amount of hours you want to search back for events. Default is 1 hour.
• Regions - enter in the regions you want to search for events in. Default is all regions. You can filter for up to 10, comma separated (e.g. us-east-1, us-east-2).
• ManagementAccountRoleArn - Leave it default empty value
• ExcludeAccountIDs - type just the name of the .csv file you want to upload if needed to exclude accounts from monitoring, else leave it to empty.
• ManagementAccountRoleArn - In ARN of the AWS Organizations Management Account assume role leave it set to default None as this is only for customers using AWS Organizations.

3. Deploy the solution using terraform commands below.

$ terraform init
$ terraform plan
$ terraform apply

AHA WITH AWS Organizations on Management Account using Terraform

1. Enable Health Organizational View from the console, so that you can aggregate all Personal Health Dashboard (PHD) events for all accounts in your AWS Organization.
2. Have at least 1 endpoint configured (you can have multiple)

**NOTE: ** For Multi region deployment, DynamoDB table will be created with PAY_PER_REQUEST billing mode insted of PROVISIONED due to limitation with terraform.

Deployment - Terraform

1. Clone the AHA package that from this repository. If you're not familiar with the process, here is some documentation. The URL to clone is in the upper right-hand corner labeled Clone uri

$ git clone https://github.com/aws-samples/aws-health-aware.git
$ cd aws-health-aware/terraform/Terraform_DEPLOY_AHA

5. Update parameters file terraform.tfvars as below

• aha_primary_region - change to region where you want to deploy AHA solution
• aha_secondary_region - Required if needed to deploy in AHA solution in multiple regions, change to another region (Secondary) where you want to deploy AHA solution, Otherwise leave to default empty value.
• AWSOrganizationsEnabled - change the value to Yes. If you do NOT have AWS Organizations enabled you should be following the steps for AHA for users who are NOT using AWS Organizations
• AWSHealthEventType - select whether you want to receive all event types or only issues.
• Communications Channels section - enter the URLs, Emails and/or ARN of the endpoints you configured previously.
• Email Setup section - enter the From and To Email addresses as well as the Email subject. If you aren't configuring email, just leave it as is.
• EventSearchBack - enter in the amount of hours you want to search back for events. Default is 1 hour.
• Regions enter in the regions you want to search for events in. Default is all regions. You can filter for up to 10, comma separated (e.g. us-east-1, us-east-2).
• ManagementAccountRoleArn - Leave it default empty value
• S3Bucket - type just the name of the S3 bucket where exclude file .csv you upload. leave it empty if exclude Account feature is not used.
• ExcludeAccountIDs - type just the name of the .csv file you want to upload if needed to exclude accounts from monitoring, else leave it to empty.
• ManagementAccountRoleArn - In ARN of the AWS Organizations Management Account assume role leave it set to default None, unless you are using a member account instead of the management account. Instructions for this configuration are in the next section.

3. Deploy the solution using terraform commands below.

$ terraform init
$ terraform plan
$ terraform apply

AHA WITH AWS Organizations on Member Account using Terraform

Note: On 2023-07-27, AWS Health released the Delegated Admin feature which enables AHA deployments in member accounts without the extra steps below. See: Using AWS Health Delegated Administrator with AHA

1. Enable Health Organizational View from the console, so that you can aggregate all Personal Health Dashboard (PHD) events for all accounts in your AWS Organization.
2. Have at least 1 endpoint configured (you can have multiple)

**NOTE: ** For Multi region deployment, DynamoDB table will be created with PAY_PER_REQUEST billing mode insted of PROVISIONED due to limitation with terraform.

Deployment - Terraform

1. Clone the AHA package that from this repository. If you're not familiar with the process, here is some documentation. The URL to clone is in the upper right-hand corner labeled Clone uri

$ git clone https://github.com/aws-samples/aws-health-aware.git

2. In your top-level management account deploy terraform module Terraform_MGMT_ROLE.tf to create Cross-Account Role for PHD access

$ cd aws-health-aware/terraform/Terraform_MGMT_ROLE
$ terraform init
$ terraform plan
$ terraform apply
 Input *OrgMemberAccountId*  Enter the account id of the member account you plan to run AHA in (e.g. 000123456789).

3. Wait for deployment to complete. This will create an IAM role with the necessary AWS Organizations and AWS Health API permissions for the member account to assume. and note the AWSHealthAwareRoleForPHDEventsArn role name, this will be used during deploying solution in member account
4. In the Outputs section, there will be a value for AWSHealthAwareRoleForPHDEventsArn (e.g. arn:aws:iam::000123456789:role/aha-org-role-AWSHealthAwareRoleForPHDEvents-ABCSDE12201), copy that down as you will need to update params file (variable ManagementAccountRoleArn).
5. Change directory to terraform/Terraform_DEPLOY_AHA to deploy the solution
6. Update parameters file terraform.tfvars as below

• aha_primary_region - change to region where you want to deploy AHA solution
• aha_secondary_region - Required if needed to deploy in AHA solution in multiple regions, change to another region (Secondary) where you want to deploy AHA solution, Otherwise leave to default empty value.
• AWSOrganizationsEnabled - change the value to Yes. If you do NOT have AWS Organizations enabled you should be following the steps for AHA for users who are NOT using AWS Organizations
• AWSHealthEventType - select whether you want to receive all event types or only issues.
• Communications Channels section - enter the URLs, Emails and/or ARN of the endpoints you configured previously.
• Email Setup section - enter the From and To Email addresses as well as the Email subject. If you aren't configuring email, just leave it as is.
• EventSearchBack - enter in the amount of hours you want to search back for events. Default is 1 hour.
• Regions enter in the regions you want to search for events in. Default is all regions. You can filter for up to 10, comma separated (e.g. us-east-1, us-east-2).
• ManagementAccountRoleArn - Enter in the full IAM arn from step 10 (e.g. arn:aws:iam::000123456789:role/aha-org-role-AWSHealthAwareRoleForPHDEvents-ABCSDE12201)
• S3Bucket - type just the name of the S3 bucket where exclude file .csv you upload. leave it empty if exclude Account feature is not used.
• ExcludeAccountIDs - type just the name of the .csv file you want to upload if needed to exclude accounts from monitoring, else leave it to empty.

4. Deploy the solution using terraform commands below.

$ terraform init
$ terraform plan
$ terraform apply

Updating using CloudFormation

Until this project is migrated to the AWS Serverless Application Model (SAM), updates will have to be done as described below:

1. Download the updated CloudFormation Template .yml file and 2 .py files.
2. Zip up the 2 .py files and name the .zip with a different version number than before (e.g. if the .zip you originally uploaded is aha-v1.8.zip the new one should be aha-v1.9.zip)
3. In the AWS CloudFormation console click on the name of your stack, then click Update.
4. In the Prepare template section click Replace current template, click Upload a template file, click Choose file, select the newer CFN_DEPLOY_AHA.yml file you downloaded and finally click Next.
5. In the S3Key text box change the version number in the name of the .zip to match name of the .zip you uploaded in Step 2 (The name of the .zip has to be different for CloudFormation to recognize a change). Click Next.
6. At the next screen click Next and finally click Update stack. This will now upgrade your environment to the latest version you downloaded.

If for some reason, you still have issues after updating, you can easily just delete the stack and redeploy. The infrastructure can be destroyed and rebuilt within minutes through CloudFormation.

Updating using Terraform

Until this project is migrated to the AWS Serverless Application Model (SAM), updates will have to be done as described below:

1. Pull the latest code from git repository for AHA.
2. Update the parameters file terraform.tfvars per your requirement
3. Copy the terraform template files to directory where your previous state exists
4. Deploy the templates as below

$ cd aws-health-aware
$ git pull https://github.com/aws-samples/aws-health-aware.git
$ cd terraform/Terraform_DEPLOY_AHA
$ terraform init
$ terraform plan - This command should show any difference existing config and latest code.
$ terraform apply

If for some reason, you still have issues after updating, you can easily just delete the stack and redeploy. The infrastructure can be destroyed and rebuilt within minutes through Terraform.

New Features

We are happy to announce the launch of new enhancements to AHA. Please try them out and keep sending us your feedback!

1. A revised schema for AHA events sent to EventBridge which enables new filtering and routing options. See the new AHA event schema readme for more detail.
2. Multi-region deployment option
3. Updated file names for improved clarity
4. Ability to filter accounts (Refer to AccountIDs CFN parameter for more info on how to exclude accounts from AHA notifications)
5. Ability to view Account Names for a given Account ID in the PHD alerts
6. If you are running AHA with the Non-Org mode, AHA will send the Account #' and resource(s) impacts if applicable for a given alert
7. Ability to deploy AHA with the Org mode on a member account
8. Support for a new Health Event Type - "Investigation"
9. Terraform support to deploy the solution

Troubleshooting

• If for whatever reason you need to update the Webhook URL; just update the CloudFormation or terraform Template with the new Webhook URL.
• If you are expecting an event and it did not show up it may be an oddly formed event. Take a look at CloudWatch > Log groups and search for the name of your Lambda function. See what the error is and reach out to us email for help.
• If for any errors related to duplicate secrets during deployment, try deleting manually and redeploy the solution. Example command to delete SlackChannelID secret in us-east-1 region.

$ aws secretsmanager delete-secret --secret-id SlackChannelID --force-delete-without-recovery --region us-east-1

aws-health-aware's Issues

Add support for setting up on a Member Account for all member accounts without the Management Account role policy

Many partner-resold customers do not have Management account permissions, and partners are also unwilling to create this role, So it is good to add this support

It's statusCode=closed, But no 'endTime' field

This is my event. statusCode="closed". But no 'endTime' in event. So my lambda is died.
"messagegenerator.py line 65, in get_message_for_slack" should be modified.

/AHA-LambdaFunction calls GetSecretValue even though no MS Channel is defined

Our cloudtrail alarming is reporting this error when deploying the lambda with only a slack url.
arn:aws:sts::xxx:assumed-role/AHA-LambdaExecutionRole-ejo5owz1/AHA-LambdaFunction-ejo5owz1 called GetSecretValue but failed due to AccessDenied

Cause: IAM Policy is only created when string is not empty, but the code can not know if the channel id was empty. Therfore it try to fetch it and fails.

Route messages to (Slack) channels based on account name (or number) & CDK?

Just to ensure we aren't about to reinvent a wheel, is anyone aware of an existing method to direct different messages to different webhooks based on (ideally) account name?

We have a slack channel per account and we'd like to send messages accordingly. Slack have made mention that having one app across multiple channels is on their radar but without ETA.

In the meantime we found the slack Echo Bot which matches a keyword and will echo a message to another channel which is very handy, no code approach for our Ops team but it quotes the message rather than sending it a new which kind of messes up the formatting.

I had a look but no one has so far forked or submitted a PR for this have they?
Also no one has plans to reproduce this in CDK have they? It's our go to so might see about the effort involved.

Duplicate notifications for events

Hello

I have AHA deployed in my organisation master account and several member accounts. They have all been deployed with teams notifications enabled.

Early this morning there was an event for a direct connect issue in a member account. In the AWS console, there is one event listed (same if i query the API directly).

However I received 4 teams notifications, all seemingly identical. When the issue was resolved, I again received 4 notifications.

Logs for the function only show 'Sending the alert to Teams' occurring once. What is causing this notification to be spammed to Teams? I cannot see anything in the function that might explain it, so perhaps this is a teams issue?

Thanks

Error when sending email

[ERROR] TypeError: send_email() takes 2 positional arguments but 4 were givenTraceback (most recent call last):  File "/var/task/handler.py", line 836, in main    describe_events(health_client)  File "/var/task/handler.py", line 686, in describe_events    update_ddb(event_arn, str_update, status_code, event_details, affected_accounts, affected_entities)  File "/var/task/handler.py", line 484, in update_ddb    send_alert(event_details, affected_accounts_details, affected_entities, event_type="create")  File "/var/task/handler.py", line 95, in send_alert    send_email(event_details, event_type, affected_accounts, affected_entities)

[ERROR] TypeError: send_email() takes 2 positional arguments but 4 were given Traceback (most recent call last):   File "/var/task/handler.py", line 836, in main     describe_events(health_client)   File "/var/task/handler.py", line 686, in describe_events     update_ddb(event_arn, str_update, status_code, event_details, affected_accounts, affected_entities)   File "/var/task/handler.py", line 484, in update_ddb     send_alert(event_details, affected_accounts_details, affected_entities, event_type="create")   File "/var/task/handler.py", line 95, in send_alert     send_email(event_details, event_type, affected_accounts, affected_entities)

To fix:
change send_email function
def send_email(event_details, eventType):

to match similar send_org_email
def send_email(event_details, eventType, affected_accounts, affected_entities):

also need to update get_message_for_email(event_details, eventType, affected_accounts, affected_entities):

@gmridula I would create a fork but hope you can just fix from this

template provider is deprecated by Hashicorp

The template provider is deprecated by Hashicorp and templatefile or file functions are recommended to be used instead.

template v2.2.0 does not have a package available for your current platform, darwin_arm64.

Error: Incompatible provider version

Provider registry.terraform.io/hashicorp/template v2.2.0 does not have a package available for your current platform, darwin_arm64.

Provider releases are separate from Terraform CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.

Some major problems with setting up project in member account with AWS Organizations enabled

I want to setup this project's resources in a member aws account and then have aws organizations enabled. This would mean that I create a role to access the personal health dashboard info on the payer account and that all the other resources like Lambda, DDB table are on the member account.

The project's README describes that you can setup resources(Lambda, Dynamodb table etc) in a member acccount and then access the Personal Health Dashboard info via a role on the payer/top level account.

That is a false assumption, because in order for that to be possible, you will need to assume a role on the payer account and while doing so also access the Dynamodb table in the member account. That is impossible because its not possible to assume a role in another account and access resources in your current account, atleast not at the same time/moment.

Another issue I found is that the docs mention that for this type of setup you will need to make use of the variable (and give it a value with the ARN of the role you would assume from the member account in the payer account), the variable is MANAGEMENT_ROLE_ARN in either your cloudformation or terraform code.

Just do a quick search in the code for https://github.com/aws-samples/aws-health-aware/search?q=MANAGEMENT_ROLE_ARN
and you will see it is only referenced in the cloudformation and terraform code in the environment variable section. It is not referenced anywhere in the code section. So I'm not sure how the code is supposed to make use of this variable. Maybe I'm missing something here? Please correct me if I'm wrong.

Throttling exception when calling "describe_affected_entities_for_organization"

@jordanaroth @gmridula Lambda function is throwing a throttling exception, I think the issue is at L829,
health_client = get_sts_token('health')

we could use
config = Config(
retries = {
'max_attempts': 10,
'mode': 'standard'
}
)
but this is not supported with get_sts_token

{
  "errorMessage": "An error occurred (ThrottlingException) when calling the DescribeAffectedAccountsForOrganization operation: Rate exceeded",
  "errorType": "ClientError",
  "stackTrace": [
    "  File \"/var/task/handler.py\", line 849, in main\n    describe_org_events(health_client)\n",
    "  File \"/var/task/handler.py\", line 742, in describe_org_events\n    affected_org_accounts = get_health_org_accounts(health_client, event, event_arn)\n",
    "  File \"/var/task/handler.py\", line 323, in get_health_org_accounts\n    for event_accounts_page in event_accounts_page_iterator:\n",
    "  File \"/var/runtime/botocore/paginate.py\", line 255, in __iter__\n    response = self._make_request(current_kwargs)\n",
    "  File \"/var/runtime/botocore/paginate.py\", line 332, in _make_request\n    return self._method(**current_kwargs)\n",
    "  File \"/var/runtime/botocore/client.py\", line 386, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n",
    "  File \"/var/runtime/botocore/client.py\", line 705, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n"
  ]
}

Trigger lambda on an event rather than on a schedule

It strikes me as quite inefficient that this lambda is triggered on a schedule of every minute. Is there a reason this is the case, rather than executing on every Health event received that matches an event pattern?

Slack Workflow creation example image has typo

The Creating a Slack Webook URL section has an typo in the example image for the workflow option:

The status variable is duplicated against the both the start_time and the status labels.

Add support for Splunk ingestion

Add support for ingesting AWS health alerts across organizations into Splunk.

Add SourceArn parameter to ses send_email

Hi,

SES send_email call should have the SendArn parameter.
In the organizational context, SES identities are often defined on a dedicated aws account

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ses/client/send_email.html

aws-health-aware/handler.py

Lines 267 to 283 in eae99cc

	response = client.send_email(
	Source=SENDER,
	Destination={
	'ToAddresses': RECIPIENT
	},
	Message={
	'Body': {
	'Html': {
	'Data': BODY_HTML
	},
	},
	'Subject': {
	'Charset': 'UTF-8',
	'Data': SUBJECT,
	},
	},
	)

add "Action Required" indication to action required events

When aha sending emails to stakeholders, the important ones like: "AWS_RDS_PLANNED_LIFECYCLE_EVENT"
getting the "[Action Required]" prefix in email subject. (see pic)

Is it possible to add this indication for each event json (e.g. action required: true/false)? we would like to promote this kind of event as they are critical

How to get notification from event bus ARN

HI,
We want this notification in the email, where the only option to get it via SES or event bus ARN then SNS. SES required email verification and exchange policy to be allowed for delivery which is difficult

We want to explore event bus option. after the event is received on the event bus it should trigger to SNS, SQS or any other service which required event pattern to be created. What will be event pattern sample or parameter to get that all events to SNS or SQS from event bus? and it should be in json format. So is lambda sending events into json format?

Please assist

Adnan

LambdaExecutionRole creation error

Policy statement must contain resources. (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID:redacted ; Proxy: null)

Lambda cold/warm start issue when DNS changes region

There is an issue in the code when the DNS is switching from one region to another.

We have noticed that when there is a failover from one region to another from the loadbalancer, although the health_active_region is changing, due to the fact that codeblock is not wrapped into a function or in the function handler, the code is not in warm state and wrong results are returned. Currently the initialisation of the config is done on the cold state. As a result when one runs the lambda every x minutes when a failover occurs it will return an error.

In the function logs we notice an error message like:

Client is configured with the deprecated endpoint: us-east-2

aws-health-aware/handler.py

Lines 24 to 38 in 6f047af

	health_dns = socket.gethostbyname_ex('global.health.amazonaws.com')
	(current_endpoint, global_endpoint, ip_endpoint) = health_dns
	health_active_list = current_endpoint.split('.')
	health_active_region = health_active_list[1]
	print("current health region: ", health_active_region)
	# create a boto3 health client w/ backoff/retry
	config = Config(
	region_name=health_active_region,
	retries=dict(
	max_attempts=10 # org view apis have a lower tps than the single
	# account apis so we need to use larger
	# backoff/retry values than than the boto defaults
	)
	)

A solution for this issue could be to wrap the initialisations inside a function.

Multi region deployment issue

I deployed AHA with Multi region option then I found my lambda function on Secondary region failed to run. The reason is misconfiguration of lambda Environment variables. ORG_STATUS value should be “No” but was “false” even though primary region lambda ORG_STATUS value is “No”. Are there anyone who faced same as me?

Ignore Accounts File - Syntax Unclear

Calling the "ignored accounts" file a CSV is a bit confusing. I put all the accounts on one line, separated by commas, and it didnt work. I looked at the code and saw that they're actually separated by a new line.

Might want to make some clarification in the docs.

Further Customizing Delivery Subscriptions

Would it be possible to have a way to easily customize the delivery based on custom metadata or perhaps AWS Organizations Account Tags to send emails to custom addresses on a per account basis?

Use case is to deliver appropriate notifications to: Custom Account Owners, Tenant Application Owners or specific resource owners based on metadata / tags not really supported by AWS today.

I suspect it would be possible to custom code something off EventBridge and Dynamo by accountId, but I get concerned about the API limits for querying Account Tags in organizations. It would be better to have a solution to register/subscribe custom destinations in the DynamoDB.

Clarify docs a bit: Enabling HEALTH API on Org Level

Could you please update the docs a bit just to clarify the following, I think it might be useful for other people:

1. When enabling the Health API, when referring to the "management" account, does this refer to the payeer account?
2. When making use of the Health API (when it was enabled on an organizational level), should one access the API from the payeer account itself or can you make use of account that is below the payeer account in terms of account hierarchy, or perhaps even access it from one of the member accounts?

Multi-Region Deployment failing with "Stack set operation [xxx...] was unexpectedly stopped or failed"

I am unable to deploy in multiple regions. While I have successfully deployed to us-east-1 and tested, I have since tried to add us-west-1 or us-west-2 (on separate attempts) as alternate regions and they fail with the same error:

Resource handler returned message: "Stack set operation [55c...] was unexpectedly stopped or failed" (RequestToken: a820..., HandlerErrorCode: InternalFailure)

The following resource(s) failed to create: [AHASecondaryRegionStackSet]. The following resource(s) failed to update: [LambdaFunction].

Unfortunately, that isn't very much to go on. Any ideas or suggestions? Has anyone else experienced this issue?

Multi-Region deployment fails to create all lambda environment variables

When deploying AHA in multi-region mode, I am getting the error:

[ERROR] KeyError: 'ACCOUNT_IDS'
Traceback (most recent call last):
  File "/var/task/handler.py", line 872, in main
    describe_org_events(health_client)
  File "/var/task/handler.py", line 734, in describe_org_events
    if os.environ['ACCOUNT_IDS'] == "None" or os.environ['ACCOUNT_IDS'] == "":
  File "/var/lang/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None

Looking at cloudformation, it appears the line starting below doesn't create ACCOUNT_IDS as an environment variable for the secondary region -

aws-health-aware/CFN_DEPLOY_AHA.yml

Line 408 in 06229f1

Environment:

Support more filtering on event types

At the moment the event types can only be filtered as "all" or "issue":

https://github.com/aws-samples/aws-health-aware/blob/main/handler.py#L646-L649

It should be possible to filter these more granularly: https://docs.aws.amazon.com/health/latest/APIReference/API_EventType.html

Request to allow passing in any combination of valid event types to filter on

Eventbridge message structure

At the moment the data sent to Eventbridge contains the account information as one field, in this format:

account-name (012345678) - That is account_name (account_id)

References:

https://github.com/aws-samples/aws-health-aware/blob/main/handler.py#L480-L481
https://github.com/aws-samples/aws-health-aware/blob/main/messagegenerator.py#L136

In order to action these events downstream using eventbridge we're having to pull the account id out of that field. It'd be nicer to keep these separate for example two fields: "Account name" and "Account ID"

This comment suggests some upcoming changes to eventbus structure: #29 (comment)

Are there any updates?

CloudTrail AccessDenied entry, for any unconfigured secret

Another user reported to me that they receive many "AccessDenied" entries in Cloudtrail.
Upon testing this in my account environment, I was able to replicate this.

Essentially, for any endpoint not configured, I receive a Cloudtrail entry with access denied stating that the role for AHA does not have access for "null" secret:
" is not authorized to perform: secretsmanager:GetSecretValue on resource: ChimeChannelID because no identity-based policy allows the secretsmanager:GetSecretValue action","requestParameters":null,"responseElements":null"

For them it's an issue because their SIEM is monitoring for those kind of events.

I want to know if this is by design, since I see that handler.py, when it's checking if there is a configured endpoint - is set to look for AccessDenied error coming from the client.

Question: What is the reason for building the an S3 bucket for each region

Dear Folks,

I have a question regarding the following Terraform resources:

resource "aws_s3_bucket" "AHA-S3Bucket-PrimaryRegion" {
    count      = "${var.ExcludeAccountIDs != "" ? 1 : 0}"
    bucket     = "aha-bucket-${var.aha_primary_region}-${random_string.resource_code.result}"
    tags = {
      Name        = "aha-bucket"
    }
}

resource "aws_s3_bucket" "AHA-S3Bucket-SecondaryRegion" {
    count      = "${var.aha_secondary_region != "" && var.ExcludeAccountIDs != "" ? 1 : 0}"
    provider   = aws.secondary_region
    bucket     = "aha-bucket-${var.aha_secondary_region}-${random_string.resource_code.result}"
    tags = {
      Name        = "aha-bucket"
    }
}

I was not able to figure out what they are used for.

I think they are used for a CSV file holding data about excluded accounts if so I do not see a reason to create these buckets if I were just to pass the excluded accounts as a list in Terraform that is interpreted in python as a string and parsed.

If someone could tell me what these buckets are used for that would be great.

Many thanks.

SHD updates beyond initial posting are not shown by AHA

When updates are made to the Service Health Dashboard after an initial event has been opened (i.e. CloudFront goes from Green to Blue), they are not propagated or picked up by AHA and sent to the Slack webhook. Similarly, when a service/event goes back to green, that is not reflected in AHA. It appears that only the initial event notification sends out an alert via AHA but it requires the user to continue to monitor the SHD/PHD for further updates.

Conversely, an RSS feed for Slack picks up all updates to the SHD (including after a service is not green).

SubscriptionRequiredException Error

Getting this error when I go to deploy the solution in a member account that belongs to AWS Organization. Member account does have business support plan.

[ERROR] ClientError: An error occurred (SubscriptionRequiredException) when calling the DescribeEventsForOrganization operation:
Traceback (most recent call last):
  File "/var/task/handler.py", line 872, in main
    describe_org_events(health_client)
  File "/var/task/handler.py", line 720, in describe_org_events
    for response in org_event_page_iterator:
  File "/var/runtime/botocore/paginate.py", line 255, in iter
    response = self._make_request(current_kwargs)
  File "/var/runtime/botocore/paginate.py", line 334, in _make_request
    return self._method(**current_kwargs)
  File "/var/runtime/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)

This can be closed. Issue was because we don't have a business plan in our AWS root account where AWS Organization lives.

Thanks

Business or Enterprise Support in Prerequisites

It looks like Business or Enterprise Support is required. Can this be added to the prerequisites sections?

Terraform Lambda doesn't include env var for channels (Slack, Teams or Chime)

On Lambda handler.py lines 702 to 709 there are calls to get secrets in case environment variables are set for channels to be communicated. ( get_secret(secret_teams_name, client) if "Teams" in os.environ else "None")

The Terraform code, however, doesn't populate that variable dynamically, therefore there will never be those environment variables and the notifications won't be sent.

I corrected it on my code by making the lambda variables as a local variable and merging them to the notifications channel in case they're populated.

Hope this helps.

Health Events With No End Date

Some health events never provide an end date or closure event. We are having a difficult time determining when an event has ended or closed. What logic does the AWS Health Dashboard use for events without a provided end date, closure event, or status? It looks like it may use Last update time.

Example Events
RDS operational notification
OpenSearchService service software update available
VPN redundancy loss

*I see a new EventBridge schema for [aha-2.1-beta] and it looks like this may be solvable in that release.

Question: Is it possible to simulate Health events in AWS?

This is an AWS question, but I figured you might know due to the nature of this project.

I have recently installed AHA for our organization. I was able to test some events by setting the hours back to 4000 for one lambda run. The problem is that only issues showed up for the timeframe before I had enabled Organizational View for Health. I would like to test other event types proactively instead of waiting for them to happen in the wild.

Is there a mechanism in AWS that allows test health events to be created that come in through the API's like a normal event? If not, how do you test the code for this project?

aws-cdk support

It would be awesome if there were out-of-the-box aws-cdk support for this. I saw that there's beta support for terraform, but CDK support would be awesome!

Notifications not associated to a region

Hello Jordan. The solutions is working great in ms-teams. Thanks!

I've configured a couple of regions of my interest, I see some notifications doesn't appear when they are not associated to a region, do I need to select global for this?. What will happen if I select "us-east-1,us-east-2, global". I'll receive all the events world wide or only us-east-1,us-east-2 + those not associated to a region?

thanks

Skip alerts for some Issue Types

In the case of noisy alerts, looking for a way to reduce the noise by excluding some event types from generating alerts (e.g. AWS_VPN_SINGLE_TUNNEL_NOTIFICATION,AWS_VPN_REDUNDANCY_LOSS).

Forked the repo and added an attempt at:
https://github.com/benrobinsonsonos/aws-health-aware/blob/b60138a73b8ba3dea2695eafa503b140324c190a/handler.py#L48
https://github.com/benrobinsonsonos/aws-health-aware/blob/b60138a73b8ba3dea2695eafa503b140324c190a/handler.py#L120

Reminders of AWS_EC2_INSTANCE_STOP_SCHEDULED

Can this monitor be extended to repeat notifications of upcoming AWS_EC2_INSTANCE_STOP_SCHEDULED alerts.
Possibly in a reverse fibonacci sequence - 13, 8, 5, 3, 2, 1.

It would also be nice if the instance NAME was pulled from the Name Tag.

Organization Account Name Lookup Too Frequent

aws-health-aware/handler.py

Line 354 in 6f047af

entity['awsAccountName'] = get_account_name(entity['awsAccountId'])

The call to get_account_name in get_affected_entities is currently in inside the loop around the returned entities for the Health event. As this is only used when Org Mode is turned on, and the call to describe_affected_entities_for_organization uses an awsAccountId filter, this means that the account name is looked up for the same account for each entity that is returned.

It would be more efficient to move the get_account_name call to the outside account loop so that it is only done once for each account in the affected_accounts list when running in Org Mode.

Terraform deploy to single region fails on creation of AWS Secret Manager secret for AssumeRoleArn

The current state of the main branch of this repo is not deployable using terraform provided in this repo.

When deploying to an Organisation member account, in a single region, using Terraform there are 2 errors:

Error 1:

Error: error creating Secrets Manager Secret: InvalidParameterException: Invalid replica region.

   with aws_secretsmanager_secret.AssumeRoleArn[0],
   on Terraform_DEPLOY_AHA.tf line 416, in resource "aws_secretsmanager_secret" "AssumeRoleArn":
  416: resource "aws_secretsmanager_secret" "AssumeRoleArn" {

Error 2:

 Error: Invalid index
 
   on Terraform_DEPLOY_AHA.tf line 207, in resource "aws_s3_bucket_acl" "AHA-S3Bucket-PrimaryRegion":
  207:     bucket = aws_s3_bucket.AHA-S3Bucket-PrimaryRegion[0].id
     ├────────────────
     │ aws_s3_bucket.AHA-S3Bucket-PrimaryRegion is empty tuple
 
 The given key does not identify an element in this collection value: the
 collection has no elements.

Please Note: Error 1 was already attempted to be fixed in this PR-32 of this project

Move to EventBridge notifications

AHA currently uses AWS Health API polling to get event notifications. Instead, AHA could use EventBridge notifications to respond to AWS Health events. This would be more efficient than polling every minute.

Is there a plan to consider using CDK as an alternative to Cloudformation deployment ?

Just wonder if anyone consider to add CDK deploy capability, as more and more teams started using it.

Lambda issue - 413 Request Entity Too Large

Due to recent AWS Health Event occurring during the past two days. The Lambda started to fail on second day posting messages to Slack and Chime due to 413 Request Entity Too Large error. SNS notification worked just fine.

Name caching

Is there an issue with the way the lookup of the name is set as a global variable in the Lambda? I might be lacking in knowledge of how lambda functions work but my understanding is once the lambda has been invoked a global variable would be set and not initialised again. As the lambda function is generally kept warm for more than the period of invocation (1 minute looking at the terraform example) then I don't see how a change in the DNS record would be picked up.

I will do a PR with a proposed fix.

AHA sample only fetches resources from first affected account

https://github.com/aws-samples/aws-health-aware/blob/main/handler.py#L333 - it should iterate over all accounts

Cannot consume terraform as a module

because *.py scripts are in the root folder i cannot use terraform code as a module:
source github.com/aws-samples/aws-health-aware//terraform/Terraform_DEPLOY_AHA?ref=v2.01
so i cant store configuration in my source code - need to fork/dowoad and update code manually

if you can change this that would be great

Text width in teams is wider than the card

The width of the card in Teams in wider than Teams displays:

Slack Workflow

Has anyone ran into issues in getting this to work with Slack Workflows?

I've created as per the docs but the workflow just reports Received a webhook request that was missing a required field. No indication of what it received. If I call the workflow via its webhook via curl and the following input it works fine.

curl -X POST https://hooks.slack.com/workflows/<redacted> -H 'Content-Type: application/json' -d '{"text":"test","account":"1234","resources":"my_resource","service":"my_service","region":"my_region","status":"my_status","start_time":"start_time","event_arn":"my_arn","updates":"none"}'

Wondering if anyone had ran into this at all before?

Typo in readme instructions for multi-account deployment

README.md

Line 255 : 9. In the Outputs tab, there will be a value for AWSHealthAwareRoleForPHDEventsArn (e.g. arn:aws:iam::000123456789:role/aha-org-role-AWSHealthAwareRoleForPHDEvents-ABCSDE12201), copy that down as you will need it for step 16.

Should reference step 14, not 16.