jogold/cloudstructs

High-level constructs for AWS CDK

cloudstructs

High-level constructs for AWS CDK

Installation

npm install cloudstructs or yarn add cloudstructs

Version >= 0.2.0 requires AWS CDK v2.

Constructs

• CodeCommitMirror Mirror a repository to AWS CodeCommit on schedule

• EcsServiceRoller Roll your ECS service tasks on schedule or with a rule

• EmailReceiver Receive emails through SES, save them to S3 and invoke a Lambda function

• 'MjmlTemplate` SES email template from MJML

• SlackApp Deploy Slack apps from manifests

• SlackEvents Send Slack events to Amazon EventBridge

• SlackTextract Extract text from images posted to Slack using Amazon Textract. The extracted text is posted in a thread under the image and gets indexed!

• SslServerTest Test a server/host for SSL/TLS on schedule and get notified when the overall rating is not satisfactory. Powered by Qualys SSL Labs.

• StateMachineCustomResourceProvider Implement custom resources with AWS Step Functions state machines

• StaticWebsite A CloudFront static website hosted on S3 with HTTPS redirect, SPA redirect, HTTP security headers and backend configuration saved to the bucket.

• ToolkitCleaner Clean unused S3 and ECR assets from your CDK Toolkit.

• UrlShortener Deploy an URL shortener API

cloudstructs's Issues

Vulnerability through bundled dependency got > cacheable-request > http-cache-semantics

There is a vulnerability in http-cache-semantics.

Having a bundled dep seemingly makes it impossible to override the vulnerable http-cache-semantics package version (I could be wrong, but I can't see a way).

Is it possible to update deps to avoid the vulnerability, or change to not use bundled packages (excuse me if I have some assumptions wrong, I wasn't aware of bundled dependencies until this came up)?

npm audit

npm why http-cache-semantics

Thanks!

Toolkit Cleaner: Images with prefixed tags always deleted

The CDK App class can be given a custom synthesizer, with the option to prefix the image tag for any Docker image in the app, i.e.:

const app = new cdk.App({
  defaultStackSynthesizer: new DefaultStackSynthesizer({
    dockerTagPrefix: 'api-svc'
  })
})

Creating a Docker image in this stack results in an image tag like api-svc5a7abf30ce10141adcb73c9b836ec68479c65fb4d1693df160563e36ece0d55e.

The Extract Template Hashes function specifically looks for an asset with just a hash (no prefix)...

const hashes = template.TemplateBody.match(/[a-f0-9]{64}/g);

...which means a Docker image with an additional prefix does not get extracted and ultimately gets deleted.

Admittedly, I originally used the image prefix feature as a means of tagging images to apply a lifecycle rule to remove old images, and I've moved toward using the Toolkit Cleaner instead (and have since removed the prefix), but in the interim where I had both the prefix and was using the Toolkit Cleaner, the Toolkit Cleaner was removing all of my images, even when they were in use.

A few routes that might work:

1. Document that the Toolkit Cleaner is not compatible with usage of ECR image tag prefixes
2. Relax the Extract Template Hashes function so it matches something like (just conceptual, not a regex) ':<64-char-hash'
3. If (2) is untenable, add an optional prop to the ToolkitCleaner construct for image prefixes and pass those in to the Extract Template Hashes function to search for any assets starting with one of those prefixes and ending with the traditional 64-char hash. This would leave it to users to specify the prefix to prevent their images from always being deleted.

ToolkitCleaner is using deprecated API

I'm getting the following warning when using the ToolkitCleaner construct

      [WARNING] aws-cdk-lib.aws_stepfunctions.Map#iterator is deprecated.
        - use `itemProcessor` instead.
        This API will be removed in the next major release.

cloudstructs version: 0.7.3
NodeJS version: 20.9.0

ToolkitCleaner is using deprecated API

When using the ToolkitCleaner construct:

[WARNING] aws-cdk-lib.aws_stepfunctions.StateMachineProps#definition is deprecated.
  use definitionBody: DefinitionBody.fromChainable()
  This API will be removed in the next major release.

Be more specific about locating hashes

cloudstructs/src/toolkit-cleaner/extract-template-hashes.lambda.ts

Line 14 in 02958bf

const hashes = template.TemplateBody.match(/[a-f0-9]{64}/g);

Do you think it would be safe to find the assets from the cdk metadata i.e. modify our pattern to be:

(/asset.([a-f0-9]{64})/g);

I'm a little concerned this pattern is too generic and will pick out other guid looking things that just happen to be the wrong length. It doesn't matter a lot since other random guids probably won't be files in the bucket. But this seems a little coarse.

That said, I tried to rewrite using an actual template parser and it was rough. The CfnParser in CDK wasn't really meant to be used externally and I can't get it to work. So I'm not sure I have a better alternative (yet).

Cleaning large amount of images

Hi, I had to run the state machine multiple times, to clean all the ECR images.
Even though the output stated it e.g. deleted 300+ images, only a fraction of the images got deleted.

I guess it's either due to the 100 id length limit on the ECR batch delete command (https://docs.aws.amazon.com/AmazonECR/latest/APIReference/API_BatchDeleteImage.html#API_BatchDeleteImage_RequestSyntax)

or

maybe it makes sense to not silently increase the deleted counter (https://github.com/jogold/cloudstructs/blob/master/src/toolkit-cleaner/clean-images.lambda.ts#L47), but instead inspect the response of the batch delete command, seems like it can also contain information about both successful and failed image ids (https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/Package/-aws-sdk-client-ecr/Interface/BatchDeleteImageCommandOutput/)

anyway, very useful so far, I will just run it multiple times.

Should the trigger re-run the state machine if the state machine has been modified?

Hi there, thanks for this awesome library!

I think the StateMachineCustomResourceProvider may save me 2-3 days of work figuring out how to make a custom resource that executes a state machine.

One thing my team and I noticed: it seems that if we do a cdk deploy wherein we added new tasks to our state machine, the trigger does not actually execute our state machine.

Better said: If the cdk deploy is creating the state machine from scratch, it gets triggered. If the cdk deploy is adding/modifying tasks in an existing state machine's definition, it doesn't get triggered.

Is this the desired behavior? If so, does a workaround come to mind we could use to achieve that result?

Here's our usage of the resource:

def trigger_state_machine(self, state_machine: sfn.StateMachine) -> CustomResource:
    """
    Execute the ETL state machine.

    Find documentation on the ``cloudstructs.StateMachineCustomResourceProvider``
    here: https://github.com/jogold/cloudstructs/tree/18116c10466b3cf7c7a8e1acde1eb5c22dfb1b64/src/state-machine-cr-provider
    """
    state_machine_trigger_provider = StateMachineCustomResourceProvider(
        self,
        self.namer("trigger-etl-state-machine-provider"),
        # due to this dependency, should it be the case that the provider/custom resource should
        # cause the state machine trigger to execute the state machine EXACTLY
        # when we've modified the state machine?
        state_machine=state_machine,
    )

    # creation of this resource will actually execute the ETL job for the customer ✨
    state_machine_trigger = CustomResource(
        self,
        self.namer("trigger-etl-state-machine"),
        service_token=state_machine_trigger_provider.service_token,
        # these could be inputs to the resource
        properties={},
    )

    return state_machine_trigger

Docker requirement necessary for ToolkitCleaner?

Thank you for creating this! It's super helpful.

I was using ToolkitCleaner and noticed it required Docker running, which I believe is due to this.

I don't use ECR currently and I'm wondering whether it would be possible to have a property in ToolkitCleanerProps to conditionally deploy the image cleanup code/step function branch so Docker isn't necessary. Does DOCKER_IMAGE_ASSET_HASH need to be set from dockerImageAsset or could it be set from fileAsset?

If this is possible and makes sense, let me know, I'd be happy to attempt submitting a PR.

Feature Request: DMARC report analyzer

Hey, @jogold, I just stumbled upon this repo. There are a bunch of really cool and helpful things here. Thanks for making them available!

One construct I've been working on is an automated way to parse DMARC report emails via a lambda + SES and notify a Slack channel of any failures. Unfortunately, I got sidetracked and still need to finish it.

I see an EmailReceiver and a bunch of Slack-related constructs here, so perhaps you would also be interested in this.

I'll see if I can finish mine and contribute it here. Is this something that would be interesting to merge?

[StaticWebsite]: CNAME is already associated with a different resource

The following basic example fails for me when running cdk deploy:

const accountDomain = 'dev.fubar.com';
const importedHostedZone = HostedZone.fromHostedZoneAttributes(this, 'ImportedHostedZone', {
  hostedZoneId: importedHostedZoneId,
  zoneName: accountDomain 
});
const staticWebsite = new cloudstructs.StaticWebsite(this, 'StaticWebsite', {
    domainName: accountDomain,
    hostedZone: importedHostedZone,
    backendConfiguration: {
      baseUrl: accountDomain
    }
  });

The HostedZone for 'dev.fubar.com' has no CNAME record, only the NS and the SOA.
There are no other CloudFront distributions in this account.

3:37:24 PM | CREATE_FAILED | AWS::CloudFront::Distribution | StaticWebsite319123A6
Resource handler returned message: "Invalid request provided: One or more of the CNAMEs you provided are already associated with a different resource. (Service: CloudFront, Status Code: 409, Request ID: 04c3eb61-4314-43e1-b511-8979a4bce421)"

Add Html2Pdf construct

The Html2Pdf construct deploys an API that converts HTML files to PDF.

Add CodeCommitMirror construct

The CodeCommitMirror construct mirrors a GitHub repo to AWS CodeCommit (backup).

Vulnerable slack/web-api version

It seems that cloudstructs is using @slack/[email protected] which has a dependency to axios@^0.21.1 and that contains a vulnerability (axios/axios#6022). It would be great if you could upgrade the Slack Web API version to 6.*

Cross dependencies between constructs caused ToolkitCleaner failed to run

It required @aws-cdk/aws-apigatewayv2-*

Error: Cannot find module '@aws-cdk/aws-apigatewayv2-integrations-alpha'
Require stack:
- /home/jack/workspace/github/test-ToolkitCleaner/node_modules/cloudstructs/lib/slack-events/index.js
- /home/jack/workspace/github/test-ToolkitCleaner/node_modules/cloudstructs/lib/index.js
- /home/jack/workspace/github/test-ToolkitCleaner/src/main.ts
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/home/jack/workspace/github/test-ToolkitCleaner/node_modules/cloudstructs/src/slack-events/index.ts:2:1)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
Subprocess exited with error 1

But I don't see how the ToolkitCleaner needs those modules or where to use them

ToolkitCleaner throws States.DataLimitExceeded at GetTemplate

{
  "error": "States.DataLimitExceeded",
  "cause": "The state/task 'aws-sdk:cloudformation' returned a result with a size exceeding the maximum number of bytes service limit."
}

TookitCleaner: Node 18 Lambdas do not have aws-sdk@2

I attempted to deploy ToolkitCleaner as a brand new stack but the lambdas fail to execute. Seems that they attempt to require aws-sdk which is assumed to be available in the lambda environment but that's not the case in node 18 lambdas which only have aws-sdk@3

Runtime.ImportModuleError in step: GetStackNames

{
  "errorType": "Runtime.ImportModuleError",
  "errorMessage": "Error: Cannot find module 'aws-sdk'\nRequire stack:\n- /var/task/index.js\n- /var/runtime/index.mjs",
  "trace": [
    "Runtime.ImportModuleError: Error: Cannot find module 'aws-sdk'",
    "Require stack:",
    "- /var/task/index.js",
    "- /var/runtime/index.mjs",
    "    at _loadUserApp (file:///var/runtime/index.mjs:1000:17)",
    "    at async UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:1035:21)",
    "    at async start (file:///var/runtime/index.mjs:1200:23)",
    "    at async file:///var/runtime/index.mjs:1206:1"
  ]
}

Publish to Maven is broken

Latest published version is v0.7.2

cdklabs/publib#1107

Missing ListStacks permission

Hello!

I just deployed this stack into my App Stack and the Step Functions is failing with the following error:

is not authorized to perform: cloudformation:ListStacks on resource: arn:aws:cloudformation:us-east-1:XXX:stack/*/* because no identity-based policy allows the cloudformation:ListStacks action

If manually added the permisson and it seems to work propertly. Any idea if I am doing something wrong? If not, is it ok to send a PR with this fix?

Deployment detail

Quick question, do I need to deploy this once per region or once per account?

Reporting a vulnerability

Hello!

I hope you are doing well!

We are a security research team. Our tool automatically detected a vulnerability in this repository. We want to disclose it responsibly. GitHub has a feature called Private vulnerability reporting, which enables security research to privately disclose a vulnerability. Unfortunately, it is not enabled for this repository.

Can you enable it, so that we can report it?

Thanks in advance!

PS: you can read about how to enable private vulnerability reporting here: https://docs.github.com/en/code-security/security-advisories/repository-security-advisories/configuring-private-vulnerability-reporting-for-a-repository

`@aws-cdk/aws-apigatewayv2-alpha` was deprecated

cloudstructs/src/slack-events/index.ts

Lines 1 to 2 in 0ad4e6f

	import * as apigatewayv2 from '@aws-cdk/aws-apigatewayv2-alpha';
	import * as integrations from '@aws-cdk/aws-apigatewayv2-integrations-alpha';

It appears that @aws-cdk/aws-apigatewayv2-alpha is deprecated and now part of aws-cdk-lib.

Fixing this will need a breaking change version bump.

Toolkit cleaner Python

Thanks a lot for your work! Any plans on creating a Python version of the toolkit cleaner? Would be awesome!