compose-x/ecs_composex

Manage, Configure and Deploy your services and AWS services and applications from your docker-compose definitions

ECS ComposeX

The no-code CDK for docker-compose & AWS ECS

Deploy your services to AWS ECS from your docker-compose files in 3 steps

• Step 1. Install ECS Compose-x
• Step 2. Use your existing docker-compose files. Optionally, add Compose-X extensions.
• Step 3. Deploy to AWS via CloudFormation.

What does it do?

• Automatically deploy applications to AWS using existing docker-compose files

  • Deploys multiple applications to AWS in a single command
  • Creates AWS resources such as EC2 Instances, ECS Clusters and Containers
  • Automatically configures IAM roles and Networking for secure and reliable access

• Expand the definitions with AWS CloudFormation resources
• Allows to use existing resources in your AWS Account
• Can be extended with custom modules/hooks to customize the deployment process
• Automatically rolls back the application in case of errors, to previous version or to a stable state

Useful Links

• Documentation
• Labs
• Feature requests
• Issues
• Compatibility Matrix

Installation

# Inside a python virtual environment
python3 -m venv venv
source venv/bin/activate
pip install pip -U
pip install ecs-composex

# For your user only
pip install ecs-composex --user

Usage

# Get all the options
ecs-compose-x -h

# Simple example using docker-compose file and an extension with your AWS Settings
ecs-compose-x render -d templates -n my-new-stack -f docker-compose.yaml -f aws-settings.yaml

ecs_composex's Issues

FR - x-s3

Feature request to add S3 buckets as one type of resources that can be added up into ecs_composex definition file and allow services to RW/RO from the buckets.

Enable VPC Endpoints

Allow to define VPC Endpoints required in the VPC.
I should be able to add the VPC endpoints as a group addon to the original VPC and specify which endpoints are created.

By default, all ECS task should have network access to tall VPC Endpoints.

x-sns

Support for SNS topics
Original requirement for basic topic definition and creation and providing access to services to publish messages to the topic.
Basic subscriptions should be supported (https endpoints and mail).

Enable X-Ray

Is your feature request related to a problem? Please describe.
I want to be able to do the analysis of my application performance via XRay.

Describe the solution you'd like
Via the service configs (local and/or global) I would be able to indicate whether or not I want the AWS XRay container to be running as part of my task definition

configs:
  composex:
    use_xray: False

services:
  serviceA:
    configs:
     use_xray: True

Describe alternatives you've considered
In the CLI, add --enable-xray would override configuration settings and set to true for all.

Additional context
The X-Ray container should be contained to the minimum compute footprint possible.

Stop using labels for config and only rely on config

Using labels is abusing of the structure allowed by docker compose.
A configs section can be use directly within the service definition.

To apply configuration we would then evaluate in order

• configs set in the service directly
• configs set in the docker compose file
• when applicable, from the globals in the configs
• use ECS ComposeX default settings

Refactor the Load-Balancing config to enable more settings.

Is your feature request related to a problem? Please describe.
I want to be able to define extra settings for the ALB or NLB which points to my container.
Instead of using booleans for use_alb / use_nlb, I would prefer to have a configuration object, containing a variable lb_type value set to network or application.

Describe the solution you'd like

services:
  serviceA:
    configs:
      network:
        ingress:
          lb_type: application
          acm: use::blah # example of extra settings.

Additional context

Notification / Alaram for messages in DLQ

When I have a message in DLQ, I would like to be notified through a given topic.

Refactor - Change the DNS structure to make the private discovery map be the subzone.

Describe the bug
Currently, we have abcd.cluster.local (where abcd is the root stack name) namespace used for the VPC Private Namespace (CloudMap) and we get sub.abcd.cluster.local for manual extra hosts.

This is proving counter intuitive. Instead, I would like to have

svc.abcd.cluster.local as the namespace used for the services to be registered into.
Still could use sub.abcd.cluster.local but essentially, allows to get abcd.cluster.local be created outside of that stack.

FR - Allow to write all templates of all components into a directory

Feature to allow to use --output-all-templates which will write all templates into the destination folder.

Allow import CFN template for a given resource

I already have a CFN template for a specific type of resource and all I want is to deploy it along with the stacks defined in composex.
ComposeX should only modify the template to add outputs to allow access and such but nothing else.

x-sqs:
  queueA:
    TemplatePath: ./path_to_template.yml
    TemplateParametersPath: ./path_to_params.json
    TemplateConfigPath: ./path_to_config.json

x-cluster to allow changing cluster provisioning configuration.

As an ECS user, I want to be able to define the cluster capacity providers for my cluster.

x-cluster:
  Properties:
      CapacityProviders:
        - FARGATE
        - FARGATE_SPOT
      ClusterName: Name
      DefaultCapacityProviderStrategy:
        - CapacityProvider: FARGATE_SPOT
          Weight: 2
        - CapacityProvider: FARGATE
          Weight: 1

In the future, I would like to be able to define an ASG to add as a Capacity Provider.
By default, I would like to use FARGATE and FARGATE_SPOT if not provided.

Enable to expand roles with PermissionsBoundary

Is your feature request related to a problem? Please describe.
I have permissions to create roles only with a given set of permissions boundaries. Therefore, all my IAM roles I create should have IAM boundaries set.

Describe the solution you'd like
In the configs/composex section, add a new section for IAM which allows to list boundary policy names

configs:
  composex:
    iam:
      boundaries:
        - policy1
        - policy2

Describe alternatives you've considered
Via the CLI, provide a list of IAM boundary names or ARNs which need to be added.

Additional context
Before adding the boundary, if an ARN is provided, it should be applicable to the current account.

ClusterWideSecurityGroup needs a condition to not break the template when used standalone.

Describe the bug
When using the CFN template of the service itself, there might not be a SG for all services in the cluster. There is currently no condition to ignore the value in case none is provided

Expected behavior
When I use the service template in standalone, and do not provide a SG-ID for the ClusterWideSG, then everything should work.

Logs
If applicable, add logs of errors using from ECS ComposeX

Desktop (please complete the following information):

• Version 0.2.3

x-rds/Settings/EnvNames are no exposed to the microservice

Describe the bug
In the x-rds section, for the databases, the EnvNames are not iterated over to expose the secret to the microservice

To Reproduce
Steps to reproduce the behavior:
Have a docker-compose file with x-rds DBs. Set EnvNames. Run ecs_composex.
In the secrets list associated with the container definition, only the DB name as defined in compose file is present.

Expected behavior
The source of the secret is the same, but there is a list of them with the various EnvNames defined in settings.

Version 0.2.2

Allow snapshot for RDS DB creations

Is your feature request related to a problem? Please describe.
I want to be able to give a RDS Snapshot ID which will be used to create my RDS DB.

Describe the solution you'd like

x-rds
  dba:
    Properties:
      Engine: aurora-mysql
      EngineVersion: 5.7.12
      SnapshotIdentifier: <snap-ID>

It will still create a new Secret for the DB an link to it, then expose to the service, BAU.

Enable Blue/Green for ECS

Is your feature request related to a problem? Please describe.
Implement to use Blue/Green via CFN for ECS

Refactor compute subackage

Refactor the Compute module so that we can use the CLI in a better standalone fashion and integrate once again back to ECS ComposeX root script

x-acm for private Certificate, with x-appmesh integration

Is your feature request related to a problem? Please describe.
When using x-appmesh, I want to be able to define mTLS using aws private CA between nodes.

Describe the solution you'd like
Allow to instruct whether to create AWS CA structure or simply use existing CA.

For well known images, replace with the x-section accordingly.

Is your feature request related to a problem? Please describe.
If I use a docker image like postgres or redis which are well known ones, I would like ecs composex to interpolate with x-resources (namely RDS or elastic cache)

Describe the solution you'd like

services:
  db:
     image: postgres
     config:
       composex:
         xreplace: true

Results into

x-rds:
  db:
    Properties: {}
    Services: []
    Settings: {}

Enable AppMesh Support

AWS AppMesh integration with ECS enables the following:

• better end-to-end visibility for ECS services
• additional network control and monitoring

Prerequisites

• ECS Service Discovery needs to be enabled

List all the IAM permissions required to run ECS ComposeX

Provide a policy template that contains all the permissions needed for ECS ComposeX to work properly.

x-dynamodb

Support for AWS DynamoDB tables.
Initial support should allow to create the tables and provide RW access and RO access to the tables.

Allow for extra IAM permissions to be given to the service

In addition to access to SQS etc, I would like my container to have access to other services on AWS I need IAM access for.

Describe the solution you'd like

x-configs:
  iam:
    boundary: abcd
    managed_policies:
      - policyArn
    policies:
      - name: abcd
        document: {}

Enable VPC "discovery" from tags

Describe what you are trying to achieve
As a user of ECS ComposeX, I would like to be able to define some tags or properties that will allow me to map an existing VPC and its subnets to the public, application and storage.

Describe the solution you'd like
In the compose file, I could have a section x-mappings with a vpc subsection that allows to define
the subnets

Describe alternatives you've considered
If a VPC stack has been created from ecs_composex-vpc I could refer to its name in the CLI with --vpc-stack-name and it will import the values from output.

Similarly to docker-compose, allow multiple -f to implement override.

As a developer, I have multiple docker-compose files, one for each environment, which changes the settings / environment variables of my services.
I would like to be able to have a docker-compose.yml file and override with another one.

docker-compose -f docker-compose.yml -f composex.yml

Where docker-compose.yml contains my services, and composex contains the x-Resources and possibly update settings of docker services

Support Docker compose v3 compute definition

As per https://docs.docker.com/compose/compose-file/#deploy

Although, given the requirements for configuration with Fargate, some settings will be ignored.
Settings of deploy to support in a first stage:

• replicas -> sets the desired number of resources
• resources -> sets memory and CPU profile
  => Given Fargate requirements, if the settings do not match the fargate settings, default settings will be applied

To allow a default profile, a specific configuration setting in configs/composex will be created.

Add KMS link between resources and newly defined/imported KMS keys

Is your feature request related to a problem? Please describe.
I want to use SQS queues with KMS to encrypt the payload.
I should be given the option to use a CMK otherwise use the default key.
When using CMK, the services linked to the queue must be granted decryption access to the queue.

Describe the solution you'd like

Using the AWS account KMS key for sqs

x-sqs:
  Queue01:
    Properties:
      KmsMasterKeyId: use::alias/aws/sqs

Describe alternatives you've considered
Alternatively, the following would indicate to create the key with AWS default settings.

x-sqs:
  Queue01:
    Properties:
      KmsMasterKeyId: create::sqskey

Or to lookup an existing KMS key

x-sqs:
  Queue01:
    Properties:
      KmsMasterKeyId: use::sqskey

Implement x-rds to create RDS DBs

Implement x-rds to create RDS DBs that services connect to.
The credentials to connect to the database should be created in AWS Secrets Manager and exposed to the container via Environment variables, providing the IAM Execution role to decrypt and get the secrets value.

x-tags are not propagated to Launch Template tag specifications

Tags defined in x-tags should be added to the Launch template tag specifications so that the EC2 resources created out of that template also have the tags as specified.

create `--deploy` and `--diff` to deploy to CFN from ECS ComposeX

Is your feature request related to a problem? Please describe.
As a user, I would like to immediately deploy to CFN, which would either create or update the stack.
Optionally, I would like to run --diff which would tell me which changes are going to happen on the stacks, recursively.

Describe the solution you'd like

ecs_composex -f docker-compose.yml -n appstack --deploy

ecs_composex -f docker-compose.yml -n appstack --diff

Add x-kms to create KMS CMK

Is your feature request related to a problem? Please describe.
I want to create KMS keys which are going to be used by topics,queues,dbs etc.
When a service is listed to be able to use the key, the service should be able to encrypt and decrypt
Otherwise, when the key is assigned to queues/topics, the services who can access the service will also be allowed to use the kms key.

Describe the solution you'd like

x-kms:
  key01:
    Properties: {}
    Settings:
      Alias: if specified, creates a KMS Key alias
      AwsServices:
        - name: AWS Service allowed to use the key, ie. abcd
          access: AllCrypto | Decrypt | Encrypt
    AwsAccounts:
     - id: [0-9]{12} # account ID
       access: AllCrypto | Decrypt | Encrypt
    Services: []

Allow to create templates without VPC or cluster without any input, only warnings.

When not using --create-vpc or --create-cluster, allow to create all the templates anyway without building the parameters file.

Allow different LB listener port and target port

In some cases, it might be useful for development perspective to have a load balancer listen on one port and sent traffic to the microservice on a different one.

In the local environment, people for ports would because of ports opening do

serviceA:
  ports:
    - 8080:80
    - 8081:443

serviceB:
  ports:
    - 8180:80
    - 8181:443

that way both service A and B are exposed to the user and do not have overlapping ports.

Similarly, if a micro-service has a port defined with <port>:<port> and uses a load-balancer then the load-balancer should have a listener using the first port to target the microservice, and the second port used as the exposed port of the microservice.

Multiple families for service

Is your feature request related to a problem? Please describe.
I'd like to use a sidecar container to run my migrations when my services start. Multiple services all require the same migration container. Currently I need to have duplicate service definitions because a service can only be in one ecs.task.family.

Describe the solution you'd like
I would like to specify more than one family a service belongs to.

Duplicate environment variable key

Describe the bug
Setting an EnvName on a resource (ie, SQS queue) and have the same environment variable on the service, ECS ComposeX doesn't use the one from the service but has two environment variables with the same name.

To Reproduce

services:
  targetone:
    image: ${SFTP_TRANSFER_IMAGE}
    environment:
      QUEUE_NAME: targetonequeue


x-sqs:
  targetonequeue:
    Properties: {}
    Settings:
      EnvNames:
        QUEUE_NAME: targetone
    Services:
      - name: targetone
        access: RWMessages

Expected behavior
Only one environment variable should be set, and it should use the one coming from the x-resource.

Add global x-tags to all resources

Allow to define generic tags that get added everywhere and generate parameters from that which are then passed on from stack to stack and apply to all resources that support tags

Error in ecs_composex/ecs/SYNTAX.rst#L90

Is your feature request related to a problem? Please describe.
Since the change to lb_type, use_alb and use_nlb is no longer necessary.

Describe the solution you'd like
Correct the documentation

Allow support for ACM SSL cert for publicly exposed services.

Is your feature request related to a problem? Please describe.

• When I already have a public DNS Zone, I provide with its ID
• I indicate what public hostname I would like the family/service to be published on
• ACM creates a public certificate for it that gets automatically validated by Route53 record.
• If I already have a public DNS Namespace I pass the namespace ID and the service gets registered there whereas the
  certificate is added to the route53 zone.

ACM properties page

Describe the solution you'd like

x-acm:
  publicApp:
    Properties: {}
    Settings: {}
    Services:
      - name: serviceA
        ports: [443]

services:
  serviceA:
    x-configs:
      network:
        lb_type: application
        is_public: True

Allow multiple services to be merged into one Task definition

Is your feature request related to a problem? Please describe.
I would like to be able to have multiple containers in a single task, such as a ngnix webserver container and java application server container.

Describe the solution you'd like
Leverage docker labels to annotate which ECS service and cluster a container belongs to, so it can be placed into the same task defintion

Example:

nginx:
    image: nginx
    labels:
      ecs.service: myapp
java:
	image: springboot
	labels:
		ecs.service: myapp

Describe alternatives you've considered

Additional context
I believe this design pattern can be extended to annotate which sqs queues are in use, and also ECS specific parameters such as task CPU and memory configurations.

Enable access to existing resources (IAM based only).

Is your feature request related to a problem? Please describe.
I have SQS queues defined outside of ECS ComposeX, I would like my microservices to get access to these queues.

Describe the solution you'd like
I can give an existing queue ARN and the microservice will be granted access to that existing queue.
Alternatively, I would like to be able to provide an existing queue name and indicate that I would like to search for it.

Describe alternatives you've considered

x-sqs:
  queue01:
    Properties:
      Name: queue01abcd
    Settings:
     Exists: True # indicates that the queue already exists
     Lookup: True # indicates that I provided the name only, one should find the ARN
    Services: []

Add --generate-stack-parameters

CLI option which will generate a file containing the stack parameters for Codepipeline as per documented in https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline-cfn-artifacts.html

root template is not validated

root template is not uploaded and not validated and therefore issues have gone through.

Add `--no-upload` to not upload templates to S3

Is your feature request related to a problem? Please describe.
I want to produce the templates only locally

Describe the solution you'd like
Add --no-upload to the CLI

Define SQS metrics alarms to scale in/out ECS services

Is your feature request related to a problem? Please describe.
I want to define alarms based on SQS metrics for a given queue which is going to drive scaling activities of the designated services.

Describe the solution you'd like
When metrics reach an alarm state, trigger a scaling activity on the application service.

Generate parameters files for each individual stack

At present, each nested stack parameters are hard-written into the templates and one would have to extract the values for these from the CFN stack created.
One want to have access to the parameters file for the stack individually directly from stored file.

Two extra files should be created and uploaded:

• The parameters file [{Key:Value}]
• The CFN Template config file (can be used in CodePipeline).

Incorrect documentation for x-vpc lookup.

Add AutoScaling Support

Feature Request

Feature request to add autoscaling support for ECS workloads deployed. Autoscaling support should be added for the following:

• CPU utilization - (ECS)
• Memory utilization - (ECS)
• Request Count Per Target (ALB)
• Target Response Time (ALB)
• HTTPCode_Target_3XX_Count, HTTPCode_Target_4XX_Count, HTTPCode_Target_5XX_Count (ALB)

Some additional ones that would be helpful:

• ApproximateNumberOfMessagesVisible (SQS)

Use Cases

• Autoscale ECS workloads based upon CPU/Memory metrics
• Autoscale ECS workloads based upon ALB metrics (this is helpful if a given workload deployed on ECS has a certain performance characteristic that would require a faster scaling out response instead of waiting for CPU/Memory on ECS to get to a specific level)
• Autoscale on SQS queue when X number of messages are visible (this would allow a ratio mapping of the number of ECS tasks per number of messages)

Custom DependsOn.Condition for sidecar containers

Is your feature request related to a problem? Please describe.
I'd like to use a sidecar container to run my migrations when my service starts. This container will run and then exit. My main container should not start until this has finished. If it exits with a failure, the main service container should not start at all.

Describe the solution you'd like
Currently additional containers using ecs.task.family are hardcoded to a DependsOn.Condition of START or HEALTHY. I would like to be able to override this to SUCCESS

https://github.com/lambda-my-aws/ecs_composex/blob/f6021ec64ca253093985c428abf08d1a92f04d48/ecs_composex/ecs/ecs_service.py#L158

Enable VPC Flow Logs

As a Cloud Engineer, I want to be able to enable VPC Flow Logs for my subnets.
I want the resources created for the flow logs to be independent from the core VPC stack so I can use the CFN template for VPC Flow Logs as an addon, and ensure that when the stack is deleted, the logs remain so I can do forensic or archival accordingly.

VPC CLI does not respect correctly the AWS AZ argument

Used --az eu-west-1a and it created it for all zones in eu-west-1