stroeer/terraform-aws-ecs-fargate

Terraform module for ECS Fargate Services

terraform-aws-ecs-fargate's Issues

Add policy attachment to task role for firehose

To simplify client code, we could attach an IAM policy to the existing ecs task role in order to allow containers to put log events into firehose.

Module clients could remove the following code and rely on the module:

data "aws_iam_policy_document" "policy" {
  statement {
    actions = ["firehose:PutRecordBatch"]
    // todo limit this service stream
    resources = ["*"]
  }
}

Replacing target groups fails

If the a ALB target group needs to be replaced (e.g. aws_alb_target_group.public because the container_port changed), this fails with:

Error deleting Target Group: ResourceInUse: Target group 'arn:aws:elasticloadbalancing:eu-west-1:12345678:targetgroup/my-service/1541e04161e7f3ef' is currently in use by a listener or a rule

see https://github.com/telia-oss/terraform-aws-ecs-fargate/blob/0f3cbc67c057625f1354f28f6ad4efc3763a8978/main.tf#L208 for a possible solution

BUG: Call to function "lookup" failed: lookup failed to find key "resource_label".

│ Error: Error in function call
│ 
│   on .terraform/modules/alb-ecs-service/main.tf line 278, in resource "aws_appautoscaling_policy" "ecs":
│  278:       resource_label         = lookup(var.appautoscaling_settings, "resource_label")
│     ├────────────────
│     │ while calling lookup(inputMap, key, default...)
│     │ var.appautoscaling_settings is map of string with 7 elements
│ 
│ Call to function "lookup" failed: lookup failed to find key "resource_label".

ECR scan on push should be enabled by default

deployment module relies on existing IAM roles

The deployment (sub) module uses data sources of existing IAM roles for codepipeline and codebuild.

There is no documentation about the necessary permissions for those roles and their names can't be configured so this won't be usable outside our team.

Options:

• create those roles inside the deployment (sub) module (service specific naming)
• document required permissions and make ARNs of the required roles configurable

I'd prefer option one since it's more coherent IMO and requires less upfront terraforming outside the module.

description of deployment modules variables

the variables of the deployment (sub) module have no description at the moment.

deployment module relies on existing S3 bucket

The deployment (sub) module uses a data source of an existing S3 bucket for storing pipeline artifacts. This bucket is (re-used) for all pipelines.

There is no documentation about this bucket and it's name can't be configured so this won't be usable outside our team.

Options:

• create the bucket (one for each service) inside the deployment (sub) module (service specific naming)
• make id of required bucket configurable

I'd prefer option one since it's more coherent IMO and requires less upfront terraforming outside the module.

Provide log group for fluent-bit sidecar

With enabled logs sub-module, we should provide the opportunity to provide an existing CloudWatch Log Group for logs of the fluent-bit sidecar or create one (which should be the default).

using module w/o deployment pipeline fails

If create_deployment_pipeline is set to false, terraform plan fails with:

Error: Failed getting S3 bucket: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadBucketInput.Bucket.
 Bucket: ""

  on .terraform/modules/client/terraform-aws-ecs-fargate-0.1.1/modules/deployment/s3.tf line 13, in data "aws_s3_bucket" "codepipeline":
  13: data "aws_s3_bucket" "codepipeline" {

Add contribution guidelines

Add a CONTRIBUTING.md describing how to contribute to this project.

Remove dependencies to 3rd party terraform modules

We might remove internal dependencies to 3rd party terraform modules like terraform-aws-modules/s3-bucket/aws which cuts interdependencies in terms of required/supported aws provider and terraform versions.

Creating new services with `0.29.1` fails

│ Error: Invalid count argument
│ 
│   on .terraform/modules/services.otel_container_definition/modules/assert/main.tf line 2, in data "external" "assertion":
│    2:   count   = var.condition ? 0 : 1
│ 
│ The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To
│ work around this, use the -target argument to first apply only the resources that the count depends on.
╵
make: *** [tf] Error 1

Workaround: apply using 0.29.0 first.

Elasticsearch domain is not configurable

If using the log sub-module, an Elasticsearch domain named application-logs is expected. This variable is not yet configurable from the main module.

use newer codebuild image for faster provisioning

Problem Description

The Provisioning phase for the CodeBuild step currently takes several minutes, which slows down deployments of ecs tasks by a lot.

Quick Analysis & Solution Proposal

• Currently we use aws/codebuild/amazonlinux2-x86_64-standard:1.0, which is pretty old.
• The Internet says that older images get dropped on the AWS caches, so we should probably switch to a newer image.
• According to this, aws/codebuild/amazonlinux2-x86_64-standard:3.0 should be the one to to switch to:
https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-available.html

firehose todo's

• example(s)
• documentation

Remove all internal specific datasources/remote states

We should list and cut all dependencies to our internal datasources and remote states, to make this module more generic to team/organization external users.

Affected resources:

• ssm_ecs_task_execution_role data source
• aws_service_discovery_service namespace_id

module fails with terraform 0.13

This module can't be applied using terraform 0.13. It fails with Error: ECR Repository (service-name) not found.

Details:

2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: 2020/08/25 09:49:16 [DEBUG] [aws-sdk-go] DEBUG: Response ecr/DescribeRepositories Details:
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: ---[ RESPONSE ]--------------------------------------
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: HTTP/1.1 400 Bad Request
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: Connection: close
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: Content-Length: 153
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: Content-Type: application/x-amz-json-1.1
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: Date: Tue, 25 Aug 2020 07:49:16 GMT
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: X-Amzn-Requestid: 99522912-b9c0-44a3-9be2-1fc3b09ef2a7
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5:
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5:
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: -----------------------------------------------------
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: 2020/08/25 09:49:16 [DEBUG] [aws-sdk-go] {"__type":"RepositoryNotFoundException","message":"The repository with name 'service-name' does not exist in the registry with id '12432442342'"}
2020-08-25T09:49:16.831+0200 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: 2020/08/25 09:49:16 [DEBUG] [aws-sdk-go] DEBUG: Validate Response ecr/DescribeRepositories failed, attempt 0/25, error RepositoryNotFoundException: The repository with name 'service-name' does not exist in the registry with id '12432442342'
2020/08/25 09:49:16 [ERROR] eval: *terraform.evalReadDataPlan, err: ECR Repository (service-name) not found
2020/08/25 09:49:16 [ERROR] eval: *terraform.EvalSequence, err: ECR Repository (service-name) not found

remove tagging in favor of `provider.default_tags`

There is an easier way to automatically add tags to all resources, see blog post or aws tf provider docs

unmanaged CloudWatch log groups for CodeBuild

aws_codebuild_project(https://www.terraform.io/docs/providers/aws/r/codebuild_project.html) defaults to create a CloudWatch log group which is not managed by terraform lifecycle. Thus those log groups are not destroyed.

If those log groups are needed at all, we should manage them in the module (including definition of retention time).

Enhance examples

We should provide examples depicting major features of this module. Those examples need to be terraform apply'able by users without errors.

Possible examples:

• complete example with all basic variables set
• with ALB
• with deployment pipeline
• with app mesh
• with NLB

Support for more than just `production`

It would be useful if the deployment process supported more than only production as a tag in ECR, or at least allow this tag to be specified.

support apps w/o exposed ports

Providing a container_port used for the load balancer and/or mesh proxy configuration is mandatory at the moment. We might want to support apps w/o an exposed port as well, e.g. apps consuming Kinesis/Dynamodb streams or sqs queues.

Review deployment pipeline permissions

With #14 we introduced an IAM role with required permissions to run CodePipeline for services. This role currently contains commented code and it's not finally clear, which permissions are actually needed.

Separate deployment module

Currently the deployment is part of the main stack, which can be deactivated using a variable.

We might also separate those 2 stacks like in terraform-aws-lambda/tree/master/modules/deploy.

Unable to deploy a service with cloudwatch.enabled=false and no task_role_arn

AWS `source` tag should be overrideable

All aws resources get default tags when created with this module.

When I create an ECS service via another github repository where I use terraform-aws-buzzgate the source tag should be of the service repository and not of github.com/stroeer/terraform-aws-buzzgate.

Example
in the github.com/stroeer/polyphase repo i use this module and I create aws resources with tf files, that are in the given repo. So to be able to terraform these resources you would need to know, that the tf files reside in github.com/stroeer/polyphase and not in github.com/stroeer/terraform-aws-buzzgate

send app logs directly to ES via fluent-bit

Monitoring aws/containers-roadmap#698 in order to send app logs directly to Elasticsearch instead of piping through Kinesis Firehose streams.