alphagov/govuk-infrastructure

Terraform turnup automation for the EKS Kubernetes clusters that host GOV.UK. See https://github.com/alphagov/govuk-helm-charts for application config.

GOV.UK Infrastructure

What's in this repo

The govuk-infrastructure repo contains:

• terraform/: Terraform modules for turning up an Kubernetes cluster on EKS for GOV.UK.
• images/: Container image definitions for utilities such as the toolbox image.
• .github/: GitHub Actions and workflows used by other GOV.UK repos, for example release automation, test runners and security analysis tools.

What's not in this repo

Helm charts for GOV.UK applications are in alphagov/govuk-helm-charts.

Base image definitions for GOV.UK Ruby apps are in alphagov/govuk-ruby-images.

Some AWS services for GOV.UK are still configured using the legacy alphagov/govuk-aws (public) and alphagov/govuk-aws-data (private) repos.

Usage

To install the currently-used version of Terraform:

brew install tfenv
cd terraform/
tfenv install

Pre-commit hooks

We have some recommended pre-commit hooks. You need to install pre-commit for these to run.

Documentation

See the docs/ directory.

There are also docs in terraform/docs/ and inline READMEs in some directories.

Team

GOV.UK Platform Engineering team looks after this repo. If you're inside GDS, you can find us in #govuk-platform-engineering or view our kanban board.

Licence

MIT License

govuk-infrastructure's Issues

Move resources out of govuk-aws and retire it

Move all of the still in-use resources out of govuk-aws and into this repository so everything is managed in one place by Terraform Cloud

Move the rest of govuk-infrastructure into Terraform Cloud

Terraform Cloud SSO user/team mappings from via SAML / Google Workspace aren't working.

We have Terrform Cloud hooked up to Google Workspace for single-sign-on, but it isn’t assigning groups to people correctly. Pretty sure it used to work back when we first started using TFC.

This means we’re having to manually assign groups as people request them, which is toil (and raises the chance of mistakes).

Update terraform-aws-eks module from 19 to 20.

https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/UPGRADE-20.0.md#upgrade-from-v19x-to-v20x

Amazon Controllers for Kubernetes

We might be able to reduce toil and offer more flexibility to dev teams (and a better experience) by using Amazon Controllers for Kubernetes to manage AWS resources declaratively.

Most of what's currently under terraform/deployments/govuk-publishing-infrastructure would potentially benefit from this.

Last time we looked at ACK was a few years ago and it's probably matured since then so worth another look.

EKS IRSA Roles

Hello 👋🏽 !

I'm one of the maintainers of https://github.com/terraform-aws-modules

I see you are using the iam-assumable-role-with-oidc sub-module and just wanted to let you know we have created some commonly used EKS IRSA roles with policies defined for common addons that might be of interest. This sub-module for EKS IRSA roles has been designed in conjunction with the terraform-aws-eks module.

• iam-role-for-service-accounts-eks sub-module
• iam-role-for-service-accounts-eks examples

No action is required - just wanted to share that information if you were not aware. Thank you and feel free to close this issue at any time!

Validate syntax of `.tfvars` and `.backend` config files pre-merge.

We don't currently syntax-check these files pre-merge, but we should. #1024 is an example in the wild (merge markers in a .backend file but tests still pass).

Use OIDC connection for Github Actions to authenticate with AWS

We're using long-lived credentials at the moment, we should move to using an IAM role and the Github Actions -> AWS OIDC connection

Prefer `aws_iam_policy_document` to `policy = jsonencode(...)`.

aws_iam_policy_document provides better ahead-of-time checking and helps avoid some IAM gotchas. It also makes policies easier to read by cutting out boilerplate and simplifying the syntax.

Here's one I made earlier: 42402a00

Grep for cases that need fixing:

git grep 'policy *= *jsonencode'

Extra credit: implement a lint rule for this.
Extra extra credit: implement a fixer for this!

Fix cloudfront module always wanting to make changes

The cloudfront module always wants to make the following change to both assets/www cloudfront distributions, even if nothing has changed:

default_cache_behavior { 
  default_ttl: 0 -> 86400
  max_ttl: 0 ->  31536000
}

Update k8s 1.27 -> 1.28

1.28 has been available for a while now, so we should update to it to keep current.

https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html

https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions-standard.html#kubernetes-1.28

Can't test alternate CDN for assets domain in staging?

Seems there was some confusion around how assets.staging.publishing.service.gov.uk is set up in regard to testing switching between the main (Fastly) and alternative (AWS CloudFront) CDNs.

We should be able to test the assets domain on the alt CDN. Might already be fine and there was just confusion about how to do it, or maybe there's something to fix/improve about the way it's set up.

Moved from https://trello.com/c/vXZ5WQNd.

Upgrade RabbitMQ to a vendor-supported version

We're currently still running RabbitMQ 3.9. Amazon still claims to support this, but it's well past end-of-life with the RabbitMQ maintainers (Broadcom/VMware).

Upgrade to 3.11, which is the latest version that Amazon supports.

Kubernetes 1.29

We're currently on 1.28 but 1.29 is available for EKS now (since 23 Jan).

EKS Release notes

k8s release announcement

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• Update aws-actions/configure-aws-credentials action to v4.0.2

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update Helm release argo-bootstrap to v0.3.4
• Update Helm release argo-cd to v6.7.11
• Update public.ecr.aws/lts/ubuntu Docker tag to v22.04
• Update Helm release argo-workflows to v0.41.1
• Update Helm release aws-load-balancer-controller to v1.7.2
• Update Helm release tempo-distributed to v1.9.1
• Update TFLint plugin terraform-linters/tflint-ruleset-aws to v0.30.0
• Update Terraform alexbasista/workspacer/tfe to v0.10.0
• Update Terraform sentry to v0.12.2
• Update Terraform tfe to v0.53.0
• Update clamav/clamav-debian Docker tag to v1.3
• Update dependency hashicorp/terraform to v1.8.0
• Update Terraform fastly to v5
• Update Terraform github to v6
• Update Terraform terraform-aws-modules/eks/aws to v20
• Update Terraform terraform-aws-modules/iam/aws to v5
• Update Terraform terraform-aws-modules/rds-aurora/aws to v9
• Update actions/cache action to v4
• Update actions/setup-node action to v4
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

dockerfile

images/clamav/Dockerfile

• clamav/clamav-debian 1.2

images/mongodb/Dockerfile

• public.ecr.aws/lts/ubuntu 22.04_stable

images/toolbox/Dockerfile

• public.ecr.aws/lts/ubuntu 22.04_stable
• peakcom/s5cmd v2.2.2

github-actions

.github/actions/precompile-rails-assets/action.yml

• actions/cache v3

.github/actions/setup-node/action.yml

• actions/setup-node v3

.github/workflows/autorelease-rubygem.yml

• actions/checkout v4
• ruby/setup-ruby v1

.github/workflows/brakeman.yml

• actions/checkout v4
• ruby/setup-ruby v1

.github/workflows/build-and-push-image.yml

• actions/checkout v4
• aws-actions/configure-aws-credentials v4.0.1
• aws-actions/amazon-ecr-login v2
• docker/setup-buildx-action v3
• docker/metadata-action v5
• docker/build-push-action v5

.github/workflows/build-and-push-multiarch-image.yaml

• actions/checkout v4
• aws-actions/configure-aws-credentials v4.0.1
• aws-actions/amazon-ecr-login v2
• docker/setup-qemu-action v3
• docker/setup-buildx-action v3
• docker/metadata-action v5
• docker/build-push-action v5
• actions/upload-artifact v4
• actions/download-artifact v4
• docker/setup-buildx-action v3
• aws-actions/configure-aws-credentials v4.0.1
• aws-actions/amazon-ecr-login v2
• docker/metadata-action v5

.github/workflows/build-clamav-image.yml

.github/workflows/build-mongodb-image.yml

.github/workflows/build-toolbox-image.yml

.github/workflows/ci-terraform.yml

• actions/checkout v4
• dflook/terraform-version fddf858c28a7d454135c9973c7445d33a31f936e
• hashicorp/setup-terraform v3
• actions/cache v4
• actions/cache v4
• terraform-linters/setup-tflint v4

.github/workflows/codeql-analysis.yml

• actions/checkout v4
• github/codeql-action v3
• github/codeql-action v3
• github/codeql-action v3

.github/workflows/dependency-review.yml

• actions/checkout v4
• actions/dependency-review-action v4
• actions/checkout v4
• actions/dependency-review-action v4

.github/workflows/deploy.yml

.github/workflows/jasmine.yml

• actions/checkout v4
• ruby/setup-ruby v1

.github/workflows/mirror-repos.yml

• aws-actions/configure-aws-credentials v4.0.1@010d0da01d0b5a38af31e9c3470dbfdabdecca3a

.github/workflows/publish-rubygem.yml

• actions/checkout v4
• ruby/setup-ruby v1

.github/workflows/release.yml

• actions/checkout v4
• actions/github-script v7

.github/workflows/rubocop.yml

• actions/checkout v4
• ruby/setup-ruby v1

.github/workflows/set-automatic-deploys.yml

.github/workflows/snyk-security.yml

• actions/checkout v4
• actions/cache v4
• github/codeql-action v3
• actions/checkout v4
• actions/cache v4
• github/codeql-action v3

.github/workflows/standardx.yml

• actions/checkout v4

.github/workflows/stylelint.yml

• actions/checkout v4

terraform

terraform/deployments/cloudfront/main.tf

• hashicorp/terraform ~> 1.5

terraform/deployments/cluster-infrastructure/aws_ebs_csi_iam.tf

• terraform-aws-modules/iam/aws ~> 4.0

terraform/deployments/cluster-infrastructure/aws_lb_controller_iam.tf

• terraform-aws-modules/iam/aws ~> 4.0

terraform/deployments/cluster-infrastructure/cluster_autoscaler_iam.tf

• terraform-aws-modules/iam/aws ~> 4.0

terraform/deployments/cluster-infrastructure/external_dns.tf

• terraform-aws-modules/iam/aws ~> 4.0

terraform/deployments/cluster-infrastructure/external_secrets_iam.tf

• terraform-aws-modules/iam/aws ~> 4.0

terraform/deployments/cluster-infrastructure/grafana.tf

• terraform-aws-modules/rds-aurora/aws ~> 8.5
• terraform-aws-modules/iam/aws ~> 4.0

terraform/deployments/cluster-infrastructure/main.tf

• aws ~> 5.0
• hashicorp/terraform ~> 1.5
• terraform-aws-modules/eks/aws ~> 19.0

terraform/deployments/cluster-services/argo.tf

• argo-bootstrap 0.3.2
• argo-cd 6.7.7
• argo-workflows 0.40.14

terraform/deployments/cluster-services/argo_workflows.tf

• terraform-aws-modules/iam/aws ~> 5.5

terraform/deployments/cluster-services/aws_lb_controller.tf

• aws-load-balancer-controller 1.4.4
• ingress-class 0.1.1

terraform/deployments/cluster-services/external_secrets.tf

• cluster-secret-store 0.1.1
• cluster-secrets 0.9.5

terraform/deployments/cluster-services/kubescape.tf

• armo-cluster-components 1.7.18

terraform/deployments/cluster-services/logging.tf

• filebeat 8.5.1

terraform/deployments/cluster-services/main.tf

• aws ~> 5.0
• helm ~> 2.0
• kubernetes ~> 2.0
• hashicorp/terraform ~> 1.5

terraform/deployments/cluster-services/tempo.tf

• tempo-distributed 1.7.0
• terraform-aws-modules/iam/aws ~> 5.27

terraform/deployments/datagovuk-infrastructure/argo_bootstrap.tf

• argo-bootstrap 1.1.0

terraform/deployments/datagovuk-infrastructure/ckan_iam.tf

• terraform-aws-modules/iam/aws ~> 4.0

terraform/deployments/datagovuk-infrastructure/main.tf

• aws ~> 5.0
• fastly ~> 5.7
• hashicorp/terraform ~> 1.5

terraform/deployments/ecr/main.tf

• aws ~> 5.0
• github ~> 5.0
• hashicorp/terraform ~> 1.5

terraform/deployments/github/main.tf

• aws ~> 5.0
• github ~> 5.23
• hashicorp/terraform ~> 1.5

terraform/deployments/govuk-publishing-infrastructure/db_backup_s3.tf

• terraform-aws-modules/iam/aws ~> 5.20

terraform/deployments/govuk-publishing-infrastructure/govuk_mirror_sync.tf

• terraform-aws-modules/iam/aws ~> 5.28

terraform/deployments/govuk-publishing-infrastructure/main.tf

• aws ~> 5.0
• fastly ~> 2.1
• random ~> 3.1
• tfe 0.51.1
• hashicorp/terraform ~> 1.5

terraform/deployments/govuk-publishing-infrastructure/search_api_learn_to_rank_role.tf

• terraform-aws-modules/iam/aws ~> 5.20

terraform/deployments/rds/main.tf

• aws ~> 5.0
• random ~> 3.6

terraform/deployments/sentry/provider.tf

• sentry 0.11.2
• hashicorp/terraform ~> 1.5

terraform/deployments/tfc-bootstrap/main.tf

• alexbasista/workspacer/tfe 0.9.0

terraform/deployments/tfc-bootstrap/provider.tf

• tfe ~> 0.47.0
• hashicorp/terraform ~> 1.5

terraform/deployments/tfc-configuration/aws/provider.tf

• aws ~> 5.5
• tfe ~> 0.47.0
• hashicorp/terraform ~> 1.5

terraform/deployments/tfc-configuration/cloudfront.tf

• alexbasista/workspacer/tfe 0.10.0
• alexbasista/workspacer/tfe 0.10.0

terraform/deployments/tfc-configuration/cluster-infrastructure.tf

• alexbasista/workspacer/tfe 0.10.0
• alexbasista/workspacer/tfe 0.10.0
• alexbasista/workspacer/tfe 0.10.0

terraform/deployments/tfc-configuration/cluster-services.tf

• alexbasista/workspacer/tfe 0.10.0
• alexbasista/workspacer/tfe 0.10.0
• alexbasista/workspacer/tfe 0.10.0

terraform/deployments/tfc-configuration/datagovuk-infrastructure.tf

• alexbasista/workspacer/tfe 0.9.0
• alexbasista/workspacer/tfe 0.9.0
• alexbasista/workspacer/tfe 0.9.0

terraform/deployments/tfc-configuration/ecr.tf

• alexbasista/workspacer/tfe 0.10.0

terraform/deployments/tfc-configuration/govuk-publishing-infrastructure.tf

• alexbasista/workspacer/tfe 0.10.0
• alexbasista/workspacer/tfe 0.10.0
• alexbasista/workspacer/tfe 0.10.0

terraform/deployments/tfc-configuration/provider.tf

• tfe 0.51.1
• hashicorp/terraform ~> 1.5

terraform/deployments/tfc-configuration/rds.tf

• alexbasista/workspacer/tfe 0.10.0
• alexbasista/workspacer/tfe 0.10.0
• alexbasista/workspacer/tfe 0.10.0

terraform/deployments/tfc-configuration/variables-common.tf

terraform/deployments/tfc-configuration/variables-integration.tf

terraform/deployments/tfc-configuration/variables-production.tf

terraform/deployments/tfc-configuration/variables-staging.tf

terraform/deployments/tfc-configuration/vpc.tf

• alexbasista/workspacer/tfe 0.9.0
• alexbasista/workspacer/tfe 0.9.0
• alexbasista/workspacer/tfe 0.9.0

terraform/deployments/vpc/main.tf

• aws ~> 5.0
• hashicorp/terraform ~> 1.5

terraform-version

terraform/.terraform-version

• hashicorp/terraform 1.7.0

tflint-plugin

terraform/deployments/cluster-infrastructure/.tflint.hcl

• terraform-linters/tflint-ruleset-aws 0.25.0

terraform/deployments/cluster-services/.tflint.hcl

• terraform-linters/tflint-ruleset-aws 0.25.0

terraform/deployments/datagovuk-infrastructure/.tflint.hcl

• terraform-linters/tflint-ruleset-aws 0.25.0

terraform/deployments/ecr/.tflint.hcl

• terraform-linters/tflint-ruleset-aws 0.25.0

terraform/deployments/github/.tflint.hcl

• terraform-linters/tflint-ruleset-aws 0.25.0

terraform/deployments/govuk-publishing-infrastructure/.tflint.hcl

• terraform-linters/tflint-ruleset-aws 0.25.0

terraform/deployments/tfc-configuration/.tflint.hcl

• terraform-linters/tflint-ruleset-aws 0.25.0

Drop the `eks.` subdomain from `govuk.digital.` domains

Very low priority enhancement. As we no longer are two sets of infrastructure in parallel, having the eks. subdomain for our govuk.digital. domains isn't very useful. We don't need to distinguish between sets of infrastructure, nor do we really need to know that it's host on EKS.

We could simplify domains for example:

argo.eks.integration.govuk.digital -> argo.integration.govuk.digital

Saves a few extra keystrokes when filling in a URL (particularly before autocomplete kicks in for the environment).

Realise that this might be high effort/risk vs value, due to things like Dex callback URLs, CDN config etc etc.

Snyk reusable workflow downloads + installs Snyk binaries on every run.

This burns resources and runner time (which we pay for on private repos when we go over our included quota, which we've been doing lately).

It also occasionally causes flakes (unsurprising — if you download a 50M file enough times you're bound to see it fail once in a while), which is annoying and an unnecessary test dependency.

We need to be caching this stuff so that we're not being wasteful with runner minutes (and other resources).

Example output showing a (failed) download

Run /home/runner/work/_actions/snyk/actions/master/setup/setup_snyk.sh latest Linux
  /home/runner/work/_actions/snyk/actions/master/setup/setup_snyk.sh latest Linux
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
Installing the latest version of Snyk on Linux
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 83 44.2M   83 37.0M    0     0  36.1M      0  0:00:01  0:00:01 --:--:-- 36.1M
 97 44.2M   97 43.1M    0     0  40.0M      0  0:00:01  0:00:01 --:--:-- 39.9M
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104
Error: Process completed with exit code 56.

From https://github.com/alphagov/signon/actions/runs/7729965195/job/21074309271#step:3:17

Update amazon-load-balancer-controller from 2.4 to 2.7

This was blocked for a while on k8s upgrades some time ago, but we've since kinda forgotten about it.

Definitely not one to YOLO: we'll wanna check all the release notes and verify that our complicated ingress configs like www-origin still reconcile ok in the staging cluster before pushing this to prod.

Update nodes to Amazon Linux 2023.

This gets us kernel 6 as well as a bunch of other good stuff.

https://docs.aws.amazon.com/linux/al2023/ug/compare-with-al2.html

https://aws.amazon.com/blogs/containers/amazon-eks-optimized-amazon-linux-2023-amis-now-available/

Configure custom issuer for GitHub Actions OIDC.

This straightforward change would help to mitigate accidental misconfiguration, without any significant downside (not likely to increase complexity, brittleness, maintenance burden or introduce any UX issues).

https://docs.github.com/en/enterprise-cloud@latest/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect#customizing-the-issuer-value-for-an-enterprise

https://awsteele.com/blog/2023/01/11/improve-github-actions-oidc-security-posture-with-custom-issuer.html

Kudos to @christophetd and @aidansteele for putting us onto this.

Run actionlint as a pre-merge check.

We have a bunch of reusable GitHub Actions workflows and actions in this repo. GitHub Actions is full of footshooters so there's a lot of value in running a linter to catch common gotchas pre-merge.

https://github.com/rhysd/actionlint