ministryofjustice/staff-device-shared-services-infrastructure

Staff Device AWS Infrastructure for build pipelines

MoJ Official Shared Services Infrastructure

This creates the shared infrastructure for the main account, named Shared Services. This account is used to host AWS CodePipeline CI/CD pipelines, for new CI/CD use GitHub Actions as per ADR 011.

For the code that creates infrastructure for each environment please see this repository, as an example.

This repository holds the Terraform code to create a CodeBuild / CodePipeline service in AWS.

Applying the terraform

You will require Docker

To apply the Terraform in this project using AWS Vault to authenticate:

1. Prepare your working directory for Terraform

make init

2. Check the changes in with a plan

 make plan

3. Apply the changes

make apply

To view all the available target commands in the Makefile just type

make

How to use this repo

The source code in this repository is provided only as a reference.

Please consult with someone on the Cloud Ops team before you use this repository to have a pipeline set up for your own project.

The pipeline you set will be integrated with a GitHub repository, and will build your project according to your buildspec files.

This repository upon execution will create a couple of s3 buckets and a DynamoDB table. So, if your project uses Terraform, make sure that the backend for that Terraform is configured to use the newly created s3 bucket and the DynamoDB table.

Depending on your build process, you may require 3 files to do linting, testing and deployment.

Linting

If you are doing static code analysis as part of your build, please create a buildspec.lint.yml file, and place it in the root of your project.

example:

version: 0.2

phases:
  install:
    commands:
      - make lint

Testing

To run automated tests, create a buildspec.test.yml file, and place it in the root of your project.

example:

version: 0.2

phases:
  install:
    commands:
      - make test

Deployment

For deployments, create a buildspec.yml file.

example:

version: 0.2

env:
  variables:
    key: "value"
    key: "value"

phases:
  install:
    commands:
      - pip install boto3
      - wget https://releases.hashicorp.com/terraform/0.12.24/terraform_0.12.24_linux_amd64.zip
      - unzip terraform_0.12.24_linux_amd64.zip
      - mv terraform /bin
      - terraform init
  build:
    commands:
      - terraform apply --auto-approve

To create your own Pipeline

To have a Pipeline for your own project with AWS CodePipeline / CodeBuild, you can execute the Terraform in this repository.

Re-use the module ./modules/ci-pipeline in the main.tf file to setup your own Pipeline.

Run Terraform

make apply

Secrets management

We use SSM Parameter store for all secrets.

These secrets are decrypted at build time on CI to inject into Terraform.

To add or update a secret:

make shell

aws ssm put-parameter --name "/your/top/secret/name" \
  --key-id "kms key ID to encrypt with" \
  --description "Secret description" \
  --type SecureString \
  --value "tops3cr3t" \
  --overwrite

Pipeline flags

2 flags exist for pipelines and can be turned on or off when invoking the pipeline module.

manual_production_deploy

This option adds a stage to the pipeline where manual confirmation is required before deploying to production.

production_plan

This option adds stage where changes to infrastructure can be inspected before applying. Typically used in combination with the manual_production_deploy. This will set an environment variable on the stage of PLAN="true". Buildspec files can be modified to look for the existence of this variable to do either a terraform plan or terraform apply.

CI CD with GitHub Actions

We have the following repository which we use for shared Actions.

Following table displays all the repositories nvvs-devops-admins GithUb team have access to

Owner	REPO NAME	AWS/Github	Description
operations-engineering	aws-root-account	Github	Terraform for the Ministry of Justice AWS root account
Gary H	aws-ta-testing	Delete?	Terraform for testing
nvvs-devops-admins	aws-trusted-advisor-to-github-issues	Github	Automates creation of GitHub Issues from AWS Trusted Advisor checks
nvvs-devops-admins	cloud-operations-slack-bot	Github	Official Slack bot app for Cloud Operations team Slack channel
nvvs-devops-admins	deployment-tgw	Manual?
nvvs-devops-admins	mojo-aws-github-oidc-provider	Manual?	To manage GitHub AWS OpenID Connector provider on MoJO AWS Shared Services account.
nvvs-devops-admins	network-access-control-admin	AWS CodePipeline	Self service admin portal for the Network Access Control Service
nvvs-devops-admins	network-access-control-disaster-recovery	Manual?	Rollback scripts for S3 configuration and ECR containers
nvvs-devops-admins	network-access-control-infrastructure	AWS CodePipeline	Terraform infrastructure for the 802.1x Network Access Control Service
nvvs-devops-admins	network-access-control-integration-tests	Manual?	Integration tests for the Network Access Control Service
nvvs-devops-admins	network-access-control-server	AWS CodePipeline	FreeRadius server for the 802.1x Network Access Control Service
nvvs-devops-admins	nvvs-devops	Github	Documentation for the NVVS DevOps Team
nvvs-devops-admins	nvvs-devops-github-actions	Github	Used for Workflow GitHub actions from other repositories
nvvs-devops-admins	nvvs-devops-monitor	Github	Terraform to create new VPC and EKS Cluster for the MoJO IMA
nvvs-devops-admins	PaloAlto-pipelines	Manual ??	Terraform configuration to manage AWS CodePipelines
nvvs-devops-admins	provision-ubuntu2004-on-wsl2	Manual (y)	Automating provisoning ubuntu 20.04 with Ansible on WSL 2
nvvs-devops-admins	staff-device-dhcp-server	AWS CodePipeline	The ISC KEA server for serving DHCP requests (via a Docker image)
nvvs-devops-admins	staff-device-dns-dhcp-admin	AWS CodePipeline	Web frontend for managing Staff Device DNS / DHCP servers
nvvs-devops-admins	staff-device-dns-dhcp-disaster-recovery	Manual ?	Disaster recovery script for DNS and DHCP services.
nvvs-devops-admins	staff-device-dns-dhcp-infrastructure	AWS CodePipeline	Staff Device DHCP and DNS Terraform infrastructure
nvvs-devops-admins	staff-device-dns-server	AWS CodePipeline	Staff Device DNS Server repository
nvvs-devops-admins	staff-device-logging-dns-dhcp-integration-tests	Manual (y)	Remote full stack integration tests currently run from Corsham test site. Services being
nvvs-devops-admins	staff-device-logging-infrastructure	AWS CodePipeline	Log proxy and forwarding infrastructure
nvvs-devops-admins	staff-device-logging-syslog-to-cloudwatch	AWS CodePipeline	Docker container to forward syslog events to CloudWatch
nvvs-devops-admins	staff-device-management-intune-scripts	??	Scripts that are deployed to run on MoJ OFFICIAL devices managed through Microsoft Intune
nvvs-devops-admins	staff-device-private-dns-zone	Github	This repository contains the Terraform code to create and maintain private DNS zones in AWS Route 53.
nvvs-devops-admins	staff-device-shared-services-infrastructure	Manual	Staff Device AWS Infrastructure for build pipelines
nvvs-devops-admins	staff-infrastructure-admin-sso	Not Applicable	Terraform management of AzureAD Users and Groups for staff management services
nvvs-devops-admins	staff-infrastructure-certificate-services	Github	Infrastructure to support Public Key Infrastructure for devices users and applications
nvvs-devops-admins	staff-infrastructure-metric-aggregation-server	Github	This repository is for building our custom prometheus docker image with all the required config to pull data from our collectors
nvvs-devops-admins	staff-infrastructure-metric-aggregator-cloud	Not Applicable	Prometheus server for AWS Cloudwatch and Azure Monitor Exporters
nvvs-devops-admins	staff-infrastructure-monitoring-app-reachability	Github	Lightweight all-in-one docker image for monitoring http endpoints and shipping metrics back to a central prometheus over the internet.
nvvs-devops-admins	staff-infrastructure-monitoring-blackbox-exporter	Manual ??	This project is part of the Infrastructure Monitoring and Alerting (IMA) Platform. It holds the Docker image for pulling data from the Physical Devices.
nvvs-devops-admins	staff-infrastructure-monitoring-dns-reachability	Manual ??	To monitor MoJ Official DNS
nvvs-devops-admins	staff-infrastructure-monitoring-snmpexporter	Manual ??	This is an exporter that exposes information gathered from SNMP to be scraped by Prometheus.
nvvs-devops-admins	staff-infrastructure-network-operations	Manual ??	Repository for Network Operations Centre tooling
nvvs-devops-admins	staff-infrastructure-network-services	AWS CodePipeline	This repository deploys the underlying base infrastructure for several network based services for staff devices and applications in a single VPC in AWS.
nvvs-devops-admins	staff-infrastructure-smtp-relay-server	AWS CodePipeline	This repository builds the docker image for the SMTP Relay server and pushes it to the Shared Services Elastic Container Repository
nvvs-devops-admins	staff-technology-services-github-teams	Github	To define and maintain some GitHub teams for Technology Services in Code using Terraform.
nvvs-devops-admins	tech-docs-monitor	Github	Part of alphagov/tech-docs-template (issues 👉https://github.com/alphagov/tech-docs-template/issues)
nvvs-devops-admins	terraform-panorama-config	Manual ??
nvvs-devops-admins	transit-gateways	Manual ??

staff-device-shared-services-infrastructure's Issues

📊 Improve Monitoring of LBs

User Story

As an engineer
I expect to be able to be able to understand traffic flows on Load Balancers in the solution quickly and easily
So that I can quickly rule these out of being at fault when troubleshooting

Value / Purpose

During a recent incident it occurred to us that to check this would be difficult, it would be better if these metrics are in a dashboard.

Useful Contacts

No response

Additional Information

https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-access-logs.html

Definition of Done

• Logging to S3 buckets for all load balancer requests
• Metric
• Number of requests
• Latency of requests to the backend (connection time)

No response

Upgrade pipeline "staff-infrastructure-network-services" with new AWS codestar connection[internal repo]

User Story

this pipeline was not upgraded as a part of the upgrading GitHub version 2 source action as the existing AWS codestar connection does not have access to the internal repo in MOJ GitHub. A new AWS codestar connection has been created and pending for approval.

Value / Purpose

This will allow the pipeline to be compliant with AWS code pipeline recommendation.

Useful Contacts

No response

Additional Information

This issue was discovered during the progress on#63
This ticket is dependent on #68 being completed.

Definition of Done

• README has been updated
• the new codestar connection has been approved (#68)
• Pipeline has been upgraded to use GitHub version 2 source action in AWS pipeline
• Another team member has reviewed
• Tests are green

Configure CodeStar Connection

User Story

As a devops Engineer
I need to configure the newly created AWS Codestar Connection for use with the "nvvs-devops-admins" team repositories.
So that PRs on new and internal repositories that are not configured for use with an existing CodeStar Connection are able to trigger AWS CodePipeline builds.

Value / Purpose

During the spike for Issue #63 we discovered that final configuration of an AWS CodeStar Connection is manual and requires somebody with GitHub Organisation management rights to implement and approve.

We have created and deployed a new CodeStarConnection (CSC) from within Terraform but now require that last step to change the status of the CSC from pending.

Useful Contacts

Jake Mulley

Additional Information

@jakemulley has the role to complete this and has kindly agreed to

https://mojdt.slack.com/archives/C01BUKJSZD4/p1686815046088339?thread_ts=1686672663.104879&cid=C01BUKJSZD4

The AWS Account where the CodeStar Connection is
MOJ Official (Shared Services) #683290208331 | [email protected]

The Connection console URL

Name
nvvs github connection
Arn
arn:aws:codestar-connections:eu-west-2:683290208331:connection/494d5bc3-587a-4514-ad63-30805d6a494f

List of nvss-devops-admin repos

aws-ta-testing
aws-trusted-advisor-to-github-issues
cloud-operations-slack-bot
deployment-tgw
mojo-aws-github-oidc-provider
network-access-control-admin
network-access-control-disaster-recovery
network-access-control-infrastructure
network-access-control-integration-tests
network-access-control-server
nvvs-devops
nvvs-devops-github-actions
nvvs-devops-monitor
PaloAlto-pipelines
provision-ubuntu2004-on-wsl2
staff-device-dhcp-server
staff-device-dns-dhcp-admin
staff-device-dns-dhcp-disaster-recovery
staff-device-dns-dhcp-infrastructure
staff-device-dns-server
staff-device-logging-dns-dhcp-integration-tests
staff-device-logging-infrastructure
staff-device-logging-syslog-to-cloudwatch
staff-device-management-intune-scripts
staff-device-private-dns-zone
staff-device-shared-services-infrastructure
staff-infrastructure-admin-sso
staff-infrastructure-certificate-services
staff-infrastructure-metric-aggregation-server
staff-infrastructure-metric-aggregator-cloud
staff-infrastructure-monitoring-app-reachability
staff-infrastructure-monitoring-blackbox-exporter
staff-infrastructure-monitoring-dns-reachability
staff-infrastructure-monitoring-snmpexporter
staff-infrastructure-network-operations
staff-infrastructure-network-services
staff-infrastructure-smtp-relay-server
staff-technology-services-github-teams
tech-docs-monitor
terraform-panorama-config
transit-gateways

Repos specifically managed in this repository

staff-device-logging-infrastructure
staff-device-dns-dhcp-infrastructure
staff-device-dhcp-server
staff-device-logging-syslog-to-cloudwatch
staff-device-dns-dhcp-admin
staff-device-dns-server
staff-device-private-dns-zone *
network-access-control-infrastructure
network-access-control-server
network-access-control-admin
staff-infrastructure-network-services **
staff-infrastructure-smtp-relay-server

Definition of Done

• Another team member has reviewed
• Issue #67 can be completed and tested.

upgrade GitHub version 1 source action to a GitHub version 2 source action

User Story

Currently we are using Github version 1 to connect to all of our source repos in Github this is no longer a recommended way of connecting to the source code and AWS code pipeline may force us from continue using this in future.

Not recommended: The GitHub version 1 action uses OAuth tokens to authenticate with GitHub and uses a separate webhook to detect changes. This is no longer the recommended method.

Recommended: The GitHub version 2 action uses Github app-based auth backed by a [CodeStarSourceConnection for GitHub It installs an AWS CodeStar Connections application into your GitHub organization so that you can manage access in GitHub.

Value / Purpose

To be compliant and enabling secure integration of source to AWS code pipelines.

Useful Contacts

No response

Additional Information

https://docs.aws.amazon.com/codepipeline/latest/userguide/update-github-action-connections.html
https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/differences-between-github-apps-and-oauth-apps

Definition of Done

• Migrated all existing pipeline to use Github version 2 action to connect.
• staff-infrastructure-smtp-relay-server
• Staff-Device-DNS-Server
• Staff-Device-Admin-Portal
• staff-infrastructure-network-services **internal could not upgrade TODO
• network-access-control-admin
• staff-device-private-dns-zone **internal could not upgrade TODO
• Staff-Device-Logging
• network-access-control-infrastructure
• network-access-control-server
• Staff-Device-DNS-DHCP-Infrastructure
• Staff-Device-DHCP-Server
• Staff-Device-Logging-Syslog-To-Cloudwatch
• Documentation has been written / updated
• README has been updated
• Another team member has reviewed
• Tests are green

A branch protection setting is not enabled: administrators require review

Hi there
The default branch protection setting called administrators require review is not enabled for this repository
See repository settings/Branches/Branch protection rules
Either add a new Branch protection rule or edit the existing branch protection rule and select the Require a pull request before merging option
See the repository standards: https://github.com/ministryofjustice/github-repository-standards
See the report: https://operations-engineering-reports.cloud-platform.service.justice.gov.uk/github_repositories
Please contact Operations Engineering on Slack #ask-operations-engineering, if you need any assistance

Reinitiate SSM Parameterstore CodeBuild Role keys

User Story

As an engineer
I need to store the ARN of the "SharedServicesCodeBuild" role in SSM ParemeterStore with a project and environment specific key name
So that
The value can be retrieved by CI tooling, Engineers can reference the role and possibly a per project specific role can be created if necessary.

Value / Purpose

The SSM Parameter key is created by the ci-pipeline-webhook module in a very specific format for each project.
This no longer exists for projects such as "staff-device-private-dns-zone" as CodeBuild has been deleted and it's using the new GitHub Actions CI method.
This has blocked an update to the documentation and make file update for above project
https://app.zenhub.com/workspaces/nvvs-devops-622a0b371800e400133bb924/issues/gh/ministryofjustice/staff-device-private-dns-zone/39

Useful Contacts

No response

Additional Information

No response

Definition of Done

Example - [ ] Documentation has been written / updated

• README has been updated
• User docs have been updated
• Another team member has reviewed
• Tests are green

👀 Review changes to AWS CodePipelines IAM policies

User Story

Investigate if this is an issue.

Value / Purpose

We are reaching out to you because you have had at least one active pipeline with AWS CodePipeline in the last 4 weeks, and we would like to notify you of some updates to the service. As part of our continuous customer experience and security improvements, AWS CodePipeline is updating two IAM Managed Policies: AWSCodePipelineFullAccess and AWSCodePipelineReadOnlyAccess. The newest versions of these policies will have updated permissions that are scoped down even further, consistent with least privilege principles. These policies will be identical to the existing managed policies AWSCodePipeline_FullAccess and AWSCodePipeline_ReadOnlyAccess, respectively. Refer to the user guide for CodePipeline Managed policies [1] for more information.

We will begin enforcing the use of the newer versions of these policies, with further scoped down permissions on August 22, 2022. If you use either the AWSCodePipelineFullAccess or AWSCodePipelineReadOnlyAccess IAM Managed Policy, the updated versions will be in effect on August 22, 2022, and will be applied to your active CodePipelines.

If you have any questions or concerns, please contact AWS Support [2].

[1] https://docs.aws.amazon.com/codepipeline/latest/userguide/managed-policies.html
[2] https://aws.amazon.com/support

Useful Contacts

No response

Additional Information

No response

Definition of Done

• Changes to IAM have been reviewed

👁‍🗨 Issue to add labels

User access removed, access is now via a team

Hi there

The user emileswarts had Direct Member access to this repository and access via a team.

Access is now only via a team.

You may have less access it is dependant upon the teams access to the repo.

If you have any questions, please post in #ask-operations-engineering on Slack.

This issue can be closed.

Collaborator review date expires soon for user c-gyorfi

Hi there

The user @C-gyorfi has its access for this repository maintained in code here: https://github.com/ministryofjustice/github-collaborators

The review_after date is due to expire within one month, please update this via a PR if they still require access.

If you have any questions, please post in #ask-operations-engineering on Slack.

Failure to update the review_date will result in the collaborator being removed from the repository via our automation.

Review after date expiry is upcoming for user: richrace

Hi there

The user @richrace has its access for this repository maintained in code here:
https://github.com/ministryofjustice/github-collaborators/blob/main/README.md#github-external-collaborators

The review_after date is due to expire within one month, please update this via a PR if they still require access.

If you have any questions, please post in #ask-operations-engineering on Slack.

:wastebasket: Decommission AWS CodePipeline "staff-device-private-dns-zone"

User Story

As a DevOps Engineer
I want to decommission the AWS CodePipeline staff-device-private-dns-zone
So that this deprecated pipeline no longer is operational.

Value / Purpose

During the progression of Issue-63 on this same repository we became aware of the existence of GitHub Actions that had been developed to replace the AWS CodePipeline.
On further investigation we confirmed with previous team members that it should have been decommissioned.

Useful Contacts

tommoj

Additional Information

Slack discussion https://mojdt.slack.com/archives/C04MN9N2ZKN/p1686238321432309

Definition of Done

• Terraform code removed
• AWS CodePipeline removed
• README has been updated in https://github.com/ministryofjustice/staff-device-private-dns-zone
• Another team member has reviewed

Collaborator review date expires soon for user emileswarts

Hi there

The user @emileswarts has its access for this repository maintained in code here: https://github.com/ministryofjustice/github-collaborators

The review_after date is due to expire within one month, please update this via a PR if they still require access.

If you have any questions, please post in #ask-operations-engineering on Slack.

Failure to update the review_date will result in the collaborator being removed from the repository via our automation.

Separate main file into groups of related resources

User Story

As an engineer
I need expect to find aws resources managed by terraform grouped in related files rather one single big main file
So that it is easier to understand and read

Value / Purpose

Enable quicker maintenance and development.

Useful Contacts

No response

Additional Information

No response

Definition of Done

Example - [ ] Documentation has been written / updated

• README has been updated
• User docs have been updated
• Another team member has reviewed
• Tests are green

User access removed, access is now via a team

Hi there

The user C-gyorfi had Direct Member access to this repository and access via a team.

Access is now only via a team.

You may have less access it is dependant upon the teams access to the repo.

If you have any questions, please post in #ask-operations-engineering on Slack.

This issue can be closed.

S3 Bucket Versioning Audit

User access removed, access is now via a team

Hi there

The user emileswarts had Direct Member access to this repository and access via a team.

Access is now only via a team.

You may have less access it is dependant upon the teams access to the repo.

If you have any questions, please post in #ask-operations-engineering on Slack.

This issue can be closed.

Add S3 Bucket Versioning to Staff-device-dns-dhcp-infra

Context:
This ticket is to add S3 bucket versioning to repo ministryofjustice/staff-device-dns-dhcp-infrastructure/pulls as it is currently blocked and we cannot push into production.

Refactor module ci-codepipeline-webhook S3 bucket Terraform

User Story

As an
I expect to have a codebase that is compatible with the updated AWS Provider and Terraform Versions
So that
When running and test the code we don't have 71 warnings that

"Warning: Argument is deprecated"
Use the aws_s3_bucket_server_side_encryption_configuration resource instead
(and 71 more similar warnings elsewhere)

Value / Purpose

Keep code in a operational state.

Useful Contacts

No response

Additional Information

No response

Definition of Done

Example - [ ] Documentation has been written / updated

• README has been updated
• User docs have been updated
• Another team member has reviewed
• Tests are green

Collaborator review date expires soon for user emileswarts

Hi there

The user @emileswarts has its access for this repository maintained in code here: https://github.com/ministryofjustice/github-collaborators

The review_after date is due to expire within one month, please update this via a PR if they still require access.

If you have any questions, please post in #ask-operations-engineering on Slack.

Failure to update the review_date will result in the collaborator being removed from the repository via our automation.

Collaborator review date expires soon for user emileswarts

Hi there

The user @emileswarts has its access for this repository maintained in code here: https://github.com/ministryofjustice/github-collaborators

The review_after date is due to expire within one month, please update this via a PR if they still require access.

If you have any questions, please post in #ask-operations-engineering on Slack.

Failure to update the review_date will result in the collaborator being removed from the repository via our automation.

🔐 Regenerate AWS Codepipelines GitHub token

@bagg3rs commented on Thu May 19 2022

User Story

As a CloudOps Engineer
I need to update our GitHub token
So that it conforms to the new security standards

Value / Purpose

To understand more about this change and why it's important, visit https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats.

Useful Contacts

RB

Additional Information

// details of the account in LastPass.

Hi @staff-infrastructure-moj,

We noticed your personal access token, MoJ Official AWS Shared Services CodePipelines, has an outdated format and was used to access the GitHub API on May 19th, 2022 at 17:14 (UTC) with a user-agent header of AWS CodePipeline.

We recently updated the format of our API authentication tokens, providing additional security benefits to all our customers.

In order to benefit from this new format, please regenerate your personal access token, MoJ Official AWS Shared Services CodePipelines, using the button below.

Regenerate your personal access token

To understand more about this change and why it's important, visit https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats.

We encourage you to reset your other authentication tokens as well, or revoke any that are no longer needed. This applies to integrations (such as OAuth Apps and GitHub Apps) as well as personal access tokens. For more information on reviewing your authorized integrations, refer to https://docs.github.com/github/authenticating-to-github/reviewing-your-authorized-integrations.

Thanks,
The GitHub Team

Definition of Done

• GitHub machine account updated to CloudOps distribution list rather than user.
• token updated
• Tests are green