financial-times/upp-provisioners

Contains the various provisioning projects used by the Universal Publishing Platform

upp-provisioners

Contains the various provisioning projects used by the Universal Publishing Platform. See the relevant readme file of each project for more detail about each provisioner.

CircleCI Automated Builds

Automated builds for the provisioner projects are triggered in CircleCI.
The CircleCI configuration is located here.

Builds are triggered on commits and pull requests, and must pass to be able to merge into master. Note that only provisioners that have CircleCI configuration defined AND have been updated by your commits will be built.

Commits to branches are automatically built, tagged and pushed with the branch name as the Docker image tag.
Note that due to tag naming restrictions, branch names containing / will only use the second part as the tag.

To enable automated builds for new provisioner projects that contain a Dockerfile:

• Copy the configuration from an existing job, eg: upp-elasticsearch-provisioner
• Update the job name to match the name of the new provisioner

No further changes should be required, as the job config is fully parameterised.

Provisioners

• content-jenkins-provisioner

  • Docker image, running Ansible & CloudFormation to provision and decommission Content Jenkins infrastructure.

• content-eks-jumpbox-provisioner

  • Docker image runnning CloudFormation to provision and decommission Content Jumpbox

• upp-concept-publishing-provisioner

  • CloudFormation to spin up an S3 bucket, SNS Topic and 1/2 SQS Queues to be used as part of the event-driven concept publishing pipeline.

• upp-elasticsearch-provisioner

  • Docker image, running Ansible & CloudFormation to provision and decommission UPP ElasticSearch clusters.

• upp-factset-provisioner

  • Docker image, running Ansible & CloudFormation to provision and decommission UPP Factset RDS stacks.

• content-fluentd-kinesis

  • Docker image, running Ansible & CloudFormation to provision and decommission Content Fluentd Kinesis.

• upp-annotations-monitoring-provisioner

  • Docker image, running Ansible & CloudFormation to provision and decommission Annotations Monitoring Kinesis Analytics Application.

• upp-msk-provisioner

  • Docker image, running Ansible & CloudFormation to provision and decommission AWS Managed Kafka cluster (MSK).

• ccf-dynamodb

  • Docker image, running Ansible & CloudFormation to provision and decommission DynamoDB table used to store client settings for Copyright Cleared Feed.

upp-provisioners's Issues

Jenkins provisioner to use amazon ft-base

the content jenkins provisioner uses an ami which doesn't come with FT related things. Use amazon ftbase as base for the EC2 instance.

Elasticsearch snapshot fails

upp-provisioners/upp-elasticsearch-provisioner/ansible/files/register-es-snapshot-dir.py

Line 45 in 9ebfae4

path='/_snapshot/index-backups',

This should point towards the new s3 bucket upp-concepts-k8s-prod-eu-backup.

Otherwise, we get this:

{
    "error": {
        "root_cause": [
            {
                "type": "repository_missing_exception",
                "reason": "[upp-concepts-k8s-prod-eu-backup] missing"
            }
        ],
        "type": "repository_missing_exception",
        "reason": "[upp-concepts-k8s-prod-eu-backup] missing"
    },
    "status": 404
}

upp-concept-publishing-provisioner - creation of resources out of order causes stack roll back.

On one occasion I have experienced an Error when provisionning AWS stack with Docker image in upp-concept-publishing-provisioner project.
The error is as follows:
"Unable to validate the following destination configurations" when using S3 event notifications in CloudFormation.
The issue was to do with creating the s3 bucket, SNS Topic and Subscription in the wrong order, (possibly due to time it took CF to create). This caused the stack to roll back.

It is described here in detail.
https://aws.amazon.com/premiumsupport/knowledge-center/unable-validate-destination-s3/

No tags for buckets created by upp-elasticsearch-provisioner

Buckets created by the upp-elasticsearch-provisioner playbook does not contain the required tags.

Resource has no tags. Required tags are: [systemCode, teamDL, environment]

Here is a direct link to the ansible task that should be changed.