o3de/o3de-jenkins-pipeline

This project automates the deployment of the O3DE Jenkins Pipeline that runs the checks on all pull requests submitted to the Open 3D Engine (O3DE).

o3de-jenkins-pipeline's Issues

Setup mechanism to run automated tests on Jenkins

Setup testing mechanism that can be added to CodePipeline to run tests on Jenkins. These tests can be used to test the staging instance or prod instance to determine if a rollback is needed.

Creating the actual tests (running jobs, load testing) is out of scope.

• CDK is updated to add a stage in the deployment pipeline to run tests on the jenkins staging or prod instance
• Stage can report success/fail to determine next actions.

Create custom entrypoint for Jenkins container

Jenkins doesn't support moving the plugins directory to another location. This causes issues when using shared storage to host jenkins_home. Plugins that are removed from the plugins.txt file are not removed and need to be uninstalled manually.

Create a custom startup script to clear out the plugins directory when starting up Jenkins.

Create guide for performing local testing

It's possible to run the Jenkins docker container locally for testing. However, there are a few settings that need to be updated to allow a user to successfully start up a local instance that mimics their prod server.

• Add documentation that provides a set of instructions to easily deploy a local copy of the Jenkins pipeline server on a user's machine.

Update CDK for CodePipeline to deploy CDK stack

Using CodeDeploy for the deploy stage requires certain values to be hard-coded in the repo. Moving to CDK deploy avoids this and allows these values to be retrieved from the generated CDK stack.

Related info: https://aws.amazon.com/blogs/developer/cdk-pipelines-continuous-delivery-for-aws-cdk-applications/

Add arg to dockerfile for java memory settings

Right now we default to 8GB for our prod deployments. A user should be able to change this setting for their own deployments or testing.

Change the hard-coded java memory settings an arg in the dockerfile.

Automated Updates for Jenkins Pipeline

Develop a mechanism to automatically submit PRs to update Jenkins. While deployments are already automated through CDK Pipelines we still need to manually update the dockerfile and plugins.txt to keep components up-to-date.

• Research new or existing mechanisms that can be used to automatically submit a PR when new LTS versions of Jenkins is released. Including plugins will be nice, but not required.
• Submit RFC to sig-build/sig-security for approval.

Create a mechanism to populate required values in JCasC config files

When a user wants to use the configs in a template file, they typically need to replace the <value> entrees with the required parameters. Or they can store secret values in AWS Parameter Store.

Create a script to automatically populate these values and push secrets to parameter store if required.

Update README with instructions to bootstrap the new CDK pipeline setup

A bootstrap step will be required with the new CDK pipeline setup. Also --context values are provided instead of CFN parameters.

Update Jenkins to v2.361.1 and update plugins

Latest LTS version is v2.361.1

Update plugins to latest supported versions.

Setup automated rollback mechanism for Jenkins

Manual rollback mechanism is available through Fargate task revisions. An automated rollback would update the task to run the previous revision. ECS would then take care of replacing the task.

Prereq: Automated testing: #24

• An automated rollback is triggered if automated tests fail in prod after a deployment.
• CDK is updated to add pipeline stage to run this automated task.

Create CDK stack for ECS cluster

The current pipeline setup requires an empty ECS cluster to deploy into. At the moment this is manually created.

Create a CDK stack to replace this setup, so that the ECS cluster to host the Jenkins server is automatically created.

Define default o3de pipeline setup in job dsl

Add a config for an initial pipeline pointing to the o3de repo. That way users will not have to create a pipeline setup after the initial startup.

While the pipeline is defined in the Jenkinsfile, the pipeline job still needs to be manually created in Jenkins.

Setup JCasC to automate setup on startup

Add Jenkins Configuration as Code (JCasC) as an installed plugin and include the required configs to setup the Jenkins server on startup.

This will prevent having to manually configure jenkins on startup and minimize the time required have Jenkins up and running.

Create setup scripts for CDK Pipeline requirements

Certain dependencies need to be installed in order to perform the initial CDK deploy step for the AWS setup (e.g. node.js, cdk toolkit, etc.). Some of these steps can be automated.

• Create a script to install these requirements to minimize manual setup steps for the deployment.
• Update README with instructions to use the script