o3de/o3de-jenkins-pipeline

This project automates the deployment of the O3DE Jenkins Pipeline that runs the checks on all pull requests submitted to the Open 3D Engine (O3DE).

O3DE (Open 3D Engine)

O3DE (Open 3D Engine) is an open-source, real-time, multi-platform 3D engine that enables developers and content creators to build AAA games, cinema-quality 3D worlds, and high-fidelity simulations without any fees or commercial obligations.

Contribute

For information about contributing to Open 3D Engine, visit https://o3de.org/docs/contributing/.

Roadmap

For information about upcoming work and features, please visit https://o3de.org/roadmap. Progress against the roadmap is tracked here.

Download and Install

This repository uses Git LFS for storing large binary files.

Verify you have Git LFS installed by running the following command to print the version number.

git lfs --version 

If Git LFS is not installed, download and run the installer from: https://git-lfs.github.com/.

Install Git LFS hooks

git lfs install

Clone the repository

git clone https://github.com/o3de/o3de.git

Building the Engine

Build requirements and redistributables

For the latest details and system requirements, refer to System Requirements in the documentation.

Windows

• Visual Studio 2019 16.9.2 minimum (All editions supported, including Community): https://visualstudio.microsoft.com/downloads/
  • Check System Requirements for other supported versions.
  • Install the following workloads:
    • Game Development with C++
    • MSVC v142 - VS 2019 C++ x64/x86
    • C++ 2019 redistributable update
• CMake 3.22.0 minimum: https://cmake.org/download/#latest (Release Candidate versions are not supported)

Optional

• Wwise audio SDK
  • For the latest version requirements and setup instructions, refer to the Wwise Audio Engine Gem reference in the documentation.

Quick start engine setup

To set up a project-centric source engine, complete the following steps. For other build options, refer to Setting up O3DE from GitHub in the documentation.

1. Create a writable folder to cache downloadable third-party packages. You can also use this to store other redistributable SDKs.

2. Install the following redistributables:

   • Visual Studio and VC++ redistributable can be installed to any location.
   • CMake can be installed to any location, as long as it's available in the system path.

3. Configure the engine source into a solution using this command line, replacing <your build path>, <your source path>, and <3rdParty package path> with the paths you've created:

   cmake -B <your build path> -S <your source path> -G "Visual Studio 16" -DLY_3RDPARTY_PATH=<3rdParty package path>
   

   Example:

   cmake -B C:\o3de\build\windows -S C:\o3de -G "Visual Studio 16" -DLY_3RDPARTY_PATH=C:\o3de-packages
   

   Note: Do not use trailing slashes for the <3rdParty package path>.

4. Alternatively, you can do this through the CMake GUI:

   1. Start cmake-gui.exe.
   2. Select the local path of the repo under "Where is the source code".
   3. Select a path where to build binaries under "Where to build the binaries".
   4. Click Add Entry and add a cache entry for the <3rdParty package path> folder you created, using the following values:
      1. Name: LY_3RDPARTY_PATH
      2. Type: STRING
      3. Value: <3rdParty package path>
   5. Click Configure.
   6. Wait for the key values to populate. Update or add any additional fields that are needed for your project.
   7. Click Generate.

5. Register the engine with this command:

   scripts\o3de.bat register --this-engine
   

6. The configuration of the solution is complete. You are now ready to create a project and build the engine.

For more details on the steps above, refer to Setting up O3DE from GitHub in the documentation.

Setting up new projects and building the engine

1. From the O3DE repo folder, set up a new project using the o3de create-project command.

   scripts\o3de.bat create-project --project-path <your new project path>
   

2. Configure a solution for your project.

   cmake -B <your project build path> -S <your new project source path> -G "Visual Studio 16"
   

   Example:

   cmake -B C:\my-project\build\windows -S C:\my-project -G "Visual Studio 16"
   

   Note: Do not use trailing slashes for the <3rdParty cache path>.

3. Build the project, Asset Processor, and Editor to binaries by running this command inside your project:

   cmake --build <your project build path> --target <New Project Name>.GameLauncher Editor --config profile -- /m
   

   Note: Your project name used in the build target is the same as the directory name of your project.

This will compile after some time and binaries will be available in the project build path you've specified, under bin/profile.

For a complete tutorial on project configuration, see Creating Projects Using the Command Line Interface in the documentation.

Code Contributors

This project exists thanks to all the people who contribute. [Contribute].

License

For terms please see the LICENSE*.TXT files at the root of this distribution.

o3de-jenkins-pipeline's Issues

Setup automated rollback mechanism for Jenkins

Manual rollback mechanism is available through Fargate task revisions. An automated rollback would update the task to run the previous revision. ECS would then take care of replacing the task.

Prereq: Automated testing: #24

• An automated rollback is triggered if automated tests fail in prod after a deployment.
• CDK is updated to add pipeline stage to run this automated task.

Create guide for performing local testing

It's possible to run the Jenkins docker container locally for testing. However, there are a few settings that need to be updated to allow a user to successfully start up a local instance that mimics their prod server.

• Add documentation that provides a set of instructions to easily deploy a local copy of the Jenkins pipeline server on a user's machine.

Update README with instructions to bootstrap the new CDK pipeline setup

A bootstrap step will be required with the new CDK pipeline setup. Also --context values are provided instead of CFN parameters.

Update Jenkins to v2.361.1 and update plugins

Latest LTS version is v2.361.1

Update plugins to latest supported versions.

Create setup scripts for CDK Pipeline requirements

Certain dependencies need to be installed in order to perform the initial CDK deploy step for the AWS setup (e.g. node.js, cdk toolkit, etc.). Some of these steps can be automated.

• Create a script to install these requirements to minimize manual setup steps for the deployment.
• Update README with instructions to use the script

Setup JCasC to automate setup on startup

Add Jenkins Configuration as Code (JCasC) as an installed plugin and include the required configs to setup the Jenkins server on startup.

This will prevent having to manually configure jenkins on startup and minimize the time required have Jenkins up and running.

Update CDK for CodePipeline to deploy CDK stack

Using CodeDeploy for the deploy stage requires certain values to be hard-coded in the repo. Moving to CDK deploy avoids this and allows these values to be retrieved from the generated CDK stack.

Related info: https://aws.amazon.com/blogs/developer/cdk-pipelines-continuous-delivery-for-aws-cdk-applications/

Create custom entrypoint for Jenkins container

Jenkins doesn't support moving the plugins directory to another location. This causes issues when using shared storage to host jenkins_home. Plugins that are removed from the plugins.txt file are not removed and need to be uninstalled manually.

Create a custom startup script to clear out the plugins directory when starting up Jenkins.

Create CDK stack for ECS cluster

The current pipeline setup requires an empty ECS cluster to deploy into. At the moment this is manually created.

Create a CDK stack to replace this setup, so that the ECS cluster to host the Jenkins server is automatically created.

Setup mechanism to run automated tests on Jenkins

Setup testing mechanism that can be added to CodePipeline to run tests on Jenkins. These tests can be used to test the staging instance or prod instance to determine if a rollback is needed.

Creating the actual tests (running jobs, load testing) is out of scope.

• CDK is updated to add a stage in the deployment pipeline to run tests on the jenkins staging or prod instance
• Stage can report success/fail to determine next actions.

Automated Updates for Jenkins Pipeline

Develop a mechanism to automatically submit PRs to update Jenkins. While deployments are already automated through CDK Pipelines we still need to manually update the dockerfile and plugins.txt to keep components up-to-date.

• Research new or existing mechanisms that can be used to automatically submit a PR when new LTS versions of Jenkins is released. Including plugins will be nice, but not required.
• Submit RFC to sig-build/sig-security for approval.

Define default o3de pipeline setup in job dsl

Add a config for an initial pipeline pointing to the o3de repo. That way users will not have to create a pipeline setup after the initial startup.

While the pipeline is defined in the Jenkinsfile, the pipeline job still needs to be manually created in Jenkins.

Add arg to dockerfile for java memory settings

Right now we default to 8GB for our prod deployments. A user should be able to change this setting for their own deployments or testing.

Change the hard-coded java memory settings an arg in the dockerfile.

Create a mechanism to populate required values in JCasC config files

When a user wants to use the configs in a template file, they typically need to replace the <value> entrees with the required parameters. Or they can store secret values in AWS Parameter Store.

Create a script to automatically populate these values and push secrets to parameter store if required.