![Logo](doc/images/otp-logo.png)
One Touch Provisioning (OTP) is a pattern that enables the seamless end-to-end provisioning of Red Hat OpenShift clusters, their applications, governance and policies to Public, Private, On-Premises and both Near and Far Edge Clouds all via Code.
- About the Project
- Introduction to Concepts and Technologies leveraged
- Getting Started
- Usage
- Roadmap
- Contributing
- License
- Contact
- Acknowledgments
This method/pattern is our opinionated implementation of the GitOps principles, using the latest and greatest tooling available, to enable the hitting of one big red button (figuratively) to start provisioning a platform that provides Cluster and Virtual Machine Provisioning capabilities, Governance and policy management, observability of Clusters and workloads and finally deployment of applications, such as IBM Cloud Paks, all within a single command*.
The pattern leans very heavily on technologies such as Red Hat Advanced Cluster Management (RHACM) and OpenShift GitOps (ArgoCD), enabling a process upon which OpenShift clusters, their applications, governance and policies are defined in Git repositories and by leveraging RHACM as a function of OpenShift GitOps, enables the seamless end to end provisioning of those environments.
- Codified, Repeatable and Auditable.
Before Getting Started with this pattern, it's important to understand some of the concepts and technologies used. This will help reduce the barrier of entry when adopting the pattern and help understand why certain design decisions were made.
- Red Hat Advanced Cluster Management Hub and Spoke Clusters Concepts
- Git Repositories Context
- Use-cases for a different Git Repository folder organisation
- Minimum OpenShift v4.10+ is required.
Deploy a "vanilla" Red Hat OpenShift cluster using one of the methods below:
- Installer Provisioned Infrastructure
- User Provisioned Infrastructure
- Managed OpenShift Offerings
- OpenShift Assisted Installer
- Install the OpenShift
oc
CLI (version 4.10+). The binary can be downloaded from the Help menu from the OpenShift cluster console, or downloaded from the OpenShift Mirror website.
-
Install helm CLI from brew.sh
brew install helm
-
If you intend to use the SealedSecrets Operator, then it's recommended to install kubeseal CLI from brew.sh
brew install kubeseal
The pattern requires the use of six git repositories within the GitOps workflow, the reason for this is explained here.
-
- This repository contains all the ArgoCD Applications for the
infrastructure
,services
,policies
,clusters
andapplication
layers. Each ArgoCD Application will reference a specific resource that will be deployed to the RHACM Hub Cluster, or depending on your chosen configuration, it may include Spoke Cluster resources as well.
- This repository contains all the ArgoCD Applications for the
-
Infrastructure GitOps repository
- This repository is what we've termed a "common repository". This repository will be used across both the RHACM Hub cluster and any Spoke clusters you deploy. The repository contains the YAMLs for cluster-wide and/or infrastructure related resources managed by a cluster administrator. This would include
namespaces
,clusterroles
,clusterrolebindings
,machinesets
to name a few. By leveraging "common repositories", you can reduce depulication of code and ensure a consistant set of resources are applied each time.
- This repository is what we've termed a "common repository". This repository will be used across both the RHACM Hub cluster and any Spoke clusters you deploy. The repository contains the YAMLs for cluster-wide and/or infrastructure related resources managed by a cluster administrator. This would include
-
- This repository is what we've termed a "common repository". This repository will be used across both the RHACM Hub cluster and any Spoke clusters you deploy. The repository contains the YAMLs for resources which will be used by the RHACM Hub and Spoke clusters. This repository include
subscriptions
for Operators, YAMLs of custom resources provided, or Helm Charts for tools provided by a third party. These resource would usually be managed by the Administrator(s) and/or a DevOps team supporting application developers. By leveraging "common repositories", you can reduce depulication of code and ensure a consistant set of resources are applied each time.
- This repository is what we've termed a "common repository". This repository will be used across both the RHACM Hub cluster and any Spoke clusters you deploy. The repository contains the YAMLs for resources which will be used by the RHACM Hub and Spoke clusters. This repository include
-
- This repository is what we've termed a "common repository". The repository contains the YAMLs for resources to deploy
Policies
to both the RHACM Hub and Spoke clusters. These resource would usually be managed by the GRC and/or Security teams. By leveraging "common repositories", you can reduce depulication of code and ensure a consistant set of resources are applied each time.
- This repository is what we've termed a "common repository". The repository contains the YAMLs for resources to deploy
-
- This repository is what we've termed a "common repository". The repository contains the YAMLs for resources to deploy
OpenShift Clusters
. These resource would usually be managed by the Platform Administrator(s) and/or a Ops team supporting the Cloud Platforms. By leveraging "common repositories", you can reduce depulication of code and ensure a consistant set of resources are applied each time.
- This repository is what we've termed a "common repository". The repository contains the YAMLs for resources to deploy
-
- This repository is what we've termed a "common repository". The repository contains the YAMLs for resources to deploy the RHACM Hub or Spoke
OpenShift Clusters
. Contains the YAMLs for resources to deployapplications
. By leveraging "common repositories", you can reduce depulication of code and ensure a consistant set of resources are applied each time.
- This repository is what we've termed a "common repository". The repository contains the YAMLs for resources to deploy the RHACM Hub or Spoke
-
Create a new GitHub Organization using instructions from this GitHub documentation.
-
Create the repositories within your new GitHub Organization and clone them locally.
GIT_ORG=<new-git-organization> OUTPUT_DIR=<gitops-repos> ./scripts/create-repos.sh
-
(
Optional
) Many users may wish to use private Git repositories on GitHub to store their manifests, rather than leaving them publically readable. The steps for setting up OpenShift GitOps for Private repositories can be found here. -
Update the default Git URL and branch references in your
otp-gitops
repository by running the provided script./scripts/set-git-source.sh
script.GIT_ORG=<GIT_ORG> GIT_BRANCH=master ./scripts/set-git-source.sh
- You can unset the changes you made above by running the
./scripts/unset-git-source.sh
.
- You can unset the changes you made above by running the
If you intend to deploy the Infrastructure Automation
component of IBM Cloud Pak for Watson AIOps, then please follow the instructions here.
- Red Hat OpenShift GitOps uses Argo CD, an open-source declarative tool, to maintain and reconcile cluster resources.
-
Install the OpenShift GitOps Operator and create a
ClusterRole
andClusterRoleBinding
.cd setup oc apply -f setup while ! oc wait crd applications.argoproj.io --timeout=-1s --for=condition=Established 2>/dev/null; do sleep 30; done while ! oc wait pod --timeout=-1s --for=condition=Ready -l '!job-name' -n openshift-gitops > /dev/null; do sleep 30; done
-
Create a custom ArgoCD instance with custom checks. To customise which health checks, comment out those you don't need in
setup/argocd-instance/kustomization.yaml
.oc apply -k argocd-instance while ! oc wait pod --timeout=-1s --for=condition=ContainersReady -l app.kubernetes.io/name=openshift-gitops-otp-server -n openshift-gitops > /dev/null; do sleep 30; done
-
(
Optional
) If using IBM Cloud ROKS as a RHACM Hub Cluster, then you will need to configure TLS../scripts/patch-argocd-tls.sh
-
(
Optional
) Create a console link to OpenShift GitOpsexport ROUTE_NAME=openshift-gitops-otp-server export ROUTE_NAMESPACE=openshift-gitops export CONSOLE_LINK_URL="https://$(oc get route $ROUTE_NAME -o=jsonpath='{.spec.host}' -n $ROUTE_NAMESPACE)" envsubst < <(cat setup/4_consolelink.yaml.envsubst) | oc apply -f -
On AWS, Azure, GCP and vSphere run the following script to configure the machinesets, infra nodes and storage definitions for the Cloud
you are using for the RHACM Hub Cluster. This will deploy additional nodes to support OpenShift Data Foundation (ODF) for Persistant Storage, as well as additional nodes to support Infrastructure (aka infra) components, such as RHACM, Quay, Ingress Controllers, OpenShift Internal Registry and ACS.
This is a design choice to reduce OpenShift licensing requirements as running these components on Infrastructure nodes does not consume a subscription cost.
./scripts/infra-mod.sh
If you are running a managed OpenShift cluster on IBM Cloud, you can deploy OpenShift Data Foundation as an add-on. You will also need to label some of your worker nodes as Infra nodes, otherwise RHACM will fail to deploy.
Attach the following label to the worker nodes you intend to use as Infra nodes.
node-role.kubernetes.io/infra: ''
The bootstrap YAML follows the app of apps pattern.
-
Retrieve the ArgoCD/GitOps URL and admin password and log into the UI
oc get route -n openshift-gitops openshift-gitops-otp-server -o template --template='https://{{.spec.host}}' # Password is not required if using the OpenShift as an authorisation provider oc extract secrets/openshift-gitops-otp-cluster --keys=admin.password -n openshift-gitops --to=-
-
The resources required to be deployed for this pattern have been pre-selected. However, you can review and modify the resources deployed by editing the following.
0-bootstrap/hub/1-infra/kustomization.yaml 0-bootstrap/hub/2-services/kustomization.yaml 0-bootstrap/hub/3-policies/kustomization.yaml 0-bootstrap/hub/4-clusters/kustomization.yaml 0-bootstrap/hub/5-apps/kustomization.yaml
Any changes to the kustomization files before the Initial Bootstrap, will need to be committed back to your Git repository, otherwise they will not be picked up by OpenShift GitOps.
-
Deploy the OpenShift GitOps Bootstrap Application.
oc apply -f 0-bootstrap/hub/bootstrap.yaml
-
ArgoCD Sync waves are used to managed the order of manifest deployment, this is required as some objects have parent-child relationships and are expected to exist within the RHACM Hub before they can be successfully deployed. We have seen occassions where applying both the Infrastructure, Services and Policies layers at the same time can fail. This typically occurs when there are issues with provisioning of additional nodes to support Storage and Infrastructure components. YMMV.
Once the Infrastructure, Services and Policies layers have been deployed, update the 0-bootstrap/hub/kustomization.yaml
manifest to enable the Clusters and Apps layer and commit to Git. OpenShift GitOps will then automatically deploy any resources listed within those Kustomise files.
resources:
- 1-infra/1-infra.yaml
- 2-services/2-services.yaml
- 3-policies/3-policies.yaml
## Uncomment once the above layers have completed.
# - 4-clusters/4-clusters.yaml
# - 5-apps/5-apps.yaml
Installation is successful once all ArgoCD Applications are fully synced without errors.
You will be able to access the RHACM Hub console via the OpenShift console.
This pattern treats Managed (aka Spoke) Clusters as OpenShift GitOps Applications. This allows us to Create, Destroy, Hibernate and Import Managed Clusters into Red Hat Advanced Cluster Management via OpenShift GitOps.
-
ClusterPools and ClusterClaims
-
We've now simplied the life-cycling of OpenShift Clusters on AWS, Google Cloud and Azure via the use of Cluster Pools and ClusterClaims.
-
Cluster Pools allows you pre-set a common cluster configuration and RHACM will take that configuration and apply it to each Cluster it deploys from that Cluster Pool. An example could be that a Production Cluster may consume specific Compute resources, exist in a multi-zone configuration and requires a particular version of OpenShift to be deployed and RHACM will deploy a cluster to meet those requirements.
-
Once a Cluster Pool has been created, you can submit ClusterClaims to deploy a cluster from that pool.
-
-
ClusterDeployment
-
The ClusterDeployment method can be used to deploy AWS, Azure, GCP, VMWare, On-Premise, Edge and IBM Cloud OpenShift Clusters.
-
Review the
Clusters
layer kustomization.yaml to enable/disable the Clusters that will be deployed via OpenShift GitOps.resources: ## ClusterPools ## Example: - argocd/clusterpools/<env>/<cloud>/<clusterpoolname>/<clusterpoolname.yaml> - argocd/clusterpools/cicd/aws/aws-cicd-pool/aws-cicd-pool.yaml ## ClusterClaims ## Example : - argocd/clusterclaims/<env>/<cloud>/<clusterclaimname.yaml> - argocd/clusterclaims/dev/aws/project-simple.yaml ## ClusterDeployments ## Example : - argocd/<env>/<cloud>/<clustername>/<clustername.yaml> - argocd/clusters/prod/aws/aws-prod/aws-prod.yaml - argocd/clusters/prod/azure/azure-prod/azure-prod.yaml
-
We have have provided examples for deploying new clusters into AWS, Azure, IBM Cloud and VMWare. Cluster Deployments require the use of your Cloud Provider API Keys to allow RHACM to connect to your Cloud Provider and deploy via Terraform an OpenShift cluster. We make use of an external keystore, e.g. Vault and leveraged the use of the External Secrets Operator to pull in the Cloud Providers API keys automatically. This simplifies the creation of new clusters, reduces the values needed and works better with Scale. The deployments for the clusters is stored within the
Clusters
repository, underclusters/deploy/external-secrets/<cloud provider>
. -
Originally the pattern utilised the SealedSecrets Controller to encrypt your API Keys and provided a handy script for each Cloud Provider within the
Clusters
repository, underclusters/deploy/sealed-secrets/<cloud provider>
for your use. This was deemed an ok method for 1-5 cluster deployments, but became very cumbersome when dealing with Scale and was at risk of error and misconfiguration. We will no longer be iterating the code for cluster deployment via SealedSecret and we'll eventually remove this altogether.
-
The pattern provides full end to end deployment of not only Clusters, but also Policies, Governance and Applications.
For more usage examples, please refer to the Documentation
- OTP cli
- Ansible Automation integration with Libvirt and VMWare
- HyperShift Integration
- HyperShift with OpenShift Virtualisation for Worker nodes
- Deployment of IBM Cloud Satellite for IBM Managed OpenShift platform within chosen environment.
See the open issues for a full list of proposed features (and known issues).
Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.
If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature
) - Commit your Changes (
git commit -m 'Add some AmazingFeature'
) - Push to the Branch (
git push origin feature/AmazingFeature
) - Open a Pull Request
Distributed under the APACHE 2.0 License. See LICENSE
for more information.
- Ben Swinney | @bdgsts | Linkedin | Github
- Cong Nguyen | Linkedin | Github
- Nick Merrett | Linkedin | Github
- Marwan Attar | Linkedin
- Langley Millard | Linkedin
Project Link: https://github.com/one-touch-provisioning/otp-gitops
This asset has been built on the shoulders of giants and leverages the great work and efforts undertaken by the teams/individuals below. Without these efforts, then this pattern would have struggled to get off the ground.
The reference architecture for this GitOps workflow can be found here.