cloud-native-toolkit/multi-tenancy-gitops

Provides our opinionated point of view on how GitOps can be used to manage the infrastructure, services and application layers of K8s based systems

Cloud Native Toolkit - GitOps Production Deployment Guide

The GitOps concept originated from Weaveworks back in 2017 and the goal was to automate the operations of a Kubernetes (K8s) system using a model external to the system as the source of truth (History of GitOps).

This repository provides our opinionated point of view on how GitOps can be used to manage the infrastructure, services and application layers of K8s based systems. It takes into account the various personas interacting with the system and accounts for separation of duties. The instructions and examples are focused around the Red Hat OpenShift platform and IBM Cloud Paks.

The reference architecture for this GitOps workflow can be found here.

Table of contents

• Cloud Native Toolkit - GitOps Production Deployment Guide
  • Table of contents
  • Pre-requisites
    • Red Hat OpenShift cluster
    • CLI tools
    • IBM Entitlement Key
  • Setup git repositories
    • Tasks:
  • Install and configure OpenShift GitOps
    • Tasks:
  • Bootstrap the OpenShift cluster
    • Tasks:
  • Select resources to deploy
    • Tasks:

Pre-requisites

Red Hat OpenShift cluster

• An OpenShift v4.7+ cluster is required.

CLI tools

• Install the git CLI.

  • Configure your username for your Git commits - link.
  • Configure your email for your Git commits - link.

• Install the OpenShift CLI oc (version 4.7+) . The binary can be downloaded from the Help menu from the OpenShift Console.

  Download oc cli

  

• Log in from a terminal window.

  oc login --token=<token> --server=<server>

IBM Entitlement Key

• The IBM Entitlement Key is required to pull IBM Cloud Pak specific container images from the IBM Entitled Registry. To get an entitlement key,

  1. Log in to MyIBM Container Software Library with an IBMid and password associated with the entitled software.
  2. Select the View library option to verify your entitlement(s).
  3. Select the Get entitlement key to retrieve the key.

• A Secret containing the entitlement key is created in the tools namespace.

  oc new-project tools || true
  oc create secret docker-registry ibm-entitlement-key -n tools \
  --docker-username=cp \
  --docker-password="<entitlement_key>" \
  --docker-server=cp.icr.io

Setup git repositories

• The following set of Git repositories will be used for our GitOps workflow.
  • Main GitOps repository (https://github.com/cloud-native-toolkit/multi-tenancy-gitops): This repository contains all the ArgoCD Applications for the infrastructure, services and application layers. Each ArgoCD Application will reference a specific K8s resource (yaml resides in a separate git repository), contain the configuration of the K8s resource, and determine where it will be deployed into the cluster.
  • Infrastructure GitOps repository (https://github.com/cloud-native-toolkit/multi-tenancy-gitops-infra): Contains the YAMLs for cluster-wide and/or infrastructure related K8s resources managed by a cluster administrator. This would include namespaces, clusterroles, clusterrolebindings, machinesets to name a few.
  • Services GitOps repository (https://github.com/cloud-native-toolkit/multi-tenancy-gitops-services): Contains the YAMLs for K8s resources which will be used by the application layer. This could include subscriptions for Operators, YAMLs of custom resources provided, or Helm Charts for tools provided by a third party. These resource would usually be managed by the Administrator(s) and/or a DevOps team supporting application developers.

Tasks:

1. Create a new GitHub Organization using instructions from this GitHub documentation.
2. From each template repository, click the Use this template button and create a copy of the repository in your new GitHub Organization.
3. Clone the repositories locally.

   mkdir -p gitops-repos
   cd gitops-repos
   # Clone using SSH
   git clone [email protected]:<GIT_ORG>/multi-tenancy-gitops.git
   git clone [email protected]:<GIT_ORG>/multi-tenancy-gitops-infra.git
   git clone [email protected]:<GIT_ORG>/multi-tenancy-gitops-services.git

4. Update the default Git URl and branch references in your multi-tenancy-gitops repository by running the provided script ./scripts/set-git-source.sh script.

   cd multi-tenancy-gitops
   GIT_ORG=<GIT_ORG> GIT_BRANCH=master ./scripts/set-git-source.sh
   git commit -m "Update Git URl and branch references"
   git push origin master

Install and configure OpenShift GitOps

• Red Hat OpenShift GitOps uses Argo CD, an open-source declarative tool, to maintain and reconcile cluster resources.

Tasks:

1. Install the OpenShift GitOps Operator, create a ClusterRole and deploy a default instance of ArgoCD.

   oc apply -f setup/ocp47/
   while ! oc wait crd applications.argoproj.io --timeout=-1s --for=condition=Established  2>/dev/null; do sleep 30; done
   while ! oc wait pod --timeout=-1s --for=condition=Ready -l '!job-name' -n openshift-gitops > /dev/null; do sleep 30; done

2. Delete the default ArgoCD instance

   oc delete gitopsservice cluster -n openshift-gitops || true
   oc delete argocd openshift-gitops -n openshift-gitops || true

3. Create a custom ArgoCD instance with custom checks

   oc apply -f setup/ocp47/argocd-instance/ -n openshift-gitops
   while ! oc wait pod --timeout=-1s --for=condition=ContainersReady -l app.kubernetes.io/name=openshift-gitops-cntk-server -n openshift-gitops > /dev/null; do sleep 30; done

Bootstrap the OpenShift cluster

• The bootstrap YAML follows the app of apps pattern.

Tasks:

1. Select a profile and delete the others from the 0-bootstrap directory. If this is your first usage of the gitops workflow, use the single-cluster profile and deploy the ArgoCD Bootstrap Application.

   GITOPS_PROFILE="0-bootstrap/single-cluster"
   oc apply -f ${GITOPS_PROFILE}/bootstrap.yaml

2. Retrieve the ArgoCD/GitOps URL and admin password:

   oc get route -n openshift-gitops openshift-gitops-cntk-server -o template --template='https://{{.spec.host}}'
   oc extract secrets/openshift-gitops-cntk-cluster --keys=admin.password -n openshift-gitops --to=-

Select resources to deploy

• Clone the multi-tenancy-gitops repository in your Git Organization if you have not already done so and select the K8s resources to deploy in the infrastructure and services layers.
• Existing recipes are available and additional ones will be made available in the doc directory.
  • Cloud Native Toolkit
  • App Connect Enterprise recipe
  • MQ recipe
  • API Connect recipe
  • Process Mining recipe
  • Cloud Pak for Business Automation + Business Automation Workflow
  • Cloud Pak for Data
  • Cloud Pak for Data + Watson Studio
  • Cloud Pak for Data + Watson Knowledge Catalog
  • Cloud Pak for Data + DataStage
  • Cloud Pak for Data + Analytics Engine Powered by Apache Spark
  • Cloud Pak for Data + Watson OpenScale
  • Cloud Pak for Data + Data Virtualization
  • Cloud Pak for Data + Watson Machine Learning
  • Cloud Pak for Security
  • Instana Agent
  • Spectrum Protect Plus
  • Sterling File Gateway

Tasks:

1. Select a profile and delete the others from the 0-bootstrap directory. If this is your first usage of the gitops workflow, Use the single-cluster profile.

   GITOPS_PROFILE="0-bootstrap/single-cluster"

2. Review the Infrastructure layer kustomization.yaml and un-comment the resources to deploy.
3. Review the Services layer kustomization.yaml and un-comment the resources to deploy.
4. Commit and push changes to your git repository

   git add .
   git commit -m "initial bootstrap setup"
   git push origin

5. Validate the recipe was deployed correctly following the Validation section in the recipe.

multi-tenancy-gitops's Issues

Update `bootstrap.sh` script in cloud-native-toolkit-demos/multi-tenancy-gitops-mq to support Security & HA flags

We need to update https://github.com/cloud-native-toolkit-demos/multi-tenancy-gitops-mq/blob/ocp47-2021-2/scripts/bootstrap.sh to support flags for enabling Security and HA components respectively.

This is dependent upon #105 being completed first.

Currently, the only "activations" required are for Security, but HA is dependent on Security being enabled.

The resources required to be activated for this include but may not limited to the following:

• multi-tenancy-gitops-services/operators/cert-manager/operator.yaml installing the correct Marketplace version, and not the Community version
• multi-tenancy-gitops-mq/blob/ocp47-2021-2/0-bootstrap/argocd/single-cluster/2-services/kustomization.yaml
  • argocd/operators/cert-manager.yaml
  • argocd/instances/cert-manager-instance.yaml
• multi-tenancy-gitops-mq/blob/ocp47-2021-2/0-bootstrap/argocd/single-cluster/1-infra/kustomization.yaml
  • argocd/namespace-openldap.yaml
• Items from #105 in their respective namespaces.

Update ConsoleNotification message to be generic by default

Configure job to configure up ConsoleLinks for access to the required Routes

ConsoleLinks for CNTK tools, Cloud Paks, etc

boostrap with examples hangs waiting for kubeseal to be ready

Running the bootstrap script like this

curl -sfL https://raw.githubusercontent.com/cloud-native-toolkit-demos/multi-tenancy-gitops-ace/master/scripts/bootstrap.sh | \
GIT_USER=$REPLACE_WITH_GIT_USER \
GIT_ORG=$REPLACE_WITH_GIT_ORG \
GIT_TOKEN=$REPLACE_WITH_GIT_TOKEN \
OUTPUT_DIR=ace-production \
sh

Hangs at the point running ace_apps_bootstrap
The script https://github.com/cloud-native-toolkit-demos/multi-tenancy-gitops-apps/blob/master/scripts/ace-bootstrap.sh#L33
waits for kubeseal operator to be ready, but this doesn't happened because we have not called deploy_bootstrap_argocd

hangs like this

Bootstrap Cloud Pak examples for ACE
Github user/org is csantanapr-test-gitops-2
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found

update boostrap.sh to cover ibm roks classic storage class

if ocs, or nfs not found then check if ibm-file-gold-gid

If the default one is not found print a warning that the storage class not found

Clarify that you can use your own provisioned cluster in create cluster section

In create cluster topic, make it clear you can use your own cluster too

Tekton task for mq pr and smoke test not working with ArgoCD

MQ - Enhance Quickstart to smoothly flow in with the Qmgr and App sections

If folks want to use MQ Quickstart to configure their cluster and then try the apps section to build the qmgr or app, we do not have proper instruction in the deployment guides. In order to use this combo, we need to add in the required scripts and describe the flow.

Goal was to control everything through the bootstrap.sh and this should be addressed.

[bug] secret yaml names mismatch

secret yaml names mismatch -

1. Gitops expects ibm-entitled-registry-credentials-secret.yaml

multi-tenancy-gitops-apps/apic/environments/single-cluster/config/kustomization.yaml:

- secrets/ibm-entitled-registry-credentials-secret.yaml

2. The scripts and instructions create the secret file in a different name as shown below ibm-entitlement-key-secret.yaml

 grep ibm-entitlement-key-secret.yaml * -r
 
multi-tenancy-gitops-apps/apic/environments/single-cluster/config/secrets/ibm-entitlement-key-secret.sh:kubeseal -n tools --controller-name=${SEALED_SECRET_CONTOLLER_NAME} --controller-namespace=${SEALED_SECRET_NAMESPACE} -o yaml < delete-ibm-entitlement-key-secret.yaml > ibm-entitlement-key-secret.yaml
multi-tenancy-gitops-apps/apic/environments/single-cluster/config/secrets/ibm-entitlement-key-secret.sh:rm delete-ibm-entitlement-key-secret.yaml
multi-tenancy-gitops-apps/ace/environments/ci/secrets/ibm-entitlement-key-secret.sh:    -o yaml > ibm-entitlement-key-secret.yaml
multi-tenancy-gitops-apps/ace/environments/base/secrets/ibm-entitlement-key-secret.sh:    -o yaml > ibm-entitlement-key-secret.yaml

As a result, ArgoCD application apic-config-prod won't be created properly, instead, showing an error indicating the file ibm-entitled-registry-credentials-secret.yaml is not found.

Install OpenShift Pipelines operator using GitOps and remove from bootstrap.sh

dashboard icons route not working with ArgoCD

Add Watson Studio and CP4S recipes

Update the configuration of the ArgoCD instance

• Enable OpenShift SSO by default
• Add custom checks for CP instances for current recipes
• Configure ArgoCD instance with tls secret if a default certificate is used

add default Cloud Native Toolkit License

[bootstrap.sh] add ability to pass env RWX_STORAGE_CLASS for platform navigator argocd yaml

Revisit capabilities scope (namespace vs cluster)

Creating this issue to revisit the different capabilities the Cloud Pak Production Deployment Guides allow a user to install for collisions on their scopes. That is, we often encounter people with issues because they install a capability cluster-wide while another is namespace-wide making certain operators to get installed in multiple namespaces and failing as a result.

This usually gets fixed by installing all capabilities cluster-wide but there might be some capabilities that must be installed namespace wide such as Process Mining.

Let's review the scopes of these and come up with a strategy that works for all.

Missing kustomize patch for multi-cluster profiles

The multi-cluster profiles within the others folder are missing a patch in their kustomize files to have the 1-infra.yaml, 2-services.yaml, 3-apps.yaml ArgoCD GitOps applications (gitops.tier.layer: gitops) patched with the appropriate ${GITOPS_PROFILE} for the path.

As an example, check out https://github.com/cloud-native-toolkit/multi-tenancy-gitops/blob/master/0-bootstrap/others/3-multi-cluster/cluster-1-cicd/2-services/2-services.yaml#L96

As of now, the path is hardcoded to single-cluster

InterOperatorGroupOwnerConflict

While deploying IBM CP4D following the curated scripts getting an error - "intersecting operatorgroups provide the same apis". This is happening for IBM Cloud Pak foundational services under the openshift-operators namespace. Provider API is commonservice.

ibm-common-service-operator.v3.11.0

update experimental boostrap to allow to use other git servers like gitlab

Using the gh CLI would not work with nothing else than github.com not even github enterprise as the template repos to be fork are in github.com

Typo in "10-gitops-pr-without-helm-release.yaml" Task `workingDir` parameter

https://github.com/cloud-native-toolkit/multi-tenancy-gitops/blob/demo-mq-v2/2-services/instances/namespace-ci/tasks/10-gitops-pr-without-helm-release.yaml#L44 currently has an improperly-cased parameter that was preventing ArgoCD from syncing correctly.

What is there:
workingdir: $(params.source-dir)

What is required:
workingDir: $(params.source-dir)

setup/ocp47 directory should be renamed to not be version specific

This will also affect the documentation. I have opened a companion issue to address this in the documentation.
https://github.ibm.com/cloudpakbringup/production-deployment-guides/issues/240

Update delete_default_argocd_instance function in bootstrap.sh

The current delete_default_argocd_instance function will delete the instance of gitopsservice and argocd CR. This is because the deletion of the gitopsservice instance did not delete the instance of argocd.

delete_default_argocd_instance () {
echo "Delete the default ArgoCD instance"
pushd ${OUTPUT_DIR}
oc delete gitopsservice cluster -n openshift-gitops
oc delete argocd openshift-gitops -n openshift-gitops
popd
}

This has been fixed by the Red Hat team now so the deletion of the gitopsservice will also delete the default instance of ArgoCD. The second delete command can be removed.

set-git-source.sh incorrectly assumes `master` branch as starting point

The set-git-source.sh script incorrectly assumes that it will always be replacing master as the git base branch.

This is problematic if you create a new branch (newBranch1), run the script, push your changes, and deploy. Then you want to create a new environment branch based off of your current branch (newBranch2), run the script again, push your changes, and deploy. But all your files are still pointing to newBranch1 as the targetRevision in the ArgoCD Application YAMLs.

Reference: https://github.com/cloud-native-toolkit/multi-tenancy-gitops/blob/master/scripts/set-git-source.sh#L27

give argocd service account role to create icp4aclusters

Add cp4d operator and instance

Permission issue in bootstrap.sh when running set-git-source.sh from newly created gitops-0-bootstrap folder

When trying to run bootstrap.sh in a docker container (image registry.access.redhat.com/ubi8/python-39) I receive a "Permission denied" error when it reaches the set_git_source function and tries to run GIT_ORG=${GIT_ORG} ./scripts/set-git-source.sh.

I looked into it, and it appears that when the gitops-0-bootstrap repo is cloned in, the .sh scripts aren't executable by default. So when bootstrap.sh in my initial folder creates the new gitops-0-bootstrap folder and then tries to run the set-git-source.sh file that is inside of it, it fails because the newly created file is not executable.

I was wondering if anyone has run into a similar problem, and while I can shim in a fix in the set_git_source function, I am trying to use this repo as an imported submodule and not make any changes.
logs.pdf

Add Instana robot-shop

Add delay for OpenShift GitOps deployment in bootstrap.sh

Add argocd/operators/ibm-automation-foundation-operator.yaml to services in MQ recipe

From my experience trying to run the Quickstart and the MQ Tutorial, I found that I also needed to uncomment the following entry in the 0-bootstrap\single-cluster\2-services\kustomization.yaml:

- argocd/operators/ibm-automation-foundation-operator.yaml 

You might want to adjust the MQ recipe accordingly.

Enablement of MQ enhanced monitoring

Add resources to deploy MQ metrics application.

Update default IBM MQ operator channel to v1.6

There is an issue with using the v9.2.2.0-r1 QM image with the Channel = v1.5 for the IBM MQ operator. Per https://www.ibm.com/docs/en/ibm-mq/9.2?topic=operator-version-support-mq, v9.2.3 is supported for Channel = v1.6 so will need to update the IBM MQ operator CHannel to v1.6 to align with the Queue Manager repository PR below.

cloud-native-toolkit/mq-infra#1

Parameterize CP4D Platform instance

Create helm chart for CP4D Platform instance YAMLs in https://github.com/cloud-native-toolkit/multi-tenancy-gitops-services/tree/master/instances/ibm-cpd-instance.

Fix argocd app for CP4S Operator

Currently, there are 4 labels:

  labels:
    gitops.tier.group: ibm-cloudpak
    gitops.tier.layer: services
    gitops.tier.source: helm
    gitops.tier.source: git

The label gitops.tier.source: git needs to be removed

Updates to MQ gitops promotion pipelines

Based on the call today, @hollisc and I followed up with discussion on connecting the dots from the conversation to the explicit artifacts that need to be updated to remove the GitOps pipelines pushing back into the Application Source repositories.

The changes in this issue should be made off the demo-mq-v2 branch.

There are multiple waves of changes that can happen to get to an optimal setup, but the below target state diagram is the first wave that we need to remove the anti-pattern that we have implemented currently:

NOTES:

• These changes will currently make the mq-infra-staging and mq-infra-prod pipelines obsolete. We can revisit the separation of concerns with respect to testing when & where in a subsequent wave.
• These changes remove the use of staging or prod in the upstream application source repository.

Artifacts to update:

• https://github.com/cloud-native-toolkit/multi-tenancy-gitops/blob/demo-mq-v2/2-services/instances/namespace-dev/pipelines/ibm-test-pipeline-for-dev.yaml
• https://github.com/cloud-native-toolkit/multi-tenancy-gitops/blob/demo-mq-v2/2-services/instances/namespace-staging/pipelines/ibm-test-pipeline-for-staging.yaml

Add ibm-foundation operator

[bug] soapserver not installing when using ace bom

add cp4a operator

Incorrect directory listed in namespace-openshift-storage.yaml

When deploying on IBM Cloud I noticed that this dir is not correct

multi-tenancy-gitops/1-infra/argocd/active/namespace-openshift-storage.yaml

Line 13 in fe9f718

path: 1-infra/namespaces/openshift-storage

It refers to a directory that doesn't exist, and this shows as errors in ArgoCD.

I'm getting some other errors in ArgoCD which I'm working through.

Add recipe for cp4s

dev-mq-infra-instance.yaml requires 'SyncOptions: Replace'

Based on some of the CR fields that will change in the QueueManager's deployment YAML, not all of them are allowed to be changed on a running instance. For example, you cannot switch from a deployed SingleInstance Qmgr to a NativeHA Qmgr by updating/applying/patching the YAML.

By updating the ArgoCD Application to include the following options, it allows ArgoCD to replace the object outright and create it anew in the dev namespace:

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - Replace=true

We don't need this everywhere, but it allows for cleaner deployment in the dev-related pipelines with the ability to manually navigate some of the setup for the staging and prod namespaces that would have less likelihood to switch some of those forbidden properties.

Add recipes to readme

Add recipe for Sterling File Gateway

Bootstrap script cloning wrong repos

Bootstrap script line 93 and 107 is cloning wrong repos

multi-tenancy-gitops/scripts/bootstrap.sh

Line 93 in 2c1dba8

gh repo clone ${GIT_ORG}/multi-tenancy-gitops-apps gitops-1-infra

multi-tenancy-gitops/scripts/bootstrap.sh

Line 107 in 2c1dba8

gh repo clone ${GIT_ORG}/multi-tenancy-gitops-apps gitops-2-services

problem with event-streams operator

#- argocd/operators/ibm-eventstreams-operator.yaml

OUTPUT_DIR=. ./create-repos.sh.

it creates multi-tenancy-gitops repo only
for Mac:

brew install wget
brew install gh

Client Version: 4.7.13
Creating GitHub repositories and local clones in folder: .
Github user/org is mq-apic-demo
gh: Not Found (HTTP 404)
Repository multi-tenancy-gitops not found, creating from template and cloning
✓ Created repository mq-apic-demo/multi-tenancy-gitops on GitHub
Initialized empty Git repository in /Users/markalfy/git/mq-apic-demo-root/multi-tenancy-gitops/.git/
fatal: couldn't find remote ref refs/heads/master
exit status 128

Update CP4S recipe to 1.9

Update Cloud Pak for Security recipe to version 1.9

Updates to boostrap.sh

• Remove other profiles if single-cluster profile is used to reduce the amount of files needed to be modified by set-git-source.sh script
• Add logic if git token is provided

Fix typo with Watson Studio instance

Typo found in the Watson Studio instance ArgoCD application -> https://github.com/cloud-native-toolkit/multi-tenancy-gitops/blob/master/0-bootstrap/single-cluster/2-services/argocd/instances/ibm-cp4d-watson-studio-instance.yaml#L36

[readme] remove GIT_USER no need it when not doing CP_EXAMPLES apps layer

Security artifacts missing in new kustomize-formatted repositories for MQ workstreams

In the new kustomize-formatted GitOps repositories, the security artifacts necessary to deploy MQ with security enabled are currently missing. Below is a snapshot of most of what is needed (the screencap is missing prod artifacts but those are required to be ported as well).

The corresponding source YAMLs need to be ported over to the proper place in multi-tenancy-gitops-apps/tree/master/mq/environments/{dev,staging,prod}/certificates from /multi-tenancy-gitops/tree/demo-mq-v2/2-services/instances/namespace-{dev,staging,prod}/secrets for the certificates, as well as the ClusterIssuer and Certificate from /multi-tenancy-gitops/tree/demo-mq-v2/2-services/instances/namespace-tools.

References:

• Source: https://github.com/cloud-native-toolkit/multi-tenancy-gitops/tree/demo-mq-v2
• Target: https://github.com/cloud-native-toolkit-demos/multi-tenancy-gitops-apps