cci-moc/moc-apps

ArgoCD applications deployed on MOC managed OpenShift clusters

This repository manages ArgoCD applications deployed on MOC managed clusters.

The directory layout of this repository follows the standards of the Operate First project (specifically ADR-0009).

Validations to make your life easier

Pre-commit checks

Configure linters to run before each commit by install the pre-commit tool:

pip install pre-commit

And then enabling it for your working directory. From inside this repository, run:

pre-commit install

Pre-push checks

There are some checks we may not want to run for each commit, but we do want to run before pushing changes up to GitHub. First, make sure you have the openshift-schemas repository checked out:

git -C .. clone https://github.com/cci-moc/openshift-schemas

And make sure kubeval is somewhere in your path.

Then place the following in .git/hooks/pre-push in your local checkout of the moc-apps repository:

#!/bin/sh

./ci/validate-manifests.sh

Ensure the file is executable:

chmod 755 .git/hooks/pre-push

This will run kustomize build on all overlays and validate the output using kubeval prior to each push.

Deploying Secrets

Secrets for both ocp-staging and ocp-prod are stored in AWS Secret Manager and accessed on the cluster via ExternalSecret. AWS Credentials for the mocops user can be found in BitWarden.

• Login to AWS and navigate to Secret Manager.
• Select Store a New Secret and select Other type of secret.
• Input the value under the plain text tab. Keep the default encryption key.
• The name of the secret should be cluster/<cluster_name>/<namespace>/<secret_name>.
• Under tags, Add cluster as key and the name of the cluster as value. The secret will not be accessible otherwise.
• Create a .yaml manifest for the ExternalSecret. See example.

moc-apps's Issues

Integrate ColdFront-Staging with OCP Prod

Since we're not ready to ship OCP Prod yet, it's preferable to run the acceptance tests with a staging ColdFront deployed in nerc-shift-1 rather than touch the production ColdFront.

Do issuers belong in cluster-scope application?

In operate-first/apps#947, Tom brings up the
interesting questions about whether issues are really "cluster-scope"
resources, or if they should be treated as a namespaced application.

I don't have an immediate reaction, but I wanted to note it here so
that I don't forget to think about it in the context of our own
configuration.

Bad apiversion in #132

In #132, we create the curator-auth-proxy clusterrole as:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole

But that's the wrong apiVersion, and results in the error:

no matches for kind "ClusterRole" in version "rbac.authorization.k8s.io/v1beta1"

The correct apiVersion is rbac.authorization.k8s.io/v1

Clarify usage of components folder

The only instance that the components folder is used is here https://github.com/CCI-MOC/moc-apps/tree/master/cluster-scope/components/operators.coreos.com/subscriptions/grafana-operator

We haven't used it anywhere else. Either clarify or remove.

ceph-client-network-enp4s0f0np1 nncp failed to apply

One of the nncp resources has failed to apply:

$ oc get nncp
NAME                              STATUS
ceph-client-network-eno2          SuccessfullyConfigured
ceph-client-network-enp4s0f0np1   FailedToConfigure
ceph-client-network-enp4s0f1      SuccessfullyConfigured
ceph-client-network-ens3f0        SuccessfullyConfigured

We can get details on the problem by looking at nnce
(nodenetworkconfigurationenactments) resources:

$ oc get nnce | grep -i failed
neu-5-10-compute.ocp-prod.massopen.cloud.ceph-client-network-enp4s0f0np1   FailedToConfigure

$ oc get nnce neu-5-10-compute.ocp-prod.massopen.cloud.ceph-client-network-enp4s0f0np1 -o yaml | yq -y .status
conditions:
  - lastHearbeatTime: '2021-08-26T15:24:26Z'
    lastTransitionTime: '2021-08-26T15:24:26Z'
    message: "error reconciling NodeNetworkConfigurationPolicy at desired state apply:\
      \ , failed to execute nmstatectl set --no-commit --timeout 480: 'exit status\
      \ 1' '' '/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py:325: UserWarning:\
      \ Using 'set' is deprecated, use 'apply' instead.\n  warnings.warn(\"Using 'set'\
      \ is deprecated, use 'apply' instead.\")\n2021-08-26 15:24:26,388 root     \
      \    DEBUG    Nmstate version: 1.0.2\n2021-08-26 15:24:26,388 root         DEBUG\
      \    Applying desire state: {'interfaces': [{'description': 'ceph client network',\
      \ 'ipv4': {'dhcp': True, 'enabled': True}, 'name': 'enp4s0f0np1.250', 'state':\
      \ 'up', 'type': 'vlan', 'vlan': {'base-iface': 'enp4s0f0np1', 'id': 250}}]}\n\
      2021-08-26 15:24:26,448 root         DEBUG    NetworkManager version 1.30.0\n\
      2021-08-26 15:24:26,453 root         DEBUG    Async action: Retrieve applied\
      \ config: ethernet enp4s0f0np0 started\n2021-08-26 15:24:26,454 root       \
      \  DEBUG    Async action: Retrieve applied config: ethernet enp4s0f0np0 finished\n\
      Traceback (most recent call last):\n  File \"/usr/bin/nmstatectl\", line 11,\
      \ in <module>\n    load_entry_point('nmstate==1.0.2', 'console_scripts', 'nmstatectl')()\n\
      \  File \"/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py\", line\
      \ 73, in main\n    return args.func(args)\n  File \"/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py\"\
      , line 326, in set\n    return apply(args)\n  File \"/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py\"\
      , line 354, in apply\n    args.save_to_disk,\n  File \"/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py\"\
      , line 407, in apply_state\n    save_to_disk=save_to_disk,\n  File \"/usr/lib/python3.6/site-packages/libnmstate/netapplier.py\"\
      , line 78, in apply\n    desired_state, ignored_ifnames, current_state, save_to_disk\n\
      \  File \"/usr/lib/python3.6/site-packages/libnmstate/net_state.py\", line 51,\
      \ in __init__\n    gen_conf_mode,\n  File \"/usr/lib/python3.6/site-packages/libnmstate/ifaces/ifaces.py\"\
      , line 157, in __init__\n    self._validate_unknown_parent()\n  File \"/usr/lib/python3.6/site-packages/libnmstate/ifaces/ifaces.py\"\
      , line 765, in _validate_unknown_parent\n    f\"Interface {iface.name} has unknown\
      \ parent: \"\nlibnmstate.error.NmstateValueError: Interface enp4s0f0np1.250\
      \ has unknown parent: enp4s0f0np1\n'"
    reason: FailedToConfigure
    status: 'True'
    type: Failing
  - lastHearbeatTime: '2021-08-26T15:24:26Z'
    lastTransitionTime: '2021-08-26T15:24:26Z'
    reason: FailedToConfigure
    status: 'False'
    type: Available
  - lastHearbeatTime: '2021-08-26T15:24:26Z'
    lastTransitionTime: '2021-08-26T15:24:26Z'
    reason: FailedToConfigure
    status: 'False'
    type: Progressing
  - lastHearbeatTime: '2021-08-26T15:24:23Z'
    lastTransitionTime: '2021-08-26T15:24:23Z'
    message: All policy selectors are matching the node
    reason: AllSelectorsMatching
    status: 'True'
    type: Matching
  - lastHearbeatTime: '2021-08-26T15:24:26Z'
    lastTransitionTime: '2021-08-26T15:24:26Z'
    reason: SuccessfullyConfigured
    status: 'False'
    type: Aborted
desiredState:
  interfaces:
    - description: ceph client network
      ipv4:
        dhcp: true
        enabled: true
      name: enp4s0f0np1.250
      state: up
      type: vlan
      vlan:
        base-iface: enp4s0f0np1
        id: 250
policyGeneration: 1

Deploy Coldfront with OpenShift support for the MOC

We're currently using Coldfront on the NERC to manage OpenStack users. We want to adapt the same workflow for the MOC, but with managing OpenShift instead of OpenStack.

This task refers to managing the MOC managed OpenShift and is unrelate to the NERC OpenShift effort. If we are going to deprecate the MOC maintained OpenShift in favor of NERC, this task should be closed.

• Adapt coldfront-nerc manifests and image for MOC deployment.
• Deploy via Argo CD
• Test, document and demo