quay / quay-operator Goto Github PK

View Code? Open in Web Editor NEW

119.0 24.0 78.0 40.67 MB

Operator for deploying and managing Quay

License: Apache License 2.0

Go 93.75% Dockerfile 0.25% Makefile 1.11% Shell 4.89%

quay-operator's Introduction

Quay Operator

Opinionated deployment of Quay container registry on Kubernetes.

Welcome

The original version of the quay-operator is available on the v1 branch. The next generation operator, known as TNG or v2, is developed on master branch.

Principles

Kubernetes is a powerful platform with an abundance of existing and future features. Always prefer to leverage something Kubernetes does better than try to implement it again.
Favor declarative application management to improve everyone's sanity and understanding of the state.
Make things simple by default, but always allow diving deeper to discover the details of what is going on.

Getting Started

This Operator can be installed on any Kubernetes cluster running the Operator Lifecycle Manager. Simply create the provided CatalogSource to make the package available on the cluster, then create the Subscription to install it.

You can find the latest operator release on operatorhub.io.

The fastest way to get started is by deploying the operator in an OCP/OKD cluster using the setup scripts provided in the hack directory:

./hack/storage.sh  # install noobaa via ODF operator
./hack/deploy.sh
oc create -n <your-namespace> -f ./config/samples/managed.quayregistry.yaml

Or run the steps one by one.

Step by step

Create the CatalogSource:

$ kubectl create -n openshift-marketplace -f ./bundle/quay-operator.catalogsource.yaml

Wait a few seconds for the package to become available:

$ kubectl get packagemanifest --all-namespaces | grep quay

Create the OperatorGroup:

$ kubectl create -n <your-namespace> -f ./bundle/quay-operator.operatorgroup.yaml

Create the Subscription to install the Operator:

$ kubectl create -n <your-namespace> -f ./bundle/quay-operator.subscription.yaml

Using the Operator

Component Container Images

When using a downstream build or container image overrides which are hosted in private repositories, you can provide pull secrets by adding them to the default ServiceAccount in the namespace.

Batteries-included, zero-config

Install RHOCS Operator using OperatorHub:

Create NooBaa object in openshift-storage namespace:

$ kubectl create -n openshift-storage -f ./kustomize/components/objectstorage/quay-datastore.noobaa.yaml

Wait a few minutes for Noobaa to be phase: Ready:

$ kubectl get -n openshift-storage noobaas noobaa -w
NAME     MGMT-ENDPOINTS              S3-ENDPOINTS                IMAGE                                                                                                            PHASE   AGE
noobaa   [https://10.0.32.3:30318]   [https://10.0.32.3:31958]   registry.redhat.io/ocs4/mcg-core-rhel8@sha256:56624aa7dd4ca178c1887343c7445a9425a841600b1309f6deace37ce6b8678d   Ready   3d18h

Create QuayRegistry instance:

$ kubectl create -n <your-namespace> -f ./config/samples/managed.quayregistry.yaml

Community

Mailing list: [email protected]
IRC: #quay on freenode.net
Bug tracking: https://issues.redhat.com/projects/PROJQUAY/summary
Security issues: [email protected]

Contributing

Pull requests and bug reports are always welcome!

Local Development

Prerequisites

KUBECONFIG environment variable set in shell to valid k8s cluster
go
kubectl
kubebuilder
docker

Create the QuayRegistry CRD:

$ kubectl create -f ./bundle/upstream/manifests/*.crd.yaml

Run the controller:

$ make run

Tests:

$ make test

Building custom CatalogSource:

Build and push the Quay Operator container:

$ docker build -t <some-registry>/<namespace>/quay-operator:dev .
$ docker push <some-registry>/<namespace>/quay-operator:dev

Replace the image field in bundle/upstream/manifests/quay-operator.clusterserviceversion.yaml with the image above.
Build and push an Operator bundle:

$ docker build -t <some-registry>/<namespace>/quay-operator-bundle:dev -f ./bundle/Dockerfile ./bundle
$ docker push <some-registry>/<namespace>/quay-operator-bundle:dev

Build and push an Operator index image using opm:

$ cd bundle/upstream
$ opm index add --bundles <some-registry>/<namespace>/quay-operator-bundle:dev --tag <some-registry>/<namespace>/quay-operator-index:dev
$ docker push <some-registry>/<namespace>/quay-operator-index:dev

Replace the spec.image field in bundle/quay-operator.catalogsource.yaml with the image above.
Create the custom CatalogSource:

$ kubectl create -n openshift-marketplace -f ./bundle/quay-operator.catalogsource.yaml

quay-operator's People

Contributors

Stargazers

Watchers

Forkers

sabre1041 cnuland theodor2311 based-god-fucked-my-bitch-fuckzig rutvik23 rhsyang rajeshm123 billdett dgangaia scouturier nanyte25 dlt-solutions samycoenen emreozkangit bkugler-xv shaneutt dmvolod lizhang19817 drichtarik mbach04 ossys jterceiro jjmengze jbaresfe pandeybk ssgkim elimaneg flaviolnx juozasa ianwatsonrh dmesser nwang92 nsweducation mathianasj sdirbach krishchow mosen alecmerdler thomasmckay bonomali jonathankingfc wewang58 raif-ahmed hupiper clix-dev-llc davar-playgrounds syed arajkumar crozzy maduhu bentito rmallam harishsurf dmage pombredanne flavianmissi fjcloud openshift-cherrypick-robot franklinharry arceojaviere johsonluo oatakan sylaroche hdonnay isabella232 hammermeetnail bcaton85 doytsujin jcho02 eb4x ksdeekshith modassarrana89 yussufsh michaelalang jorikseldeslachts arborite-rh david-vtuk

quay-operator's Issues

Document how to source images from a difference source than the defaults

Document how to configure the operator and its dependence resources can be sourced from a registry other than the default quay.io/registry.access.redhat.com registries:

Operator
Quay
Clair
Redis

starting container process caused "exec: \"quay-operator\": executable file not found in $PATH"

Pod error

Error: failed to start container "quay-operator": Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \"quay-operator\": executable file not found in $PATH"

Environment: minishift / okd-3.11

Steps:

oc new-project quay-enterprise
oc create -f deploy/crds/cop_v1alpha1_quayecosystem_crd.yaml
oc create -f deploy/service_account.yaml
oc create -f deploy/cluster_role.yaml
oc create -f deploy/cluster_role_binding.yaml
oc create -f deploy/role.yaml
oc create -f deploy/role_binding.yaml
oc create -f deploy/operator.yaml
oc create secret generic redhat-pull-secret --from-file=".dockerconfigjson=/home/thomasmckay/.docker/config.json" --type='kubernetes.io/dockerconfigjson'
oc project
oc create -f deploy/crds/cop_v1alpha1_quayecosystem_cr.yaml
<edit deploy/crds/cop_v1alpha1_quayecosystem_cr.yaml to add postgresql>
oc apply -f deploy/crds/cop_v1alpha1_quayecosystem_cr.yaml

OLM Integration

Create the assets to integrate the operator into the Operator Lifecycle Manager

As a user, I want to run repo mirroring services, so that the feature is enabled and works.

With quay-v3.1.0 there is a new feature named "repo mirroring". This feature defaults to off and thus must be enabled via the config.yaml/config app. In addition, by default the worker (aka service) that does the mirroring is not started in the "registry" container mode. Instead, a new entrypoint "repomirror" is available to run only the services necessary for this feature. The intent is that there will be both a "registry" pod and the "repomirror" pod running. The "repomirror" pod requires access to the database (like "registry" does), as well as access to external registries to be mirrored. An external endpoint is not required.

An alternate starting mechanism for "repomirror" is available to run the worker inside the "registry" container. To do this, the environment variable QUAY_OVERRIDE_SERVICES is given a value of repomirrorrowker=true. This mode is intended to be for POC but is fully functional and may, in fact, be the most common mode for users.

For the quay-operator, I'd suggest two phase approach: First, add the env var method to allow usage. Next, add effectively a duplicate setup as "registry" but change entry point to "repomirror".

Differences in "repomirror" pod:

At this time, there is no readiness/liveness endpoints so default them to be omitted. Leave the functionality, though, since upcoming release will support that.
An external route to this pod is unnecessary at this time. A distant release may support this, though, so if it makes sense to leave support in code but disabled, that would not be wasted effort.
Scaling, sizing, and other pod aspects should be available but separate from the "registry" pods (ie. they will be adjusted separately).

Variabilize Project Name

The project name (namespace) should be a variable, therefore giving the ability to have multiple instances of Quay running with the operator on the same cluster more easily.

config app deployment rollout failure

At the conclusion of the config app, there is a "Deploy configuration" page. This errors trying to deploy:

From the logs

K8sApiException: Kubernetes API call failed: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Deployment.apps \"example-quayecosystem-quay\" is invalid: spec.template.spec.containers[0].image: Required value","reason":"Invalid","details":{"name":"example-quayecosystem-quay","group":"apps","kind":"Deployment","causes":[{"reason":"FieldValueRequired","message":"Required value","field":"spec.template.spec.containers[0].image"}]},"code":422}

This is a permission problem, I believe, and not an error in quay. Perhaps these roles can help narrow it down:
role: https://github.com/quay/quay/blob/master/config_app/docs/k8s_templates/config-tool-servicetoken-role.yml
rolebinding: https://github.com/quay/quay/blob/master/config_app/docs/k8s_templates/config-tool-servicetoken-role-binding.yml
service account: https://github.com/quay/quay/blob/master/config_app/docs/k8s_templates/config-tool-serviceaccount.yml
config tool deployment: https://github.com/quay/quay/blob/master/config_app/docs/k8s_templates/qe-config-tool.yml#L19

Add unit and integration testing

Add unit and integration tests to the repository

Quay Registry Storage Mode changes from RWO as specified to RWX

Somewhere after the creation of the Quay Ecosystem object the storage mode is being changed from RWO to RWX causing the deployment to fail on AWS.

I'm creating like this:
registryStorage:
persistentVolumeAccessMode:
- ReadWriteOnce

And sometime it gets changed to RWX. Which is weird. Why would it even change? This is on 4.1.25 on AWS (IPI).

Quay Config not available

I followed the instructions and deployed the operator with redis/clair/external Postgres DB and deployed the ecosystem with the following config:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: QuayEcosystem
metadata:
name: quayecosystem
spec:
quay:
imagePullSecretName: redhat-pull-secret
database:
credentialsSecretName: quay-postgres-creds
server: postgres
routeHost: quay-quay-enterprise.apps.dltssf.demo-dlt.com
superuserCredentialsSecretName: quay-super
configSecretName: quay-config-secret
configRouteHost: quay-config-quay-enterprise.apps.dltssf.demo-dlt.com
deploymentStrategy: RollingUpdate
skipSetup: false
registryStorage:
persistentVolumeAccessMode:
- ReadWriteOnce
persistentVolumeSize: 10Gi
persistentVolumeStorageClassName: quay-storageclass
clair:
enabled: true
imagePullSecretName: redhat-pull-secret
updateInterval: "60m"
redis:
credentialsSecretName: redis-creds
imagePullSecretName: redhat-pull-secret

The quay-config log shows that it had nginx permission errors and never fully starts. Any suggestions?

/ \ / \ ______ _ _ __ __ __
/ /\ / /\ \ / __ \ | | | | / \ \ \ / /
/ / / / \ \ | | | | | | | | / /\ \ \ /
\ \ \ \ / / | || | | || | / ____ \ | |
\ / \ / / _ / _/ // _\ ||
_/ _/ \ __
___\ by Red Hat

Build, Store, and Distribute your Containers

Entering config mode, only copying config-app entrypoints
Using mounted ssl certs for quay-config app
2019-11-26 22:41:36,675 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2019-11-26 22:41:36,685 INFO RPC interface 'supervisor' initialized
2019-11-26 22:41:36,686 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-11-26 22:41:36,686 INFO supervisord started with pid 32
2019-11-26 22:41:37,688 INFO spawned: 'stdout' with pid 35
2019-11-26 22:41:37,690 INFO spawned: 'nginx' with pid 36
2019-11-26 22:41:37,691 INFO spawned: 'gunicorn-config' with pid 37
2019/11/26 22:41:37 [alert] 38#0: setpriority(-10) failed (13: Permission denied)
2019/11/26 22:41:37 [alert] 39#0: setpriority(-10) failed (13: Permission denied)
2019/11/26 22:41:37 [alert] 40#0: setpriority(-10) failed (13: Permission denied)
2019/11/26 22:41:37 [alert] 41#0: setpriority(-10) failed (13: Permission denied)
2019/11/26 22:41:37 [alert] 42#0: setpriority(-10) failed (13: Permission denied)
2019/11/26 22:41:37 [alert] 43#0: setpriority(-10) failed (13: Permission denied)
2019/11/26 22:41:37 [alert] 44#0: setpriority(-10) failed (13: Permission denied)
2019/11/26 22:41:37 [alert] 45#0: setpriority(-10) failed (13: Permission denied)
2019/11/26 22:41:37 [alert] 46#0: setpriority(-10) failed (13: Permission denied)
2019/11/26 22:41:37 [alert] 47#0: setpriority(-10) failed (13: Permission denied)
2019/11/26 22:41:37 [alert] 48#0: setpriority(-10) failed (13: Permission denied)
2019/11/26 22:41:37 [alert] 49#0: setpriority(-10) failed (13: Permission denied)
2019/11/26 22:41:37 [alert] 50#0: setpriority(-10) failed (13: Permission denied)
2019/11/26 22:41:37 [alert] 52#0: setpriority(-10) failed (13: Permission denied)
2019/11/26 22:41:37 [alert] 53#0: setpriority(-10) failed (13: Permission denied)
2019/11/26 22:41:37 [alert] 51#0: setpriority(-10) failed (13: Permission denied)
2019-11-26 22:41:38,707 INFO success: stdout entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-11-26 22:41:38,707 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-11-26 22:41:38,707 INFO success: gunicorn-config entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
nginx stdout | 2019/11/26 22:41:37 [alert] 38#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2019/11/26 22:41:37 [alert] 39#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2019/11/26 22:41:37 [alert] 40#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2019/11/26 22:41:37 [alert] 41#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2019/11/26 22:41:37 [alert] 42#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2019/11/26 22:41:37 [alert] 43#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2019/11/26 22:41:37 [alert] 44#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2019/11/26 22:41:37 [alert] 45#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2019/11/26 22:41:37 [alert] 46#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2019/11/26 22:41:37 [alert] 47#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2019/11/26 22:41:37 [alert] 48#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2019/11/26 22:41:37 [alert] 49#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2019/11/26 22:41:37 [alert] 50#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2019/11/26 22:41:37 [alert] 52#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2019/11/26 22:41:37 [alert] 53#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2019/11/26 22:41:37 [alert] 51#0: setpriority(-10) failed (13: Permission denied)
2019-11-26 22:41:38,943 [37] [INFO] [data.registry_model] ===============================
gunicorn-config stdout | 2019-11-26 22:41:38,943 [37] [INFO] [data.registry_model] ===============================
2019-11-26 22:41:38,943 [37] [INFO] [data.registry_model] Using registry model <data.registry_model.registry_pre_oci_model.PreOCIModel object at 0x7f281cacacd0>
2019-11-26 22:41:38,944 [37] [INFO] [data.registry_model] ===============================
gunicorn-config stdout | 2019-11-26 22:41:38,943 [37] [INFO] [data.registry_model] Using registry model <data.registry_model.registry_pre_oci_model.PreOCIModel object at 0x7f281cacacd0>
2019-11-26 22:41:38,944 [37] [INFO] [data.registry_model] ===============================
2019-11-26 22:41:38,983 [37] [DEBUG] [config_app.c_app] Configuration is on a kubernetes deployment: True
gunicorn-config stdout | 2019-11-26 22:41:38,983 [37] [DEBUG] [config_app.c_app] Configuration is on a kubernetes deployment: True
2019-11-26 22:41:38,988 [37] [DEBUG] [config_app.c_app] Loading default config.
gunicorn-config stdout | 2019-11-26 22:41:38,988 [37] [DEBUG] [config_app.c_app] Loading default config.
2019-11-26 22:41:39,096 [37] [DEBUG] [config] Starting local gunicorn with 1 workers and gevent worker class
gunicorn-config stdout | 2019-11-26 22:41:39,096 [37] [DEBUG] [config] Starting local gunicorn with 1 workers and gevent worker class

Execute tests during Travis build

WIth the introduction of testing to the repository, enable support for executing the tests as part of the Travis build

Use Quay 3.2

Once Quay 3.2 becomes available, any references to the Quay image should be modified to reflect the appropriate, newer release.

Although I have marked this as high priority, the image is not currently available. I will include the location of the new image just as soon as it is available.

Clair failed to create environment vars

quay.io/redhat-cop/quay-operator:v0.0.7
quay.io/redhat/clair-jwt:v3.1.0

I defined the CRD as follows but environment vars were not created in Clair deployment

  clair:
    enabled: true
    envVars:
      - name: HTTP_PROXY
        value: "http://proxy.example.com:80"
      - name: HTTPS_PROXY
        value: "http://proxy.example.com:80"
    ...

workaround is to manually add to Deployment

Provide the ability to customize the rollout type

Currently, the default method of updating a new deployment is to use the Rollout strategy. This can cause issues when the underlying storage is block for several of the components.

Provide the ability to specify the type of strategy in the QuayEcosystem object and have it applied to the applicable resource

Invalid DB volume size breaks deployment

While verifying the persistent volume for Postgres, I accidentally specified an invalid unit of measure. The operator did not provide any useful log information as to why my Quay Ecosystem was not being deployed.

After correcting the unit of measurement and redeploying my CR, the ecosystem deployed successfully.

Example:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: QuayEcosystem
metadata:
  name: example-quayecosystem
spec:
  quay:
    imagePullSecretName: redhat-pull-secret
    database:
      volumeSize: 10gi  # This should be 10Gi

Add nodeSelector to deployments

The following deployments have to be updated with a new nodeSelector:

example-quayecosystem-quay-config
example-quayecosystem-quay

The setting should be:

nodeSelector:
       node-role.kubernetes.io/infra: "true"

Need support for namespaces other than quay-enterprise

This turns out to be a hardcoded quay-enterprise namespace. We need a way to customize the namespace that quay is deployed (e.g. quay-aaaa, quay-bbbb, ....) for multiple Quay Deployments on one OCP Cluster.

Related to #28

Need a way to customize the route

Right now the default route is always used for Quay. Which makes for a rather "funny" URL. For a quay-enterprise installation of name 'quay' it is quay-quay-quay-enterprise.apps.cluster-shared.shared.example.opentlc.com.

It would be good to just specify the route name as "quay.apps.cluster-shared.shared.example.opentlc.com".

So instead of service-project (which is quay-quay - quay-enterprise) just 'quay' would be enough.

Add a field to the CR to change this.

Convert SSL Secrets to type kubernetes.io/tls

Currently, all secrets are of type Opaque type. Migrate SSL based certificates to type kubernetes.io/tls

Persistent storage for Clair DB is missing

There are no setting for specifying persistent storage parameters for Clair DB, leading to empty Volumes section in the deployment and all data stored ephemerally:

oc describe deployment/example-quayecosystem-clair-postgresql
...
Volumes:                  <none>
...

Add support for multiple instances per cluster

It would be interesting to add the ability to run multiple instances per cluster. This would allow having an upgraded version of Quay &/or the operator running on the same cluster and allow teams to test changes rather than potentially disrupt the existing deployment.

Quay pod does not start up with missing config secret error

When deploying a Quay the other pods come up. But the quay pod itself does not come up.

Logs:

Build, Store, and Distribute your Containers

Running both interactive and batch scripts
Running init script '/quay-registry/conf/init/01_copy_syslog_config.sh'
Running init script '/quay-registry/conf/init/02_get_kube_certs.sh'
Running on kubernetes, attempting to retrieve extra certs from secret
/opt/rh/python27/root/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
Traceback (most recent call last):
File "/quay-registry/conf/init/02_get_kube_certs.py", line 71, in
main()
File "/quay-registry/conf/init/02_get_kube_certs.py", line 55, in main
secret_data = _lookup_secret(service_token).get('data', {})
File "/quay-registry/conf/init/02_get_kube_certs.py", line 26, in _lookup_secret
raise Exception('Cannot get the config secret')
Exception: Cannot get the config secret

Redis should support password authentication

Support deploying Redis secured via a password

Deployment of Redis secured via a password
Configure Quay to communicate with Redis using a password

Add readiness and liveness probes parameters

I would be convenient to have readiness and liveness setting exposed through the custom resource in order to configure them during initial rollout, as opposed to editing them later and triggering a new deployment.

Provide various examples of QuayEcosystem custom resource

Provide various examples demonstrating various configurations of QuayEcosystem custom resources

Quay pod does not come up with v0.0.4 image - CrashLoopBackoff

Logs:

Build, Store, and Distribute your Containers

Running both interactive and batch scripts
Running init script '/quay-registry/conf/init/01_copy_syslog_config.sh'
Running init script '/quay-registry/conf/init/02_get_kube_certs.sh'
Running on kubernetes, attempting to retrieve extra certs from secret
/opt/rh/python27/root/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
Running init script '/quay-registry/conf/init/certs_create.sh'
2019/08/27 14:48:42 [INFO] generating a new CA key and certificate from CSR
2019/08/27 14:48:42 [INFO] generate received request
2019/08/27 14:48:42 [INFO] received CSR
2019/08/27 14:48:42 [INFO] generating key: rsa-2048
2019/08/27 14:48:43 [INFO] encoded CSR
2019/08/27 14:48:43 [INFO] signed certificate with serial number 634766108780862621552222810806137694881381439010
cp: cannot create regular file ‘/etc/pki/ca-trust/source/anchors/mitm.crt’: Permission denied

CR used:

[ec2-user@bastion 0 ~]$ oc get quayecosystem quay -o yaml
apiVersion: redhatcop.redhat.io/v1alpha1
kind: QuayEcosystem
metadata:
creationTimestamp: "2019-08-27T14:44:46Z"
generation: 2
name: quay
namespace: quay-enterprise
resourceVersion: "24268106"
selfLink: /apis/redhatcop.redhat.io/v1alpha1/namespaces/quay-enterprise/quayecosystems/quay
uid: 36d98f99-c8d9-11e9-972a-06699c3c6ac6
spec:
clair:
database:
deploymentStrategy: RollingUpdate
image: registry.access.redhat.com/rhscl/postgresql-96-rhel7:1
resources: {}
deploymentStrategy: RollingUpdate
enabled: true
image: quay.io/redhat/clair-jwt:v3.0.4
imagePullSecretName: quay-pull-secret
resources: {}
updateInterval: 10m
quay:
configResources: {}
configSecretName: quay-config-secret
database:
deploymentStrategy: RollingUpdate
image: registry.access.redhat.com/rhscl/postgresql-96-rhel7:1
resources: {}
volumeSize: 10Gi
deploymentStrategy: RollingUpdate
image: quay.io/redhat/quay:v3.0.4
imagePullSecretName: quay-pull-secret
registryBackends:
- local:
storage_path: /datastorage/registry
name: default
registryStorage:
persistentVolumeAccessMode:
- ReadWriteOnce
persistentVolumeSize: 15Gi
resources: {}
routeHost: quay-dev.apps.shared-dev.dev.openshift.opentlc.com
sslCertificatesSecretName: quay-ssl-certificate-secret
superuserCredentialsSecretName: quay-superuser-secret
redis:
deploymentStrategy: RollingUpdate
image: registry.access.redhat.com/rhscl/redis-32-rhel7:latest
resources: {}
status:
conditions:

lastTransitionTime: "2019-08-27T14:44:56Z"
lastUpdateTime: "2019-08-27T14:44:56Z"
message: Configuration Updated Successfully
status: "True"
type: UpdateDefaultConfigurationSuccess

Include a method for providing additional CA certificates

Quay may need to communicate with SSL endpoints (such as LDAP) which it does not trust by default. Provide a way to configure additional user provided certificates

Issue pulling Image from Quay on OpenShift v4.2.8

Hi,

I have used the Operator to successfully install Quay on OpenShift v4.2.x & pushed an image to it.

I am trying to use that image in a simple Pod but am seeing:

Failed to pull image "example-quayecosystem-quay-quay.apps-crc.testing/stevef/alpine-curl-jq": rpc error: code = Unknown desc = pinging docker registry returned: Get https://example-quayecosystem-quay-quay.apps-crc.testing/v2/: x509: certificate signed by unknown authority

I have created an imagePullSecrets entry but I don't know how to set the TLS verification flag off or add quay's certs.

Any ideas/pointers?

Regards

Steve

S3 as storage results in error deploying

I am trying to deploy using S3 storage and I am receiving the following error when trying to deploy the cr:

$ oc create -f s3_cr.yaml
The QuayEcosystem "example-quayecosystem" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"redhatcop.redhat.io/v1alpha1", "kind":"QuayEcosystem", "metadata":map[string]interface {}{"creationTimestamp":"2019-11-06T21:33:31Z", "generation":1, "name":"example-quayecosystem", "namespace":"quay-enterprise", "uid":"1427b888-00dd-11ea-a98d-02dd80a8768f"}, "spec":map[string]interface {}{"quay":map[string]interface {}{"imagePullSecretName":"redhat-pull-secret", "registryStorage":[]interface {}{map[string]interface {}{"name":"s3", "s3":map[string]interface {}{"host":"hosthere", "s3_access_key":"accesskey", "s3_bucket":"quay", "s3_secret_key":"secret+key"}}}}}}: validation failure list:
spec.quay.registryStorage in body must be of type object: "array"

This is what my cr looks like:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: QuayEcosystem
metadata:
name: example-quayecosystem
spec:
quay:
imagePullSecretName: redhat-pull-secret
registryStorage:
- name: s3
s3:
s3_bucket: quay
s3_access_key: acceskey
s3_secret_key: secret+key
host: host

using credentialsSecretName breaks deployment

when I specify a custom credentialsSecretName the operator fails to deploy the DB.

operator logs:

"level":"error","ts":1565109241.4033964,"logger":"controller_quayecosystem","msg":"Failed to Setup Quay","error":"database Validation Failed: FATAL:  password authentication failed for user \"quay\"\n","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\ngithub.com/redhat-cop/quay-operator/pkg/controller/quayecosystem.(*ReconcileQuayEcosystem).Reconcile\n\t/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/quayecosystem_controller.go:191\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1565109281.409356,"logger":"controller_quayecosystem","msg":"Reconciling QuayEcosystem","Request.Namespace":"quay-enterprise","Request.Name":"quay"}

secrete creation:

oc create secret generic quay-db-credential --from-literal=database-username=quay --from-literal=database-password=foobar --from-literal=database-root-password=foobar --from-literal=database-name=quay

quayechosystem definition:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: QuayEcosystem
metadata:
  name: quay
spec:
  quay:
    imagePullSecretName: quayio-pull-secret
    superuserCredentialsSecretName: quay-superuser-credential
    configSecretName: quay-config-credential
    routeHost: quay.apps.example.com
    configRouteHost: quay-config.apps.example.com
    database:
      credentialsSecretName: quay-db-credential
      volumeSize: 10G
    registryStorage:
      persistentVolumeAccessMode:
        - ReadWriteOnce
      persistentVolumeSize: 10Gi

Add config options for various storage backends

Currently, we only support local storage backend for configuration, which is undesirable in production environments. We should add config support for other, cloud storage backends.

Low success rate on HA environment

I have 2 environments both running on Openshift 3.11. The first is a simple sandbox environment consisting of 1 master and 1 compute node, the other is a HA environment consisting of 3 masters and 6 compute nodes.

I attempted multiple deploys on both, and this is what I noticed.

The simple sandbox environment had a 100% success rate over 6 attempts.

The HA sandbox has 1 success over perhaps 25-30 attempts. The 2 failures I see consistently in the HA environment are:

The config doesn't complete, and the quay app never gets spun up. I get a QuaySetupFailure event bubbling up to the Openshift events console:

Failed to create superuser: invalid character '<' looking for beginning value

If 1) doesn't happen, and configuration succeeds, the quay app is spun up, however, it doesn't start due to 502's and timeouts. Here is a snippet of the logs at startup until those errors start repeating:

2019/09/20 18:40:37 [crit] 261#0: *1 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.244.14.1, server: _, request: "GET /health/instance HTTP/2.0", upstream: "http://unix:/tmp/gunicorn_web.sock:/health/instance", host: "10.244.14.10:8443"
2019/09/20 18:40:37 [crit] 261#0: *1 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.244.14.1, server: _, request: "GET /health/instance HTTP/2.0", upstream: "http://unix:/tmp/gunicorn_web.sock:/quay-registry/static/502.html", host: "10.244.14.10:8443"
10.244.14.1 () - - [20/Sep/2019:18:40:37 +0000] "GET /health/instance HTTP/2.0" 502 173 "-" "kube-probe/1.11+" (0.000 49 0.000 : 0.000)
nginx stdout | 2019/09/20 18:40:37 [crit] 261#0: *1 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.244.14.1, server: _, request: "GET /health/instance HTTP/2.0", upstream: "http://unix:/tmp/gunicorn_web.sock:/health/instance", host: "10.244.14.10:8443"
2019/09/20 18:40:37 [crit] 261#0: *1 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.244.14.1, server: _, request: "GET /health/instance HTTP/2.0", upstream: "http://unix:/tmp/gunicorn_web.sock:/quay-registry/static/502.html", host: "10.244.14.10:8443"
10.244.14.1 () - - [20/Sep/2019:18:40:37 +0000] "GET /health/instance HTTP/2.0" 502 173 "-" "kube-probe/1.11+" (0.000 49 0.000 : 0.000)
2019/09/20 18:40:47 [crit] 259#0: *4 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.244.14.1, server: _, request: "GET /health/instance HTTP/2.0", upstream: "http://unix:/tmp/gunicorn_web.sock:/health/instance", host: "10.244.14.10:8443"
2019/09/20 18:40:47 [crit] 259#0: *4 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.244.14.1, server: _, request: "GET /health/instance HTTP/2.0", upstream: "http://unix:/tmp/gunicorn_web.sock:/quay-registry/static/502.html", host: "10.244.14.10:8443"
10.244.14.1 () - - [20/Sep/2019:18:40:47 +0000] "GET /health/instance HTTP/2.0" 502 173 "-" "kube-probe/1.11+" (0.000 49 0.000 : 0.000)
nginx stdout | 2019/09/20 18:40:47 [crit] 259#0: *4 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.244.14.1, server: _, request: "GET /health/instance HTTP/2.0", upstream: "http://unix:/tmp/gunicorn_web.sock:/health/instance", host: "10.244.14.10:8443"

If you require more log data or have a suggested workaround, etc. please let me know.
If you would like full log files, let me know where I can send/drop them.

Add Operator to operatorhub.io

Update OLM assets for the changes made in the repository and add to operatorhub.io

Add support for clusters with Wildcard certificates

There should be a way to skip the SSL secret generation and service/route configuration as HTTPS. As an example, one environment using wildcard certificates would have to go through the following process in order to use it's wildcard instead of the self-signed cerificate:

So far, I was able to get it up and running by doing the following:

Update the service/quayecosystem-quay following fields
1. Service Port: 80 (instead of 443)
2. Target port: 8080 (instead of 8443)
Update the route "quayecosystem-quay"
1. TLS termination: Edge (instead of Passthrough)
2. Target port: 8080 (instead of 8443)

RBAC error running on OCP 4.x

I'm getting an RBAC error inside the Operator when running on OCP 4.1.14 (locally via 'crc'):

[bdettelb@localhost ~]$ oc version
Client Version: v4.2.0
Server Version: 4.1.14
Kubernetes Version: v1.13.4+b758672

I installed the Setup Operator (v0.0.6) from the OperatorHub via the OpenShift web console. Here is my CR:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: QuayEcosystem
metadata:
  creationTimestamp: '2019-09-30T17:46:14Z'
  generation: 2
  name: example-quayecosystem
  namespace: quay-enterprise
  resourceVersion: '256660'
  selfLink: >-
    /apis/redhatcop.redhat.io/v1alpha1/namespaces/quay-enterprise/quayecosystems/example-quayecosystem
  uid: 32e1d9df-e3aa-11e9-8117-52fdfc072182
spec:
  quay:
    configResources: {}
    resources: {}
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /health/instance
        port: 8443
        scheme: HTTPS
      initialDelaySeconds: 5
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /health/instance
        port: 8443
        scheme: HTTPS
      initialDelaySeconds: 120
      timeoutSeconds: 5
    deploymentStrategy: RollingUpdate
    image: 'quay.io/redhat/quay:v3.1.0'
    database:
      deploymentStrategy: RollingUpdate
      image: 'registry.access.redhat.com/rhscl/postgresql-96-rhel7:1'
      livenessProbe:
        exec:
          command:
            - /usr/libexec/check-container
            - '--live'
        failureThreshold: 3
        initialDelaySeconds: 120
        timeoutSeconds: 10
      readinessProbe:
        exec:
          command:
            - /usr/libexec/check-container
        failureThreshold: 3
        initialDelaySeconds: 10
        timeoutSeconds: 1
      resources: {}
      volumeSize: 100Gi
    imagePullSecretName: redhat-pull-secret
    registryBackends:
      - local:
          storage_path: /datastorage/registry
        name: default
    registryStorage:
      persistentVolumeAccessMode:
        - ReadWriteOnce
      persistentVolumeSize: 100Gi
  redis:
    deploymentStrategy: RollingUpdate
    image: registry.redhat.io/rhscl/redis-32-rhel7
    readinessProbe:
      failureThreshold: 3
      initialDelaySeconds: 30
      tcpSocket:
        port: 6379
    resources: {}
status:
  conditions:
    - lastTransitionTime: '2019-09-30T17:46:14Z'
      lastUpdateTime: '2019-09-30T17:46:14Z'
      message: Configuration Updated Successfully
      status: 'True'
      type: UpdateDefaultConfigurationSuccess
    - lastTransitionTime: '2019-09-30T17:46:15Z'
      lastUpdateTime: '2019-09-30T20:02:47Z'
      message: >-
        roles.rbac.authorization.k8s.io "example-quayecosystem" is forbidden:
        user "system:serviceaccount:quay-enterprise:quay-operator"
        (groups=["system:serviceaccounts"
        "system:serviceaccounts:quay-enterprise" "system:authenticated"]) is
        attempting to grant RBAC permissions not currently held:

        {APIGroups:[""], Resources:["secrets"], Verbs:["put"]}
      reason: ProcessingError
      status: 'False'
      type: QuayEcosystemProvisioningFailure

Error lines from Operator:

{"level":"error","ts":1569869669.5055368,"logger":"util","msg":"unable to create object","object":{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"Role","namespace":"quay-enterprise","name":"example-quayecosystem"},"error":"roles.rbac.authorization.k8s.io \"example-quayecosystem\" is forbidden: user \"system:serviceaccount:quay-enterprise:quay-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:quay-enterprise\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"\"], Resources:[\"secrets\"], Verbs:[\"put\"]}","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\ngithub.com/redhat-cop/operator-utils/pkg/util.(*ReconcilerBase).CreateOrUpdateResource\n\t/home/travis/gopath/pkg/mod/github.com/redhat-cop/[email protected]/pkg/util/reconciler.go:153\ngithub.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/provisioning.(*ReconcileQuayEcosystemConfiguration).createRBAC\n\t/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/provisioning/provision.go:585\ngithub.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/provisioning.(*ReconcileQuayEcosystemConfiguration).CoreQuayResourceDeployment\n\t/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/provisioning/provision.go:59\ngithub.com/redhat-cop/quay-operator/pkg/controller/quayecosystem.(*ReconcileQuayEcosystem).Reconcile\n\t/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/quayecosystem_controller.go:136\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1569869669.5056179,"logger":"controller_quayecosystem","msg":"Failed to create RBAC","error":"roles.rbac.authorization.k8s.io \"example-quayecosystem\" is forbidden: user \"system:serviceaccount:quay-enterprise:quay-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:quay-enterprise\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"\"], Resources:[\"secrets\"], Verbs:[\"put\"]}","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\ngithub.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/provisioning.(*ReconcileQuayEcosystemConfiguration).CoreQuayResourceDeployment\n\t/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/provisioning/provision.go:60\ngithub.com/redhat-cop/quay-operator/pkg/controller/quayecosystem.(*ReconcileQuayEcosystem).Reconcile\n\t/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/quayecosystem_controller.go:136\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1569873767.5101357,"logger":"controller_quayecosystem","msg":"Reconciling QuayEcosystem","Request.Namespace":"quay-enterprise","Request.Name":"example-quayecosystem"}
{"level":"error","ts":1569873767.5324595,"logger":"util","msg":"unable to create object","object":{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"Role","namespace":"quay-enterprise","name":"example-quayecosystem"},"error":"roles.rbac.authorization.k8s.io \"example-quayecosystem\" is forbidden: user \"system:serviceaccount:quay-enterprise:quay-operator\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:quay-enterprise\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"\"], Resources:[\"secrets\"], Verbs:[\"put\"]}","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\ngithub.com/redhat-cop/operator-utils/pkg/util.(*ReconcilerBase).CreateOrUpdateResource\n\t/home/travis/gopath/pkg/mod/github.com/redhat-cop/[email protected]/pkg/util/reconciler.go:153\ngithub.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/provisioning.(*ReconcileQuayEcosystemConfiguration).createRBAC\n\t/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/provisioning/provision.go:585\ngithub.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/provisioning.(*ReconcileQuayEcosystemConfiguration).CoreQuayResourceDeployment\n\t/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/provisioning/provision.go:59\ngithub.com/redhat-cop/quay-operator/pkg/controller/quayecosystem.(*ReconcileQuayEcosystem).Reconcile\n\t/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/quayecosystem_controller.go:136\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88"}

Add CI for repository

Add CI components to enable image building per pull requests and commits to master

As a user, I want API calls from within the container to successfully TLS verify, so that repo mirroring can work.

When running repo mirroring (upcoming feature in quay:v3.1.0), skopeo errors trying to copy to the registry:

Get https://example-quayecosystem-quay-quay-enterprise.apps.thomasmckay-quay.devcluster.openshift.com/v2/: x509: certificate signed by unknown authority"

This implies, I think, that there needs to be a trusted rootCA.key installed during config setup.

Add possibility to restore an existing configuration

In case of catastrophic failure, it would be nice to have a way to restore a somewhat backedup Quay/Clair configuration with minimal to no loss.

Right now there is no way to restore anything officially, so devs have to start over.

Quay pod will not restart properly when using the config app when using persistent storage

Quay pod will not take updates from the config app when using persistent storage due to the RollingUpdate policy. Manually setting the deployment to "Recreate" solves the issue for now but would most likely be overwritten by the operator.

Notifier service is disabled

quay.io/redhat-cop/quay-operator:v0.0.7
quay.io/redhat/clair-jwt:v3.1.0
The title message appears in the container log when clair is starting. I think clair/config.yaml could be changed:

--old--

updater:
  notifier:
    attempts: 1
    renotifyinterval: 1h0m0s
    http:
      endpoint: https://example.hostname.com/secscan/notify
      proxy: http://localhost:6063
  interval: 8h20m0s

--new--

updater:
  interval: 8h20m0s
notifier:
  attempts: 1
  renotifyinterval: 1h0m0s
  http:
    endpoint: https://example.hostname.com/secscan/notify
    proxy: http://localhost:6063

RBAC error: Failed to list *v1.Build: builds.build.openshift.io on OCP 3.11

I'm getting RBAC error inside the quay operator when running on 3.11

$ oc version
Client Version: v3.11.98
Server Version: v3.11.153
kubernetes v1.11.0+d4cacc0

Here is the deployment config

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: '2'
  creationTimestamp: '2019-07-02T23:44:59Z'
  generation: 8
  labels:
    name: quay-openshift-registry-operator
  name: quay-openshift-registry-operator
  namespace: quay-integration
  resourceVersion: '305368901'
  selfLink: >-
    /apis/apps/v1/namespaces/quay-integration/deployments/quay-openshift-registry-operator
  uid: 6780777d-9d23-11e9-95b1-001a4a408efd
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      name: quay-openshift-registry-operator
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        name: quay-openshift-registry-operator
    spec:
      containers:
        - command:
            - quay-openshift-registry-operator
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: quay-openshift-registry-operator
          image: 'quay.io/redhat-cop/quay-openshift-registry-operator:latest'
          imagePullPolicy: Always
          name: quay-openshift-registry-operator
          ports:
            - containerPort: 8443
              protocol: TCP
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /etc/webhook/certs
              name: certs
              readOnly: true
      dnsPolicy: ClusterFirst
      imagePullSecrets:
        - name: redhat-pull-secret
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: quay-openshift-registry-operator
      serviceAccountName: quay-openshift-registry-operator
      terminationGracePeriodSeconds: 30
      volumes:
        - name: certs
          secret:
            defaultMode: 420
            secretName: webhook-secret
status:
  availableReplicas: 1
  conditions:
    - lastTransitionTime: '2019-11-15T04:19:47Z'
      lastUpdateTime: '2019-11-15T04:19:47Z'
      message: Deployment has minimum availability.
      reason: MinimumReplicasAvailable
      status: 'True'
      type: Available
    - lastTransitionTime: '2019-07-02T23:44:59Z'
      lastUpdateTime: '2019-11-15T04:28:02Z'
      message: >-
        ReplicaSet "quay-openshift-registry-operator-7b7664fdd4" has
        successfully progressed.
      reason: NewReplicaSetAvailable
      status: 'True'
      type: Progressing
  observedGeneration: 8
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

Error logs from Operator Pod

8s.io/[email protected]/tools/cache/reflector.go:95: Failed to list *v1.Build: builds.build.openshift.io is forbidden: User "system:serviceaccount:quay-integration:quay-openshift-registry-operator" cannot list builds.build.openshift.io at the cluster scope: no RBAC policy matched
--
  | E1115 04:29:05.515616       1 reflector.go:134] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:95: Failed to list *v1.Build: builds.build.openshift.io is forbidden: User "system:serviceaccount:quay-integration:quay-openshift-registry-operator" cannot list builds.build.openshift.io at the cluster scope: no RBAC policy matched
  | E1115 04:29:06.518043       1 reflector.go:134] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:95: Failed to list *v1.Build: builds.build.openshift.io is forbidden: User "system:serviceaccount:quay-integration:quay-openshift-registry-operator" cannot list builds.build.openshift.io at the cluster scope: no RBAC policy matched
  | E1115 04:29:07.520485       1 reflector.go:134] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:95: Failed to list *v1.Build: builds.build.openshift.io is forbidden: User "system:serviceaccount:quay-integration:quay-openshift-registry-operator" cannot list builds.build.openshift.io at the cluster scope: no RBAC policy matched
  | E1115 04:29:08.522799       1 reflector.go:134] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:95: Failed to list *v1.Build: builds.build.openshift.io is forbidden: User "system:serviceaccount:quay-integration:quay-openshift-registry-operator" cannot list builds.build.openshift.io at the cluster scope: no RBAC policy matched

As a user, I want to modify MAIL_DEFAULT_SENDER config secret in CR.

spec:
  quay:
    config:
      mailDefaultSender: [email protected]

This should change "MAIL_DEFAULT_SENDER" in the config secret and trigger redploy.

Operator trips over provided super-user secret

When creating a superuser secret:
oc create secret generic quay-superuser-secret --from-literal=superuser-username=test --from-literal=superuser-password=test --from-literal=superuser-email=[email protected]

And specifying that in the Quay CR:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: QuayEcosystem
metadata:
name: quay
spec:
quay:
imagePullSecretName: quay-pull-secret
superuserCredentialsName: quay-superuser-secret
configSecretName: quay-config-secret
database:
volumeSize: 50Gi
registryStorage:
local:
persistentVolumeAccessMode:
- ReadWriteOnce
persistentVolumeSize: 50Gi

The following error occurs:
{"level":"info","ts":1564163861.5151386,"logger":"controller_quayecosystem","msg":"Reconciling QuayEcosystem","Request.Namespace":"quay","Request.Name":"quay"}
{"level":"error","ts":1564163861.5152035,"logger":"controller_quayecosystem","msg":"Secret Validation","Namespace":"quay","Name":"quay-superuser-secret","error":"Failed to validate provided secret with required parameters","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\ngithub.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/validation.validateSecret\n\t/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/validation/validate.go:198\ngithub.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/validation.Validate\n\t/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/validation/validate.go:25\ngithub.com/redhat-cop/quay-operator/pkg/controller/quayecosystem.(*ReconcileQuayEcosystem).Reconcile\n\t/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/quayecosystem_controller.go:127\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88"}

Also in the CR:
status:
conditions:
- lastTransitionTime: "2019-07-26T17:56:49Z"
lastUpdateTime: "2019-07-26T17:56:49Z"
message: Configuration Updated Successfully
status: "True"
type: UpdateDefaultConfigurationSuccess
- lastTransitionTime: "2019-07-26T17:56:49Z"
lastUpdateTime: "2019-07-26T17:57:41Z"
message: 'Failed to validate provided secret with required parameters. Namespace:
quay, Name: quay-superuser-secret'
reason: ProcessingError
status: "False"
type: QuayEcosystemValidationFailure

Add support for deploying Clair

Integration with Clair as part of the operator

Quay pod does not come up with v0.0.7 image - CrashLoopBackoff

I created a new quay instance (0.0.7 from the OpenShift Operator Hub - 4.2) and the quay pod was crash looping with the following error:
cp: cannot create regular file ‘/etc/pki/ca-trust/source/anchors/mitm.crt’: Permission denied

After applying the fix given in this issue #60:

oc adm policy remove-scc-from-user anyuid system:serviceaccount:quay-enterprise:default
The pod could progress to the running state.

Unable to use alternative Postgres image(s)

During testing of the RC 0.0.8, it was discovered that the operator does not work when using a Postgres 10 image.

Based upon the logs, this appears related to Quay's requirement to enable a specific extension on Postgres. It's also possible the tests were performed with Quay 3.1.x. It's possible that Quay 3.2 may resolve this issue.

According to those who discovered the issue, here is the CR used:

spec:
  quay:
    imagePullSecretName: redhat-pull-secret
    image: quay.io/quay/quay
    database:
      image: registry.access.redhat.com/rhscl/postgresql-10-rhel7

And here are the logs from the operator:

{"level":"info","ts":1566399380.128667,"logger":"controller_quayecosystem","msg":"Running exec into Pod","Reason":"","Pod Name":"quay-test2-quay-postgresql-b49994985-rx2nw"}
{"level":"info","ts":1566399380.2450473,"logger":"controller_quayecosystem","msg":"Error exec'ing into pod","Error":"command terminated with exit code 127","Command":["/bin/bash","-c","echo \"SELECT * FROM pg_available_extensions\" | /opt/rh/rh-postgresql96/root/usr/bin/psql -d quay"]}
{"level":"error","ts":1566399380.2450926,"logger":"controller_quayecosystem","msg":"Failed to Quay Setup Postgresql","error":"Failed to Exec into Postgresql Pod: ","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\ngithub.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/provisioning.(*ReconcileQuayEcosystemConfiguration).CoreQuayResourceDeployment\n\t/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/provisioning/provision.go:106\ngithub.com/redhat-cop/quay-operator/pkg/controller/quayecosystem.(*ReconcileQuayEcosystem).Reconcile\n\t/home/travis/gopath/src/github.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/quayecosystem_controller.go:136\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88"}

As this was the only image tested besides the default, and it would be practically impossible to support every Postgres version and image out there, it may be worth mentioning specifically which versions have been tested or are known to be supported.

Unable to use NooBaa as Storage Backend

During testing, it was discovered that NooBaa does not work using the 0.0.8 operator. It appears that it's possible some configuration data is missing when running the setup application.

Example CR

spec:
  quay:
    registryBackends:
      - name: rhocs
        rhocs:
          hostname: CENSORED_IP
          port: 80
          secure: false
          accessKey: CENSORED
          secretKey: CENSORED
          bucketName: quay

Operator Pod Log Entry

{"level":"error","ts":1576051727.7906559,"logger":"controller_quayecosystem","msg":"Failed to Validate Component","error":"registry-storage Validation Failed: Missing required parameter(s) for storage rhocs","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/tmp/quay-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/redhat-cop/quay-operator/pkg/controller/quayecosystem/setup.(*QuaySetupManager).SetupQuay\n\t/tmp/quay-operator/pkg/controller/quayecosystem/setup/setup.go:217\ngithub.com/redhat-cop/quay-operator/pkg/controller/quayecosystem.(*ReconcileQuayEcosystem).Reconcile\n\t/tmp/quay-operator/pkg/controller/quayecosystem/quayecosystem_controller.go:180\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/tmp/quay-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/tmp/quay-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/tmp/quay-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/tmp/quay-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/tmp/quay-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/tmp/quay-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

Config Tool Log Output

2019-12-11 08:08:33,952 [51] [ERROR] [util.config.validator] Validation exception
Traceback (most recent call last):
  File "/quay-registry/util/config/validator.py", line 78, in validate_service_for_config
    VALIDATORS[service](validator_context)
  File "/quay-registry/util/config/validators/validate_storage.py", line 18, in validate
    providers = _get_storage_providers(config, ip_resolver, config_provider).items()
  File "/quay-registry/util/config/validators/validate_storage.py", line 52, in _get_storage_providers
    raise ConfigValidationException('Missing required parameter(s) for storage %s' % name)
ConfigValidationException: Missing required parameter(s) for storage rhocs
gunicorn-config stdout | 2019-12-11 08:08:33,952 [51] [ERROR] [util.config.validator] Validation exception
Traceback (most recent call last):
  File "/quay-registry/util/config/validator.py", line 78, in validate_service_for_config
    VALIDATORS[service](validator_context)
  File "/quay-registry/util/config/validators/validate_storage.py", line 18, in validate
    providers = _get_storage_providers(config, ip_resolver, config_provider).items()
  File "/quay-registry/util/config/validators/validate_storage.py", line 52, in _get_storage_providers
    raise ConfigValidationException('Missing required parameter(s) for storage %s' % name)
ConfigValidationException: Missing required parameter(s) for storage rhocs
10.131.0.29 () - quayconfig [11/Dec/2019:08:08:33 +0000] "POST /api/v1/superuser/config/validate/registry-storage HTTP/1.1" 200 79 "-" "Go-http-client/1.1" (0.003 2517 0.003)

Add config option for authenication providers

The operator currently deploys Quay with the "Local Database" authentication provider. Enterprise customers are more likely to need to integrate with something like LDAP, so having that as an option would be very useful.

Create Helm chart

Creation of a Helm chart as another deployment mechanism

Create chart
Publish to general helm repository

Add support for requests and limits

The pods that this operator creates should have reasonable requests and limits defined for cpu & mem. This is especially important when deploying to clusters that have resource quotas configured, but no default limit ranges in place.

Create operatorhub.io Deployment Instructions

Deployment of the operator is available through operatorhub.io.

Add instructions that illustrate the steps to make use of this feature

quay / quay-operator Goto Github PK

quay-operator's Introduction

Quay Operator

Welcome

Principles

Getting Started

Step by step

Using the Operator

Component Container Images

Batteries-included, zero-config

Community

Contributing

Local Development

Prerequisites

quay-operator's People

Contributors

Stargazers

Watchers

Forkers

quay-operator's Issues

Recommend Projects

Recommend Topics

Recommend Org