InvalidAccessKeyId error during upgrade

[openshift]

InvalidAccessKeyId error during upgrade

[bparees]

possibly a problem w/ the credentials minter deleting the creds it gave us?

[sjenning]

I also see this

I0129 18:11:00.573658       1 controller.go:164] status changed: *v1.Config, Name=instance
E0129 18:11:00.578534       1 controller.go:208] unable to sync: unable to sync storage configuration: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
	status code: 403, request id: F8ECB95241C54568, host id: IDV35DQ0oIdFJSAFvjenkEs3ZcxP4asYQbrk3kIRNJ+zfmpgW3Er8ktP1aOeSl2yfsS6OQPuMfQ=, requeuing
I0129 18:11:00.596131       1 generator.go:202] object *v1.Secret, Namespace=openshift-image-registry, Name=image-registry-private-configuration updated
I0129 18:11:00.608203       1 generator.go:202] object *v1.Deployment, Namespace=openshift-image-registry, Name=image-registry updated
I0129 18:11:00.734679       1 controller.go:164] status changed: *v1.Config, Name=instance
E0129 18:11:00.738416       1 controller.go:208] unable to sync: Operation cannot be fulfilled on configs.imageregistry.operator.openshift.io "instance": the object has been modified; please apply your changes to the latest version and try again, requeuing
I0129 18:11:00.872163       1 controller.go:164] status changed: *v1.Config, Name=instance
E0129 18:11:00.877541       1 controller.go:208] unable to sync: unable to sync storage configuration: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
	status code: 403, request id: BE0BAD777BE1EA96, host id: R8OFykmJu9yf6/7G5vOo+2SKIEELBodQngpdozXReirKkIeUxvEBwo1iH1QSanFQJeavjWRQW4k=, requeuing
I0129 18:11:00.987299       1 controller.go:164] status changed: *v1.Config, Name=instance
E0129 18:11:00.993051       1 controller.go:208] unable to sync: unable to sync storage configuration: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
	status code: 403, request id: 158C68317E9EE0F9, host id: E8xuVikhZQ8IyUqeHq9LZnQTH5UZeKI7kNlTYKEoMst5JsIMkpevTI2DG0HO7ZVnvbAm6Y+BuMg=, requeuing
I0129 18:11:01.088572       1 controller.go:164] status changed: *v1.Config, Name=instance
E0129 18:11:01.094191       1 controller.go:208] unable to sync: unable to sync storage configuration: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
	status code: 403, request id: 2BE70688DDA94E7E, host id: rXSfzYq5leTwZHz52/9yw5r5yY+MolKHOZveMCNk8goAFXSMx2tT2vpWGQJy9GeovpE+g0igdpg=, requeuing
I0129 18:11:01.216007       1 controller.go:164] status changed: *v1.Config, Name=instance
E0129 18:11:01.223144       1 controller.go:208] unable to sync: unable to sync storage configuration: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
	status code: 403, request id: CEBCBB396AEC69DE, host id: KT+As5wH/gCn6xCYq4rNUR0UbcAGmHU+BJX3AtHiKb5ifKBZb6YRjOkKNNTYoD2dSYxS7noN/xA=, requeuing
I0129 18:11:01.336485       1 controller.go:164] status changed: *v1.Config, Name=instance
E0129 18:11:01.342833       1 controller.go:208] unable to sync: unable to sync storage configuration: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
	status code: 403, request id: FF834DE22D55CA90, host id: 5Xm0iTRYHy1Pz+AwHrND+4orbtQWfCsehvvCPn+th0LChX4Q0n1Ptq5QakjO7hKmkcZwEDP2odE=, requeuing
I0129 18:11:01.444585       1 controller.go:164] status changed: *v1.Config, Name=instance
E0129 18:11:01.450342       1 controller.go:208] unable to sync: unable to sync storage configuration: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
	status code: 403, request id: 53CF4D9F10F665FC, host id: 3ObZbXAJFo4gGOgveAhHBOFQQwOQdRKe57zYx5499eu2Fg56CRRVOVil2X0VMpdDMOyF90jvJ/k=, requeuing

in a loop

resulting in deployments being created and deleted continuously

[smarterclayton]

hot looping bad1!!!!!!

[bparees]

so @coreydaley there are two issues:

1. why did the key become invalid (was the existing key removed during the upgrade? was the key changed as part of the upgrade and the new key isn't valid?) Guessing this is related to credential minting.

2. why are we hotlooping on the access failure.

[coreydaley]

@sjenning What does your config look like?
oc get configs.imageregistry.operator.openshift.io -o yaml

[sjenning]

$ oc get configs.imageregistry.operator.openshift.io -o yaml
apiVersion: v1
items:
- apiVersion: imageregistry.operator.openshift.io/v1
  kind: Config
  metadata:
    creationTimestamp: 2019-01-29T19:45:49Z
    finalizers:
    - imageregistry.operator.openshift.io/finalizer
    generation: 1
    name: instance
    resourceVersion: "40141"
    selfLink: /apis/imageregistry.operator.openshift.io/v1/configs/instance
    uid: 7a363ed8-23fe-11e9-92ea-06243668d6d2
  spec:
    httpSecret: <redacted>
    logging: 2
    managementState: Managed
    proxy: {}
    replicas: 1
    requests:
      read: {}
      write: {}
    storage:
      s3:
        bucket: image-registry-us-west-1-8b812d0a179542ff852fa50211c47598-7a72
        region: us-west-1
  status:
    conditions:
    - lastTransitionTime: 2019-01-29T19:46:23Z
      message: Deployment has minimum availability
      status: "True"
      type: Available
    - lastTransitionTime: 2019-01-29T20:07:43Z
      message: "Unable to apply resources: unable to sync storage configuration: InvalidAccessKeyId:
        The AWS Access Key Id you provided does not exist in our records.\n\tstatus
        code: 403, request id: <redacted>, host id: <redacted>"
      status: "True"
      type: Progressing
    - lastTransitionTime: 2019-01-29T19:45:49Z
      status: "False"
      type: Failing
    - lastTransitionTime: 2019-01-29T19:45:49Z
      status: "False"
      type: Removed
    - lastTransitionTime: 2019-01-29T20:16:46Z
      message: "InvalidAccessKeyId: The AWS Access Key Id you provided does not exist
        in our records.\n\tstatus code: 403, request id: <redacted>, host id:
        <redacted>"
      reason: InvalidAccessKeyId
      status: "False"
      type: StorageExists
    - lastTransitionTime: 2019-01-29T19:45:57Z
      message: UserTags were successfully applied to the S3 bucket
      reason: Tagging Successful
      status: "True"
      type: StorageTagged
    - lastTransitionTime: 2019-01-29T19:45:57Z
      message: Default encryption was successfully enabled on the S3 bucket
      reason: Encryption Successful
      status: "True"
      type: StorageEncrypted
    - lastTransitionTime: 2019-01-29T19:45:57Z
      message: Default cleanup of incomplete multipart uploads after one (1) day was
        successfully enabled
      reason: Enable Cleanup Successful
      status: "True"
      type: StorageIncompleteUploadCleanupEnabled
    generations: null
    internalRegistryHostname: image-registry.openshift-image-registry.svc:5000
    observedGeneration: 1
    readyReplicas: 0
    storage:
      s3:
        bucket: image-registry-us-west-1-8b812d0a179542ff852fa50211c47598-7a72
        region: us-west-1
    storageManaged: true
    version: ""
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

[sjenning]

also, something is hotloop creating the installer-cloud-credentials secret

a ~20s watch:

$ oc get secret -w
NAME                                              TYPE                                  DATA   AGE
builder-dockercfg-9cwg8                           kubernetes.io/dockercfg               1      37m
builder-token-48h2v                               kubernetes.io/service-account-token   3      38m
builder-token-6wlxb                               kubernetes.io/service-account-token   3      38m
cluster-image-registry-operator-dockercfg-l2fxw   kubernetes.io/dockercfg               1      37m
cluster-image-registry-operator-token-d2v7l       kubernetes.io/service-account-token   3      38m
cluster-image-registry-operator-token-t9nhg       kubernetes.io/service-account-token   3      38m
default-dockercfg-wr89c                           kubernetes.io/dockercfg               1      37m
default-token-4pfhm                               kubernetes.io/service-account-token   3      38m
default-token-tl4l7                               kubernetes.io/service-account-token   3      38m
deployer-dockercfg-bc8tx                          kubernetes.io/dockercfg               1      37m
deployer-token-fc77j                              kubernetes.io/service-account-token   3      38m
deployer-token-fz74s                              kubernetes.io/service-account-token   3      38m
image-registry-private-configuration              Opaque                                2      38m
image-registry-tls                                kubernetes.io/tls                     2      38m
node-ca-dockercfg-tbshp                           kubernetes.io/dockercfg               1      37m
node-ca-token-m284n                               kubernetes.io/service-account-token   3      38m
node-ca-token-vvhbd                               kubernetes.io/service-account-token   3      38m
registry-dockercfg-zkh7p                          kubernetes.io/dockercfg               1      37m
registry-token-7wn7f                              kubernetes.io/service-account-token   3      38m
registry-token-dsqd4                              kubernetes.io/service-account-token   3      37m
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
image-registry-private-configuration   Opaque   2     38m
installer-cloud-credentials   Opaque   2     2s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
image-registry-private-configuration   Opaque   2     38m
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s
installer-cloud-credentials   Opaque   2     0s

[bparees]

/cc @dgoodwin

sounds like all of this is caused by bad minted creds (and/or creds being updated when they should not be)

[coreydaley]

It looks like the initial set of credentials was correct and the storage was created, tagged, etc, then on the resync they were incorrect (according to the config & conditions)

[dgoodwin]

The cred minter logs show successfully created the secret, and then on the next sync it says the secret does not exist. Could anything else be deleting that secret?

[coreydaley]

@dgoodwin I'm checking to see if anything in the operator might be deleting it.

[dgoodwin]

From the logs I got it appears it's happening for all three creds the operator manages, so unlikely anything in the operator. Will be filing an issue this morning with everything I can come up with. I don't immediately know how to fix a hotloop that creates a secret, resyncs, and then can't see the secret it created, which appears to be what happened. I am curious if anything else was in trouble in the cluster post upgrade.

[bparees]

@dgoodwin ok, please link the issue you open once you do so, i intend to close this one in deference to that.

[dgoodwin]

openshift/cloud-credential-operator#24