Comments (14)
possibly a problem w/ the credentials minter deleting the creds it gave us?
from cluster-image-registry-operator.
I also see this
I0129 18:11:00.573658 1 controller.go:164] status changed: *v1.Config, Name=instance
E0129 18:11:00.578534 1 controller.go:208] unable to sync: unable to sync storage configuration: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
status code: 403, request id: F8ECB95241C54568, host id: IDV35DQ0oIdFJSAFvjenkEs3ZcxP4asYQbrk3kIRNJ+zfmpgW3Er8ktP1aOeSl2yfsS6OQPuMfQ=, requeuing
I0129 18:11:00.596131 1 generator.go:202] object *v1.Secret, Namespace=openshift-image-registry, Name=image-registry-private-configuration updated
I0129 18:11:00.608203 1 generator.go:202] object *v1.Deployment, Namespace=openshift-image-registry, Name=image-registry updated
I0129 18:11:00.734679 1 controller.go:164] status changed: *v1.Config, Name=instance
E0129 18:11:00.738416 1 controller.go:208] unable to sync: Operation cannot be fulfilled on configs.imageregistry.operator.openshift.io "instance": the object has been modified; please apply your changes to the latest version and try again, requeuing
I0129 18:11:00.872163 1 controller.go:164] status changed: *v1.Config, Name=instance
E0129 18:11:00.877541 1 controller.go:208] unable to sync: unable to sync storage configuration: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
status code: 403, request id: BE0BAD777BE1EA96, host id: R8OFykmJu9yf6/7G5vOo+2SKIEELBodQngpdozXReirKkIeUxvEBwo1iH1QSanFQJeavjWRQW4k=, requeuing
I0129 18:11:00.987299 1 controller.go:164] status changed: *v1.Config, Name=instance
E0129 18:11:00.993051 1 controller.go:208] unable to sync: unable to sync storage configuration: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
status code: 403, request id: 158C68317E9EE0F9, host id: E8xuVikhZQ8IyUqeHq9LZnQTH5UZeKI7kNlTYKEoMst5JsIMkpevTI2DG0HO7ZVnvbAm6Y+BuMg=, requeuing
I0129 18:11:01.088572 1 controller.go:164] status changed: *v1.Config, Name=instance
E0129 18:11:01.094191 1 controller.go:208] unable to sync: unable to sync storage configuration: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
status code: 403, request id: 2BE70688DDA94E7E, host id: rXSfzYq5leTwZHz52/9yw5r5yY+MolKHOZveMCNk8goAFXSMx2tT2vpWGQJy9GeovpE+g0igdpg=, requeuing
I0129 18:11:01.216007 1 controller.go:164] status changed: *v1.Config, Name=instance
E0129 18:11:01.223144 1 controller.go:208] unable to sync: unable to sync storage configuration: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
status code: 403, request id: CEBCBB396AEC69DE, host id: KT+As5wH/gCn6xCYq4rNUR0UbcAGmHU+BJX3AtHiKb5ifKBZb6YRjOkKNNTYoD2dSYxS7noN/xA=, requeuing
I0129 18:11:01.336485 1 controller.go:164] status changed: *v1.Config, Name=instance
E0129 18:11:01.342833 1 controller.go:208] unable to sync: unable to sync storage configuration: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
status code: 403, request id: FF834DE22D55CA90, host id: 5Xm0iTRYHy1Pz+AwHrND+4orbtQWfCsehvvCPn+th0LChX4Q0n1Ptq5QakjO7hKmkcZwEDP2odE=, requeuing
I0129 18:11:01.444585 1 controller.go:164] status changed: *v1.Config, Name=instance
E0129 18:11:01.450342 1 controller.go:208] unable to sync: unable to sync storage configuration: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
status code: 403, request id: 53CF4D9F10F665FC, host id: 3ObZbXAJFo4gGOgveAhHBOFQQwOQdRKe57zYx5499eu2Fg56CRRVOVil2X0VMpdDMOyF90jvJ/k=, requeuing
in a loop
resulting in deployments being created and deleted continuously
from cluster-image-registry-operator.
hot looping bad1!!!!!!
from cluster-image-registry-operator.
so @coreydaley there are two issues:
-
why did the key become invalid (was the existing key removed during the upgrade? was the key changed as part of the upgrade and the new key isn't valid?) Guessing this is related to credential minting.
-
why are we hotlooping on the access failure.
from cluster-image-registry-operator.
@sjenning What does your config look like?
oc get configs.imageregistry.operator.openshift.io -o yaml
from cluster-image-registry-operator.
$ oc get configs.imageregistry.operator.openshift.io -o yaml
apiVersion: v1
items:
- apiVersion: imageregistry.operator.openshift.io/v1
kind: Config
metadata:
creationTimestamp: 2019-01-29T19:45:49Z
finalizers:
- imageregistry.operator.openshift.io/finalizer
generation: 1
name: instance
resourceVersion: "40141"
selfLink: /apis/imageregistry.operator.openshift.io/v1/configs/instance
uid: 7a363ed8-23fe-11e9-92ea-06243668d6d2
spec:
httpSecret: <redacted>
logging: 2
managementState: Managed
proxy: {}
replicas: 1
requests:
read: {}
write: {}
storage:
s3:
bucket: image-registry-us-west-1-8b812d0a179542ff852fa50211c47598-7a72
region: us-west-1
status:
conditions:
- lastTransitionTime: 2019-01-29T19:46:23Z
message: Deployment has minimum availability
status: "True"
type: Available
- lastTransitionTime: 2019-01-29T20:07:43Z
message: "Unable to apply resources: unable to sync storage configuration: InvalidAccessKeyId:
The AWS Access Key Id you provided does not exist in our records.\n\tstatus
code: 403, request id: <redacted>, host id: <redacted>"
status: "True"
type: Progressing
- lastTransitionTime: 2019-01-29T19:45:49Z
status: "False"
type: Failing
- lastTransitionTime: 2019-01-29T19:45:49Z
status: "False"
type: Removed
- lastTransitionTime: 2019-01-29T20:16:46Z
message: "InvalidAccessKeyId: The AWS Access Key Id you provided does not exist
in our records.\n\tstatus code: 403, request id: <redacted>, host id:
<redacted>"
reason: InvalidAccessKeyId
status: "False"
type: StorageExists
- lastTransitionTime: 2019-01-29T19:45:57Z
message: UserTags were successfully applied to the S3 bucket
reason: Tagging Successful
status: "True"
type: StorageTagged
- lastTransitionTime: 2019-01-29T19:45:57Z
message: Default encryption was successfully enabled on the S3 bucket
reason: Encryption Successful
status: "True"
type: StorageEncrypted
- lastTransitionTime: 2019-01-29T19:45:57Z
message: Default cleanup of incomplete multipart uploads after one (1) day was
successfully enabled
reason: Enable Cleanup Successful
status: "True"
type: StorageIncompleteUploadCleanupEnabled
generations: null
internalRegistryHostname: image-registry.openshift-image-registry.svc:5000
observedGeneration: 1
readyReplicas: 0
storage:
s3:
bucket: image-registry-us-west-1-8b812d0a179542ff852fa50211c47598-7a72
region: us-west-1
storageManaged: true
version: ""
kind: List
metadata:
resourceVersion: ""
selfLink: ""
from cluster-image-registry-operator.
also, something is hotloop creating the installer-cloud-credentials
secret
a ~20s watch:
$ oc get secret -w
NAME TYPE DATA AGE
builder-dockercfg-9cwg8 kubernetes.io/dockercfg 1 37m
builder-token-48h2v kubernetes.io/service-account-token 3 38m
builder-token-6wlxb kubernetes.io/service-account-token 3 38m
cluster-image-registry-operator-dockercfg-l2fxw kubernetes.io/dockercfg 1 37m
cluster-image-registry-operator-token-d2v7l kubernetes.io/service-account-token 3 38m
cluster-image-registry-operator-token-t9nhg kubernetes.io/service-account-token 3 38m
default-dockercfg-wr89c kubernetes.io/dockercfg 1 37m
default-token-4pfhm kubernetes.io/service-account-token 3 38m
default-token-tl4l7 kubernetes.io/service-account-token 3 38m
deployer-dockercfg-bc8tx kubernetes.io/dockercfg 1 37m
deployer-token-fc77j kubernetes.io/service-account-token 3 38m
deployer-token-fz74s kubernetes.io/service-account-token 3 38m
image-registry-private-configuration Opaque 2 38m
image-registry-tls kubernetes.io/tls 2 38m
node-ca-dockercfg-tbshp kubernetes.io/dockercfg 1 37m
node-ca-token-m284n kubernetes.io/service-account-token 3 38m
node-ca-token-vvhbd kubernetes.io/service-account-token 3 38m
registry-dockercfg-zkh7p kubernetes.io/dockercfg 1 37m
registry-token-7wn7f kubernetes.io/service-account-token 3 38m
registry-token-dsqd4 kubernetes.io/service-account-token 3 37m
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
image-registry-private-configuration Opaque 2 38m
installer-cloud-credentials Opaque 2 2s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
image-registry-private-configuration Opaque 2 38m
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
installer-cloud-credentials Opaque 2 0s
from cluster-image-registry-operator.
/cc @dgoodwin
sounds like all of this is caused by bad minted creds (and/or creds being updated when they should not be)
from cluster-image-registry-operator.
It looks like the initial set of credentials was correct and the storage was created, tagged, etc, then on the resync they were incorrect (according to the config & conditions)
from cluster-image-registry-operator.
The cred minter logs show successfully created the secret, and then on the next sync it says the secret does not exist. Could anything else be deleting that secret?
from cluster-image-registry-operator.
@dgoodwin I'm checking to see if anything in the operator might be deleting it.
from cluster-image-registry-operator.
From the logs I got it appears it's happening for all three creds the operator manages, so unlikely anything in the operator. Will be filing an issue this morning with everything I can come up with. I don't immediately know how to fix a hotloop that creates a secret, resyncs, and then can't see the secret it created, which appears to be what happened. I am curious if anything else was in trouble in the cluster post upgrade.
from cluster-image-registry-operator.
@dgoodwin ok, please link the issue you open once you do so, i intend to close this one in deference to that.
from cluster-image-registry-operator.
openshift/cloud-credential-operator#24
from cluster-image-registry-operator.
Related Issues (20)
- Operartor doesn't retry create pods with status "CreateContainerError" HOT 4
- Future Release Branches Frozen For Merging | branch:release-4.16 branch:release-4.17
- Setup golangci-lint
- Default to Removed for bare metal infrastructure platform HOT 2
- RFE - Provide a 'storage' option that allows me to set the specific StorageClass for the registry HOT 13
- Disable docker redirect for S3 buckets HOT 4
- AdditionalTrustedBundle not being added to the operator pod HOT 7
- Is this the expected way to go to set up (vendor) swift storage for the registry? HOT 11
- Upgrade error: Unable to apply 4.2.16: the cluster operator image-registry has not yet successfully rolled out HOT 12
- Support for third party S3 implementations HOT 6
- Mounting additionalTrustBundle in the operator. HOT 13
- Pull from ImageStream in Dockerfile - Authentication Required HOT 7
- The s3 storage backend leaks a new https connection every 10 seconds HOT 8
- Question: Is it possible to have more than one instance of the operator deployed in a cluster? HOT 3
- Future Release Branches Frozen For Merging | branch:release-4.18 branch:release-4.19
- world-writable without sticky bit in `/etc/pki/ca-trust/extracted/edk2` etc HOT 5
- Third Party S3 configuration results in runtime panic HOT 9
- Empty IBMCOS storage config causes operator to crashloop HOT 3
- Route Through New Ingress Controller HOT 4
- Helm chart for cluster-image-registry-operator HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cluster-image-registry-operator.