Deployment failed - Unable to retrieve pull secret, the image pull may not succeed - redhat-operators-pull-secret

[kurokobo]

Deployment failed - Unable to retrieve pull secret, the image pull may not succeed - redhat-operators-pull-secret

[unixsysadmin]

Thanks @kurokobo - sorry it took me a little longer than expected to create this.

I have raised ansible/awx-operator#922

[kurokobo]

@unixsysadmin
Thanks for filling the issue.

The install failed with:
May 13 13:47:19 XXX k3s[27942]: I0513 13:47:19.715551 27942 kubelet_pods.go:891] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-675865446d-9nh27" secret="" err="secret \"redhat-operators-pull-secret\" not found"

I've never noticed this, but I found this error was recorded on /var/log/messages in my environment too.

$ sudo tail -f /var/log/messages
...
May 14 01:31:34 kuro-awx01 k3s[2268919]: I0514 01:31:34.236997 2268919 kubelet_pods.go:891] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-675865446d-lk6jq" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May 14 01:31:35 kuro-awx01 k3s[2268919]: I0514 01:31:35.238427 2268919 kubelet_pods.go:891] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-675865446d-lk6jq" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May 14 01:31:36 kuro-awx01 k3s[2268919]: I0514 01:31:36.240740 2268919 kubelet_pods.go:891] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-675865446d-lk6jq" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May 14 01:31:40 kuro-awx01 k3s[2268919]: I0514 01:31:40.178857 2268919 kubelet_pods.go:891] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-675865446d-lk6jq" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May 14 01:32:48 kuro-awx01 k3s[2268919]: I0514 01:32:48.971593 2268919 kubelet_pods.go:891] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-675865446d-lk6jq" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May 14 01:33:52 kuro-awx01 k3s[2268919]: I0514 01:33:52.970909 2268919 kubelet_pods.go:891] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-675865446d-lk6jq" secret="" err="secret \"redhat-operators-pull-secret\" not found"

However, technically, an container image that published on public registry can be pulled even if the specified imagePullSecret is not present.

Thus, in fact, the AWX Operator deployment is successful and the pod is Running state in my environment. I suspect that your pod is also Running in your environment.

$ kubectl -n awx get all
NAME                                                   READY   STATUS    RESTARTS   AGE
pod/awx-operator-controller-manager-675865446d-lk6jq   2/2     Running   0          9m13s

NAME                                                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/awx-operator-controller-manager-metrics-service   ClusterIP   10.43.153.83   <none>        8443/TCP   9m13s

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/awx-operator-controller-manager   1/1     1            1           9m13s

NAME                                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/awx-operator-controller-manager-675865446d   1         1         1       9m13s

If your pod is not in Running state, I guess the cause is something else. Check the Events of your pod by invoking kubectl -n awx describe pod <your operator's pod name>.

[kurokobo]

Even if your pod is in Running state, I agree that the error message is a bit annoying. It's recorded almost every minute.

Since this is not an issue for my guide but for the AWX Operator implementation, so could you please create new issue on the AWX Operator's repo?

Just as a workaround, you can create a dummy secret with name redhat-operators-pull-secret to suppress the error by:

kubectl -n awx create secret docker-registry redhat-operators-pull-secret \
  --docker-server=dummy.example.com \
  --docker-username=dummy \
  --docker-password=dummy

[kurokobo]

I've modified contents of base directory via #75 so that secret redhat-operators-pull-secret is created at the same time as AWX.

The errors that are logged that you mentioned immediately after the AWX Operator is deployed, but you can ignore the errors. If you deploy AWX using the latest base, the errors should disappear.

[unixsysadmin]

Hi @kurokobo

Thanks again for looking at this and for making the change.

I confirm that I now only briefly see the warning about redhat-operators-pull-secret which happens on a clean installation when I first install the AWX Operator.

May 16 14:13:00 XXX k3s[26138]: I0516 14:13:00.442021   26138 kubelet_pods.go:891] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-675865446d-qv8xk" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May 16 14:13:29 XXX k3s[26138]: I0516 14:13:29.621409   26138 kubelet_pods.go:891] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-675865446d-qv8xk" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May 16 14:13:30 XXX k3s[26138]: I0516 14:13:30.624170   26138 kubelet_pods.go:891] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-675865446d-qv8xk" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May 16 14:13:40 XXX k3s[26138]: I0516 14:13:40.143647   26138 kubelet_pods.go:891] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-675865446d-qv8xk" secret="" err="secret \"redhat-operators-pull-secret\" not found"

When I subsequently run kubectl apply -k base the message stops appearing.

You were entirely correct - the error would not have stopped the deployment from succeeding. I discovered that my Docker Hub credentials were incorrect and prevented a successful deployment.

I've tested again following your latest changes on a new installation and all works well.

Since this is not an issue for my guide but for the AWX Operator implementation, so could you please create new issue on the AWX Operator's repo?

I'm happy to do this, would you be able to give me some hints as to how to phrase this - would this be correct:

The default installation on the AWX Operator makes reference to redhat-operators-pull-secret. A container image that published on public registry can be pulled even if the specified imagePullSecret is not present so the warning messages about redhat-operators-pull-secret being not found are not necessary.

Is the logging of the secret not found something that the Operator is responsible for or it that built in to K3S?

[kurokobo]

@unixsysadmin
Thanks for confirming the issue is solved with my latest change!

Is the logging of the secret not found something that the Operator is responsible for or it that built in to K3S?

That error log is from K3s, not AWX Operator.

Therefore, rather than saying "error messages are not necessary", it would be more appropriate to say "non-existent secrets should not be referenced".

How about this, for example?

The default installation on the AWX Operator makes reference to redhat-operators-pull-secret.
However, this secret is never created during the deployment instructions. Therefore, K3s will log the error stating that the secret cannot be found and the errors continue to be logged almost every minute during lifetime.
The reference to non-existent secret should be removed.

I will be able to leave additional comment on the issue you will have created, so feel free to filling new issue on AWX Operator's repo :)

[kurokobo]

@unixsysadmin
Thanks for creating new issue on that repository!

I'm closing this issue at this time. I'll update my guide once the ansible/awx-operator#922 will be solved.