Private ECR repositories give '401 Unauthorized'

[sbkg0002]

Private ECR repositories give '401 Unauthorized'

[ckotzbauer]

Small summary from slack:

• The AWS-Cloud-Provider-Manager can handle IAM-Role-Credentials at the kubelet (but not accessible from the API): https://cloud-provider-aws.sigs.k8s.io/#aws-credential-provider https://github.com/kubernetes/enhancements/tree/master/keps/sig-cloud-provider/2133-out-of-tree-credential-provider
• Operator doesn't know anything about that, only respecting imagePullSecrets on pod-level right now.
• Solution: Adding a new cli-flag --fallback-image-pull-secret which is used by the operator, when no imagePullSecrets are set on pod-level. In the future this simple fallback could be improved with constraints if needed (e.g. only apply on specific pod-labels, ...)

Alternative: Login to ECR ourselves via http://169.254.169.254/latest/meta-data/iam/ to fetch the needed token. This would result in a deeper integration with different cloud-vendors which is (currently) not scope of this project.

[sbkg0002]

Awesome work! I ran a dev build last week and that was working fine already. I will upgrade asap.

Thanks again and please tell me how we can send you a beer/coffee.

[ckotzbauer]

Great, thanks for your quick positive feedback, I really appreciate that!! 🎉
There's a Github Sponsor button, if you want to send me a beer or coffe 😉

[sbkg0002]

Tested the 0.12.0 release this morning; it seems to work great!

[ckotzbauer]

Hi @sbkg0002,
is the auth-token still valid? So can Kubernetes still pull images with this token or are the containers of these images already created some time ago?
I know that ECR-Tokens have a short lifetime (I think a day or so) and have to be renewed periodically.

/kind question

[sbkg0002]

Hi @ckotzbauer, thanks for the quick reponse.

How can I verify that without logging into the node?
Starting a pod and pulling the instance metadata, everything seems fine.

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<rolename> | jq .Expiration
"2022-04-20T18:42:46Z"
bash-4.2# date
Wed Apr 20 12:30:12 UTC 2022

Starting new pods with images in ECR are just fine. 🤔

EDIT:
From within a simple alpine pod, running aws ecr describe-images --repository-name <repo> --registry-id <accountid> --region eu-west-1 on one of the failing repos, works just fine.

[ckotzbauer]

hm, okay. You have to decode the content from the Kubernetes-Secret which is referenced as imagePullSecret from a container-image which could not be analyzed from this operator.
The secret-content should be usable as docker's config.json which is stored at ~/.docker/config.json on your local machine. So you can try to save the file and than do a docker pull for the image of the container. If this works, the token is valid.

This ECR repo is provided by AWS and should be available for everyone.

But it's still a private image-repository right? If it's public then the token should be irrelevant. If that's the case, are there pullSecrets configured for this pod anyway?

[sbkg0002]

hm, okay. You have to decode the content from the Kubernetes-Secret which is referenced as imagePullSecret from a container-image which could not be analyzed from this operator.
The secret-content should be usable as docker's config.json which is stored at ~/.docker/config.json on your local machine. So you can try to save the file and than do a docker pull for the image of the container. If this works, the token is valid.

There are no imagePullSecrets defined since the instance roles have access to read from the ECR repositories. The kubelet also pulls all the images this way. I guess google/go-containerregistry doesn't support this?

EDIT: Adding a valid (fresh) imagePullSecret to the serviceaccount of the operator - and restarting it - doesn't change anything.

But it's still a private image-repository right? If it's public then the token should be irrelevant. If that's the case, are there pullSecrets configured for this pod anyway?

You are completely correct; you still need to be AWS authenticated in order to pull from it.

[ckotzbauer]

I think this doc covers this auth-behaviour: https://github.com/kubernetes/kops/blob/master/docs/iam_roles.md#access-to-aws-ec2-container-registry-ecr (I personally don't know much about ECR and EKS)

No, instance-roles which are respected from the kubelet are not supported. The operator uses the imagePullSecrets of a pod to authenticate against the registry. If there are no credentials, the operator will pull the image without auth.

I don't know if its easy possible to add this feature (as the operator-pod is not running on the ec2-host which has the role assigned).

EDIT: Adding a valid (fresh) imagePullSecret to the serviceaccount of the operator - and restarting it - doesn't change anything.

The imagePullSecret has to be defined at pod-level from the pods you want to analyze.

[ckotzbauer]

Do you know, if this eks-auth-feature with Instance-Roles is a native Kubernetes-Feature or a EKS-Addon?

[ckotzbauer]

@sbkg0002 Can you please update to 0.12.0 and add the --fallback-image-pull-secret parameter? The secret has to exist in the namespace of the operator.
From this release on, all imagePullSecrets configured for a pod are used for authentication. If none of them succeeds, the optional fallback-secret is tried.