trondhindenes/ecr-updater

Allows use of AWS ECR registries with Kubernetes no matter where your cluster is running.

K8s-Ecrupdater

Because of the way AWS ECR docker registries work, the credentials have to be refreshed.
This image pulls credentials from ECR every hour, and injects them into Kubernetes so that images can be pulled from a private repo.

This allows the use of AWS ECR registries also when your Kubernetes cluster is running in another cloud provider, or you don't want to set up EC2 roles for it.

Grab container images from: https://cloud.docker.com/u/trondhindenes/repository/docker/trondhindenes/k8s-ecrupdater

Configuration

Configure with the following environment variables:

K8S_PULL_SECRET_NAME: Name of the Kubernetes pull secret to update   
ECR_UPDATE_INTERVAL: (optional, time in seconds)
ECR_CREATE_MISSING: if this envvar is set to `true`, missing pull secrets will be created in all namespaces
(there's a good chance this will fail on older (pre 1.11) clusters.   
AWS_DEFAULT_REGION: (set to your region)   
AWS_ACCESS_KEY_ID: aws creds   
AWS_SECRET_ACCESS_KEY: aws creds
LOG_CONFIG: optional log config file (defaults to ./logging.json)   
LOG_LEVEL: Applies only if no log config file has been found (defaults to INFO)   

Note that if you're using alternate methods of providing the pod with AWS credentials (such as kube2iam or similar) you can skip the AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY configuration items.

Example deployment

It is assumed that you already have ECR setup, an IAM user with access to it, and that you have kubectl configured to communicate with your Kubernetes cluster.

You can also run it locally using kubectl proxy on your computer if you want to test things out. In that case, make sure the proxy listens on localhost:8001

1. (this step is only required if ECR_CREATE_MISSING is not set to true) Create a secret called ecr. This is the secret that this pod will update regularly. It doesn't matter what you put in here, as ecrupdater will update it, it just needs to exist.: kubectl create secret docker-registry ecr --docker-username=smash --docker-password=lol --docker-email [email protected]
   NOTE: ecrupdater will look for the secrets with the specified name across all your namespaces if you're using the authorization template below. So in this example any secret named ecr across all namespaces will be updated. If you want to separate them you can run multiple instances of ecrupdater, optionally with tighter (namespaces-isolated) security.

2. Create the authorization stuff that lets kubectl-proxy (running in the same pod as the ecr-updater) interact with kubernetes: kubectl apply -f example_deployment/01_authorization.yml

3. Create a IAM user that has read access to your registries. The access key and secret key need to be base64-encoded (remember to use the -n option):
   echo -n "PUT_ACCESSKEY_HERE" | base64
echo -n "PUT_SECRETKEY_HERE" | base64
   Put this info in the file example_deployment/01_aws_credentials.yml.yml in this repo.
   Now you can create a secret that will hold this info. This is how the ecr updater will log on to AWS:
   kubectl apply -f example_deployment/01_aws_credentials.yml

4. Deploy the pod. This contains both the ecr-updater and a "sidecar" container running kubectl-proxy. The proxy allows communication with the kubernetes api in a simple manner. Make sure to set your correct aws region in example_deployment/02_deployment.yml before deploying! kubectl apply -f example_deployment/02_deployment.yml

5. Test a deployment. Replace the containerimage with one from your own ecr registry, deploy it and prosper! (note that the ecrupdater initially pauses for 60 seconds, so make sure time has passed between the ecr updater pod coming online, and you run the next command) kubectl apply -f example_deployment/03_pullsecret_test.yml

Logging

By default, the service logs using the built in python logging package and the python-json-logger package for the json formatting. In order to change the log configuration, the service expects a log config file 'logging.json' in the working directory. Alternatively, this log config file path can be configured via the ENV variable LOG_CONFIG using a configMap.

Logger config docs

• https://docs.python.org/3/library/logging.config.html

example log config file to enable JSON logging on INFO level.

{
  "version": 1,
  "disable_existing_loggers": false,
  "formatters": {
    "json": {
      "format": "%(asctime)s %(levelname)s %(message)s %(pathname)s %(lineno)d %(threadName)s",
      "class": "pythonjsonlogger.jsonlogger.JsonFormatter"
    }
  },
  "handlers": {
    "json": {
      "class": "logging.StreamHandler",
      "formatter": "json"
    }
  },
  "loggers": {
    "": {
      "handlers": ["json"],
      "level": "INFO"
    }
  }
}

ecr-updater's Issues

Found 0 registry_secrets matching name ecr

Hi, I am trying to configure my cluster to use ecr and I get this error. What am I doing wrong?

Support JSON logging

In order to be able to collect the tool logs, it should support json logging output.

ECR Updater in a separate namespace - Secrets is forbidden

Hi, I'm quite new to k8s, so maybe just general guidance needed. I'm trying to run ecr-updater under a separate namespace, but I can't make it work. It works perfectly when I launch it in the default namespace tho.

starting the thing
Running update loop
starting update loop
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '028d6d8b-cddf-45e0-aaad-2a3f78c75d5c', 'Cache-Control': 'no-cache, private', 'Content-Length': '283', 'Content-Type': 'application/json', 'Date': 'Thu, 04 Mar 2021 08:32:58 GMT', 'X-Content-Type-Options': 'nosniff'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets is forbidden: User "system:serviceaccount:gcc:ecr-updater" cannot list resource "secrets" in API group "" at the cluster scope","reason":"Forbidden","details":{"kind":"secrets"},"code":403}

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ecr-updater
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  - configmaps
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
  - list

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ecr-updater
  namespace: gcc
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ecr-updater
subjects:
- kind: ServiceAccount
  name: ecr-updater
  namespace: gcc

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ecr-updater
  namespace: gcc
secrets:
- name: ecr-updater-token-fp7zx

apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: BASE64ACCESSKEY
  AWS_SECRET_ACCESS_KEY: BASE64SECRET
kind: Secret
metadata:
  name: ecr-aws-credentials
  namespace: gcc
type: Opaque

data:
  .dockerconfigjson: SOMEVALUE
kind: Secret
metadata:
  name: ecr
  namespace: gcc
type: kubernetes.io/dockerconfigjson

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "3"
  generation: 3
  labels:
    k8s-app: ecr-updater
  name: ecr-updater
  namespace: gcc
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: ecr-updater
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        k8s-app: ecr-updater
        name: ecr-updater
        namespace: gcc
    spec:
      containers:
      - env:
        - name: K8S_PULL_SECRET_NAME
          value: ecr:gcc
        - name: ECR_CREATE_MISSING
          value: "false"
        - name: AWS_DEFAULT_REGION
          value: eu-west-1
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              key: AWS_ACCESS_KEY_ID
              name: ecr-aws-credentials
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              key: AWS_SECRET_ACCESS_KEY
              name: ecr-aws-credentials
        image: trondhindenes/k8s-ecrupdater
        imagePullPolicy: Always
        name: ecrupdater
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - image: palmstonegames/kubectl-proxy
        imagePullPolicy: Always
        name: kubectl-proxy
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: ecr-updater
      serviceAccountName: ecr-updater
      terminationGracePeriodSeconds: 30

Refactor from deployment to cronjob

The project could leverage the K8s CronJob to do the scheduling. This would simplify the code and optimize the K8s resource usage.

eg

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: ecr-credentials-updater
spec:
  schedule: "* */3 * * *" # ecr token expires every 12 hours, we try every 3 hours to refresh the token
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 2
...

is there a way to exclude a namespace to avoid creating secret

Hi
ecr-updater works perfectly fine. It helps to create a secret in all the namespaces when proper ENV vars are passed.
However i dont want the secret to be created in all the namespace. Please let me know if there's a way to exclude certain namespace so the secret is not created.

Thanks,
Amjad

Better Defaults

It would be nice if "ECR_CREATE_MISSING" was set to true by default as I think it's a sensible default for ease of use here.

Error after executing 02_deployment.yml

After following steps in the readme, I get the following error which I am not too sure what's up with. Can you help me decipher what that might mean please?

container_linux.go:247: starting container process caused "process_linux.go:295: setting oom score for ready process caused \"write /proc/10349/oom_score_adj: invalid argument\""