lchown failures on `img pull` when running on k8s

[genuinetools]

lchown failures on `img pull` when running on k8s

[AkihiroSuda]

We need to wait for opencontainers/runc#1693 or use ptrace hack.

Please refer to README for further info

[jbeda]

Ah! I asssumed that r.j3ss.co/img already had a hacked runc in there to work around the setgroup issue.

[jessfraz]

I just pushed a new build with it

On Sat, Feb 24, 2018 at 14:21 Joe Beda ***@***.***> wrote: Ah! I asssumed that r.j3ss.co/img already had a hacked runc in there to work around the setgroup issue. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#40 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABYNbGGeil08uyqgHVvXj9BDIGcBPHXxks5tYGE3gaJpZM4SR93F> .

-- Jessie Frazelle 4096R / D4C4 DD60 0D66 F65A 8EFC 511E 18F3 685C 0022 BFF3 pgp.mit.edu <http://pgp.mit.edu/pks/lookup?op=get&search=0x18F3685C0022BFF3>

[jessfraz]

We actually not with that one with the other one... I'll do them all tho later

[AkihiroSuda]

@jessfraz
you can use this branch
https://github.com/AkihiroSuda/runc/commits/demo-rootless

[AkihiroSuda]

or this tag: https://github.com/AkihiroSuda/runc/tree/demo-rootless.20180116-0
(commits are same; using tag rather than branch would be more deterministic)

[jbeda]

We actually not with that one with the other one... I'll do them all tho later

I'm not sure which one you are talking about... but I just tried with your new image and I'm seeing the same behavior. I'm a bit confused still about what patches y'all are talking about and the necessary setup for those to work. But I'll let y'all figure this out. @jessfraz if this is being tracked in a different place feel free to close this issue.

[jessfraz]

I'm going to make some k8s examples and carry the patches on my flight this afternoon, sorry was doing saturday things :)

[jessfraz]

then we can work out all the rough edges, I also hope that then we can help with them testing this upstream (re comment here: moby/moby#32925 (comment)) I know the last time the builder in docker was replaced we had to test a lot to find the weird bashisms that broke and odd bugs in the old builder that were being used as features, etc

[AkihiroSuda]

seems closable now?

[ccremer]

I'm having a similar issue (Operation not permitted), but only when trying to use a different state directory (need build cache with PVC):

apiVersion: batch/v1
kind: Job
metadata:
  name: img-build
  namespace: amze-2029
spec:
  template:
    metadata:
      annotations:
        container.apparmor.security.beta.kubernetes.io/build: unconfined
        container.seccomp.security.alpha.kubernetes.io/build: unconfined
    spec:
      initContainers:
      - name: git-clone
        image: r.j3ss.co/img:v0.5.7
        command:
        - /bin/sh
        args:
        - -c
        - git clone https://github.com/amazeeio/drupal-example.git
        workingDir: /home/user/src
        volumeMounts:
        - mountPath: /home/user/src
          name: src
      containers:
      - name: build
        image: r.j3ss.co/img:v0.5.7
        command:
        - /bin/sh
        args:
        - -c
        - >
          id &&
          echo "RUN apk add --no-cache tcpdump" >> drupal-example/lagoon/php.dockerfile &&
          img build -s /tmp -t registry.ch-gva-2.exo.appuio.ch/amze-2029/drupal-example:latest -f drupal-example/lagoon/php.dockerfile
          --build-arg CLI_IMAGE=registry.ch-gva-2.exo.appuio.ch/amze-2029/drupal-example:cli drupal-example &&
          img push -s /tmp registry.ch-gva-2.exo.appuio.ch/amze-2029/drupal-example:latest
        securityContext:
        #   privileged: true
          #runAsUser: 1000
          #runAsGroup: 0
          procMount: Unmasked
        workingDir: /home/user/src
        volumeMounts:
        - mountPath: /home/user/src
          name: src
        - mountPath: /tmp
          name: cache
        - mountPath: /home/user/.docker/config.json
          subPath: config.json
          name: registries
      restartPolicy: Never
      volumes:
      - name: src
        emptyDir: {}
      - name: cache
        persistentVolumeClaim:
          claimName: img-cache
      - name: registries
        secret:
          secretName: regcred
          items:
          - key: .dockerconfigjson
            path: config.json
  backoffLimit: 0

log excerpt:

#7 sha256:831a6d750410aeacea063fbe7db0e87c626b3019ae8f56be0c9aefd3d6b8af76 14.71MB / 14.71MB done

#7 unpacking registry.ch-gva-2.exo.appuio.ch/amze-2029/drupal-example:cli@sha256:14630f6378aa48eb2e32bcdffafc5c6eec8d49dca50d4b50f89cd61db8c40195

time="2020-03-12T18:35:32Z" level=info msg="apply failure, attempting cleanup" error="failed to extract layer sha256:77cae8ab23bf486355d1b3191259705374f4a11d483b24964d2f729dd8c076a0: mount callback failed on /run/user/1000/containerd-mount827883887: lchown /run/user/1000/containerd-mount827883887/etc/shadow: operation not permitted" key="extract-844920920-8DtT sha256:77cae8ab23bf486355d1b3191259705374f4a11d483b24964d2f729dd8c076a0"

time="2020-03-12T18:35:32Z" level=info msg="apply failure, attempting cleanup" error="failed to extract layer sha256:77cae8ab23bf486355d1b3191259705374f4a11d483b24964d2f729dd8c076a0: mount callback failed on /run/user/1000/containerd-mount475060601: context canceled" key="extract-88219689-NE5W sha256:77cae8ab23bf486355d1b3191259705374f4a11d483b24964d2f729dd8c076a0"

#6 ...

I started trying different parameters (was using /cache before /tmp) and also played with the securityContext user/group ids, but to no avail.
Kubernetes is v1.14.9-eks-502bfb

[exherb]

some issue here.