Busybox not working in debug CC image on GKE

[googlecontainertools]

Busybox not working in debug CC image on GKE

[mattmoor]

That's... exciting :)

I wonder if updating will help, I found this.

[drigz]

Yep, found that too, although the version in cc:debug is much more recent:

BusyBox v1.27.1 (2017-07-18 19:24:44 CEST) multi-call binary.

[drigz]

Also, apparently that's only for certain directories:

        /*
         * Try to detect broken toolbox/busybox binaries (see
         * https://code.google.com/p/busybox-android/issues/detail?id=1)
         * 
         * It is giving "Value too large for defined data type" on certain file operations (e.g. ls
         * and chown) in certain directories (e.g. /data/data)
         */

[zdicesare]

We also ran into this, and it turns out that there is a busybox config setting CONFIG_LFS that it must be compiled with to support working with large files. Without it you may encounter errors in ls, rm, etc.

The provided busybox binaries appear to be not compiled with that flag. We built busybox from source using the default config to resolve this. The busybox http_file in distroless' WORKSPACE is using a default provided binary.

[tejal29]

I checked, the latest binary, https://busybox.net/downloads/binaries/1.27.1-i686/config has CONFIG_LFS=y set.
This does not exist anymore.

tejaldesai@tejaldesai$ kubectl exec -it testdep-77888775fb-psf82 sh
/ # ls
busybox  etc      lib64    sys      usr
dev      lib      proc     tmp      var
/ # 

[jsravn]

Is this released in the latest gcr.io/distroless/base:debug tag? Still experiencing it.

[drigz]

I'm also seeing it:

/root # busybox | head -n 1
BusyBox v1.27.1 (2017-07-18 19:24:44 CEST) multi-call binary.
/root # ls
ls: can't open '.': Value too large for defined data type

on a GKE cluster, node version 1.9.4-gke.1, machine type n1-standard-4

[hrfuller]

I am also seeing this in GKE running a busybox on top of the container optimized os 4.14.22+ kernel. The busybox version is the same one that @tejal29 listed as working BusyBox v1.27.1 (2017-07-18 19:24:44 CEST) multi-call binary.

[tejal29]

Somehow, i cant reproduce this on my end.

 kubectl run testdep --image=gcr.io/distroless/cc:debug --command -- sleep 999999
deployment.apps/testdep created

tejaldesai@tejaldesai~:$ kubectl get po
NAME                         READY     STATUS             RESTARTS   AGE
testdep-6fdff65d5b-xjsh7     1/1       Running            0          15s
tejaldesai@tejaldesai~:$ kubectl exec -it testdep-6fdff65d5b-xjsh7 sh
/ # ls
busybox  etc      lib      proc     tmp      var
dev      home     lib64    sys      usr
/ # busybox | head -n 1
BusyBox v1.27.1 (2017-07-18 19:24:44 CEST) multi-call binary.

[tejal29]

can you provide me with your docker build file?

[hrfuller]

I can't seem to recreate this in our GKE cluster anymore. Its possible some nodes were updated, either the underlying kernel or the k8s version.

[cpick]

I suspect this was the same root cause as #225