Diego fail to start a container on centos 7.1 with aufs

[cloudfoundry]

Diego fail to start a container on centos 7.1 with aufs

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/94479746.

[guanglinlv]

hi,

/dev/vdb2 is a partition of openstack ephemeral disk that mount at /var/vcap/data, pivot_root cant work correctly on ephemeral disk of OpenStack ?

[guanglinlv]

hi,

i have try it again without openstack ephemeral disk,but pivot_root also complains Invalid argument.

the partition info:

Filesystem     Type      Size  Used Avail Use% Mounted on
devtmpfs       devtmpfs  2.0G     0  2.0G   0% /dev
tmpfs          tmpfs     2.0G     0  2.0G   0% /dev/shm
tmpfs          tmpfs     2.0G  8.3M  2.0G   1% /run
tmpfs          tmpfs     2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda1      ext4       38G   11G   25G  30% /
/dev/vda3      ext4       38G  1.5G   34G   5% /var/vcap/data
tmpfs          tmpfs     1.0M   28K  996K   3% /var/vcap/data/sys/run
/dev/loop0     ext4      120M  1.6M  115M   2% /tmp
cgroup         tmpfs     2.0G  8.0K  2.0G   1% /tmp/garden-/cgroup
none           aufs       38G  1.5G   34G   5% /var/vcap/data/garden-linux/overlays/ngsn71i2p5t/rootfs
none           aufs       38G  1.5G   34G   5% /var/vcap/data/garden-linux/overlays/ngsn71i2p5t/rootfs

[guanglinlv]

hi,

I have changed pivot_root to chroot at wshd.c#L776,the container create successfully,but the garden-server run exited with status 255.

I guess the cause is that /tmp/docker_app_lifecycle is not exist in container rootfs .actually,it is downloaded in cell rootfs.

how to replace pivot_root to chroot correctly or correct pivot_root Invalid argument?

[21:47 root@a8fa3127-b50c-4478-bdc7-1aaa97fcf21a:/var/vcap/data/compile/garden-linux] > ll /tmp/docker_app_lifecycle/
total 29M
-rwxr-xr-x 1 vcap vcap 8.7M May 14 14:09 builder
-rwxr-xr-x 1 vcap vcap  15M May 14 14:08 docker
-rwxr-xr-x 1 vcap vcap 2.9M May 14 14:09 healthcheck
-rwxr-xr-x 1 vcap vcap 2.5M May 14 14:09 launcher
[21:48 root@a8fa3127-b50c-4478-bdc7-1aaa97fcf21a:/var/vcap/data/compile/garden-linux] > ll /var/vcap/data/garden-linux/overlays/nhgbf7setl1/rootfs/tmp/
total 4.0K
drwxr-xr-x 3 root root 4.0K May 14 21:46 garden-host

df information

Filesystem     Type      Size  Used Avail Use% Mounted on
devtmpfs       devtmpfs  2.0G     0  2.0G   0% /dev
tmpfs          tmpfs     2.0G     0  2.0G   0% /dev/shm
tmpfs          tmpfs     2.0G  8.3M  2.0G   1% /run
tmpfs          tmpfs     2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda1      ext4       38G   11G   25G  31% /
/dev/vda3      ext4       38G  2.3G   34G   7% /var/vcap/data
tmpfs          tmpfs     1.0M   28K  996K   3% /var/vcap/data/sys/run
/dev/loop0     ext4      120M   30M   87M  26% /tmp
cgroup         tmpfs     2.0G   12K  2.0G   1% /tmp/garden-/cgroup
none           aufs       38G  2.3G   34G   7% /var/vcap/data/garden-linux/overlays/nhgbf7setl1/rootfs
none           aufs       38G  2.3G   34G   7% /var/vcap/data/garden-linux/overlays/nhgbf7setl1/rootfs
tmpfs          tmpfs     2.0G     0  2.0G   0% /var/vcap/data/garden-linux/overlays/nhgbf7setl1/rootfs/dev/shm
tmpfs          tmpfs     2.0G     0  2.0G   0% /var/vcap/data/garden-linux/overlays/nhgbf7setl1/rootfs/dev/shm
none           aufs       38G  2.3G   34G   7% /var/vcap/data/garden-linux/overlays/nhgbf7setl1/rootfs/app
none           aufs       38G  2.3G   34G   7% /var/vcap/data/garden-linux/overlays/nhgbf7setl1/rootfs/app

the garden-linux.stdout.log with chroot

{"timestamp":"1431640008.444269419","source":"garden-linux","message":"garden-linux.pool.nhgbf7setl1.creating","log_level":1,"data":{"session":"2.2"}}
{"timestamp":"1431640008.444363356","source":"garden-linux","message":"garden-linux.pool.nhgbf7setl1.acquired-pool-resources","log_level":1,"data":{"session":"2.2"}}
{"timestamp":"1431640009.349983692","source":"garden-linux","message":"garden-linux.pool.nhgbf7setl1.created","log_level":1,"data":{"session":"2.2"}}
{"timestamp":"1431640009.734901905","source":"garden-linux","message":"garden-linux.pool.nhgbf7setl1.start.started","log_level":1,"data":{"session":"2.2.4"}}
{"timestamp":"1431640009.735139132","source":"garden-linux","message":"garden-linux.garden-server.create.created","log_level":1,"data":{"request":{"handle":"bd0abab0-a154-4c00-b79e-0dc1292a036b-059c342535d94320a57e0582b90897b3","rootfs":"/var/vcap/packages/rootfs_cflinuxfs2","properties":{"executor:action":"{\"timeout\":{\"action\":{\"serial\":{\"actions\":[{\"emit_progress\":{\"start_message\":\"\",\"success_message\":\"\",\"failure_message_prefix\":\"Failed to set up docker environment\",\"action\":{\"download\":{\"from\":\"http://file-server.service.consul:8080/v1/static/docker_app_lifecycle/docker_app_lifecycle.tgz\",\"to\":\"/tmp/docker_app_lifecycle\",\"cache_key\":\"builder-docker\"}}}},{\"emit_progress\":{\"start_message\":\"Staging...\",\"success_message\":\"Staging Complete\",\"failure_message_prefix\":\"Staging Failed\",\"action\":{\"run\":{\"path\":\"/tmp/docker_app_lifecycle/builder\",\"args\":[\"-outputMetadataJSONFilename\",\"/tmp/docker-result/result.json\",\"-dockerRef\",\"10.10.10.210:8080/lvguanglin/kodexplorer:latest\",\"-dockerRegistryAddresses\",\"10.10.10.210:8080\",\"-insecureDockerRegistries\",\"10.10.10.210:8080\",\"-cacheDockerImage\"],\"env\":[{\"name\":\"VCAP_APPLICATION\",\"value\":\"{\\\"limits\\\":{\\\"mem\\\":256,\\\"disk\\\":1024,\\\"fds\\\":16384},\\\"application_version\\\":\\\"95199d79-8929-4aa8-9ad5-2b249786b6c4\\\",\\\"application_name\\\":\\\"kod\\\",\\\"version\\\":\\\"95199d79-8929-4aa8-9ad5-2b249786b6c4\\\",\\\"name\\\":\\\"kod\\\",\\\"space_name\\\":\\\"admin\\\",\\\"space_id\\\":\\\"7b633d82-73fb-47e2-bf2d-24734bf68f9a\\\"}\"},{\"name\":\"VCAP_SERVICES\",\"value\":\"{}\"},{\"name\":\"MEMORY_LIMIT\",\"value\":\"256m\"},{\"name\":\"CF_STACK\",\"value\":\"cflinuxfs2\"},{\"name\":\"DIEGO_DOCKER_CACHE\",\"value\":\"true\"}],\"resource_limits\":{\"nofile\":16384},\"privileged\":true}}}}]}},\"timeout\":900000000000}}","executor:allocated-at":"1431640008419949990","executor:cpu-weight":"100","executor:disk-mb":"6144","executor:egress-rules":"[{\"protocol\":\"all\",\"destinations\":[\"0.0.0.0-9.255.255.255\"],\"log\":false},{\"protocol\":\"all\",\"destinations\":[\"11.0.0.0-169.253.255.255\"],\"log\":false},{\"protocol\":\"all\",\"destinations\":[\"169.255.0.0-172.15.255.255\"],\"log\":false},{\"protocol\":\"all\",\"destinations\":[\"172.32.0.0-192.167.255.255\"],\"log\":false},{\"protocol\":\"all\",\"destinations\":[\"192.169.0.0-255.255.255.255\"],\"log\":false},{\"protocol\":\"tcp\",\"destinations\":[\"0.0.0.0/0\"],\"ports\":[53],\"log\":false},{\"protocol\":\"udp\",\"destinations\":[\"0.0.0.0/0\"],\"ports\":[53],\"log\":false},{\"protocol\":\"all\",\"destinations\":[\"0.0.0.0-9.255.255.255\"],\"log\":false},{\"protocol\":\"all\",\"destinations\":[\"11.0.0.0-169.253.255.255\"],\"log\":false},{\"protocol\":\"all\",\"destinations\":[\"169.255.0.0-172.15.255.255\"],\"log\":false},{\"protocol\":\"all\",\"destinations\":[\"172.32.0.0-192.167.255.255\"],\"log\":false},{\"protocol\":\"all\",\"destinations\":[\"192.169.0.0-255.255.255.255\"],\"log\":false},{\"protocol\":\"tcp\",\"destinations\":[\"0.0.0.0/0\"],\"ports\":[53],\"log\":false},{\"protocol\":\"udp\",\"destinations\":[\"0.0.0.0/0\"],\"ports\":[53],\"log\":false},{\"protocol\":\"tcp\",\"destinations\":[\"10.10.10.210\"],\"ports\":[8080],\"log\":false}]","executor:env":"null","executor:log-config":"{\"guid\":\"bd0abab0-a154-4c00-b79e-0dc1292a036b\",\"index\":0,\"source_name\":\"STG\"}","executor:memory-mb":"1024","executor:metrics-config":"{\"guid\":\"\",\"index\":0}","executor:monitor":"null","executor:owner":"executor","executor:result":"{\"failed\":false,\"failure_reason\":\"\",\"stopped\":false}","executor:rootfs":"/var/vcap/packages/rootfs_cflinuxfs2","executor:setup":"null","executor:start-timeout":"0","executor:state":"created","tag:domain":"cf-app-staging","tag:lifecycle":"task","tag:result-file":"/tmp/docker-result/result.json"},"privileged":true},"session":"4.2221"}}
{"timestamp":"1431640009.831254721","source":"garden-linux","message":"garden-linux.garden-server.limit-memory.limited","log_level":1,"data":{"handle":"bd0abab0-a154-4c00-b79e-0dc1292a036b-059c342535d94320a57e0582b90897b3","resulting-limits":{"limit_in_bytes":1073741824},"session":"4.2238"}}
{"timestamp":"1431640009.832304239","source":"garden-linux","message":"garden-linux.garden-server.limit-disk.limited","log_level":1,"data":{"handle":"bd0abab0-a154-4c00-b79e-0dc1292a036b-059c342535d94320a57e0582b90897b3","resulting-limits":{},"session":"4.2239"}}
{"timestamp":"1431640009.833291769","source":"garden-linux","message":"garden-linux.garden-server.limit-cpu.limited","log_level":1,"data":{"handle":"bd0abab0-a154-4c00-b79e-0dc1292a036b-059c342535d94320a57e0582b90897b3","resulting-limits":{"limit_in_shares":1024},"session":"4.2240"}}
{"timestamp":"1431640009.834185839","source":"garden-linux","message":"garden-linux.garden-server.info.got-info","log_level":1,"data":{"handle":"bd0abab0-a154-4c00-b79e-0dc1292a036b-059c342535d94320a57e0582b90897b3","session":"4.2241"}}
{"timestamp":"1431640009.848406553","source":"garden-linux","message":"garden-linux.garden-server.info.got-info","log_level":1,"data":{"handle":"bd0abab0-a154-4c00-b79e-0dc1292a036b-059c342535d94320a57e0582b90897b3","session":"4.2244"}}
{"timestamp":"1431640009.860832214","source":"garden-linux","message":"garden-linux.garden-server.info.got-info","log_level":1,"data":{"handle":"bd0abab0-a154-4c00-b79e-0dc1292a036b-059c342535d94320a57e0582b90897b3","session":"4.2246"}}
{"timestamp":"1431640010.685299158","source":"garden-linux","message":"garden-linux.garden-server.stream-in.streamed-in","log_level":1,"data":{"destination":"/tmp/docker_app_lifecycle","handle":"bd0abab0-a154-4c00-b79e-0dc1292a036b-059c342535d94320a57e0582b90897b3","session":"4.2247"}}
{"timestamp":"1431640010.693485737","source":"garden-linux","message":"garden-linux.garden-server.run.spawned","log_level":1,"data":{"handle":"bd0abab0-a154-4c00-b79e-0dc1292a036b-059c342535d94320a57e0582b90897b3","id":1,"session":"4.2248","spec":{"path":"/tmp/docker_app_lifecycle/builder","args":["-outputMetadataJSONFilename","/tmp/docker-result/result.json","-dockerRef","10.10.10.210:8080/lvguanglin/kodexplorer:latest","-dockerRegistryAddresses","10.10.10.210:8080","-insecureDockerRegistries","10.10.10.210:8080","-cacheDockerImage"],"env":["VCAP_APPLICATION={\"limits\":{\"mem\":256,\"disk\":1024,\"fds\":16384},\"application_version\":\"95199d79-8929-4aa8-9ad5-2b249786b6c4\",\"application_name\":\"kod\",\"version\":\"95199d79-8929-4aa8-9ad5-2b249786b6c4\",\"name\":\"kod\",\"space_name\":\"admin\",\"space_id\":\"7b633d82-73fb-47e2-bf2d-24734bf68f9a\"}","VCAP_SERVICES={}","MEMORY_LIMIT=256m","CF_STACK=cflinuxfs2","DIEGO_DOCKER_CACHE=true","CF_INSTANCE_IP=10.10.10.133","CF_INSTANCE_PORT=","CF_INSTANCE_ADDR=","CF_INSTANCE_PORTS="],"privileged":true,"rlimits":{"nofile":16384}}}}
{"timestamp":"1431640010.700257301","source":"garden-linux","message":"garden-linux.garden-server.run.exited","log_level":1,"data":{"handle":"bd0abab0-a154-4c00-b79e-0dc1292a036b-059c342535d94320a57e0582b90897b3","id":1,"session":"4.2248","status":255}}

[emalm]

Hi, @guanglinlv,

Since you're having problems running garden-linux itself, it would be better for you to raise an issue on either the garden-linux repo or its BOSH release. The engineers on the CF Garden team would probably be able to help you resolve these issues better than the Diego team would.

Thanks,
Eric

[guanglinlv]

hi @ematpl ,

actually,i am using diego to do this, i also will raise an issue on the garden-linux repo.

thanks you all the same.

[emalm]

Hi, @guanglinlv,

Yes, I realize you've deployed garden-linux via diego-release, but the problems here seem isolated to garden-linux itself. I suspect you would see the same problems if you deployed garden-linux on its own via its BOSH release and attempted to create and operate a container via the Garden API, which is why I suggested that the Garden team would have more insight into these issues.

Thanks,
Eric

[guanglinlv]

hi all

I have fix the pivot_root invalid argument problem

thanks.