Support mounting host directories into pods

[kubernetes]

Support mounting host directories into pods

[boj]

These are the steps I look to get hostPath working on OSX with NFS. Using xhyve, but I imagine this works for other VMs.

# install bits
brew install docker kubectl docker-machine-driver-xhyve
sudo chown root:wheel $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
sudo chmod u+s $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve

# setup bits
sudo echo "/Users -network 192.168.64.0 -mask 255.255.255.0 -alldirs -maproot=root:wheel" > /etc/exports
sudo nfsd restart
minikube start --vm-driver=xhyve
minikube ssh -- sudo /usr/local/etc/init.d/nfs-client start
minikube ssh -- sudo mkdir /Users
minikube ssh -- sudo mount 192.168.64.1:/Users /Users -o rw,async,noatime,rsize=32768,wsize=32768,proto=tcp
eval $(minikube docker-env)

[aaron-prindle]

Closing. As of #1293 we now have an official minikube mount /path/to/folder/to/mount command that allows for specific directories to mapped into minikube across all platforms and these folders can be accessed from pods via a hostPath volume mount.

[dlorenc]

Hey, on a Mac using Virtualbox, the /Users/ directory is automatically mapped into the VM (also under /Users). You could then use a HostPath volume of "/Users/$USER/$DIR". Does that help?

[solsson]

Why it is not a good fit? Why do those errors happen?

Because vboxsf mounts are quite limited as file system. I did not investigate the postgres issue further.

For vbox minikube based on your NFS instructions (with minikube ip = 192.168.99.106) I did:

echo "/Users -network 192.168.99.0 -mask 255.255.255.0 -alldirs -maproot=root:wheel" | sudo tee -a /etc/exports
sudo nfsd restart
minikube start
minikube ssh -- sudo umount /Users
minikube ssh -- sudo /usr/local/etc/init.d/nfs-client start
minikube ssh -- sudo mount 192.168.99.1:/Users /Users -o rw,async,noatime,rsize=32768,wsize=32768,proto=tcp

Much better than shared folder, as uid + gid + mod is preserved.

[gjcarneiro]

Latest minikube 0.15 iso image doesn't support vboxsf anymore, it seems. I also tried NFS to no avail.

Anyway, IMHO NFS is /not/ better than vboxsf. I usually have some python code running in gunicorn with the --reload option, which uses inotify. NFS doesn't support inotify, so it will be probably have to revert to polling, which is slower and more resource consuming.

Even NFS I couldn't get to work:

$ cat /etc/lfs-release 
NAME=Buildroot
VERSION=2016.08
ID=buildroot
VERSION_ID=2016.08
PRETTY_NAME="Buildroot 2016.08"
$ sudo mount -t nfs 192.168.99.1:/home /mnt/hosthome
mount: wrong fs type, bad option, bad superblock on 192.168.99.1:/home,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

How can you release a minikube version without any working option for sharing folders between guest and host?... :(

[solsson]

The permission issues that @Ehekatl mentions are quite a blocker. With a PersistentVolume mounted to a pod with a https://hub.docker.com/_/postgres/ container I get:

root@postgres-0:/# ls -la /var/lib/postgresql/data/
drwxr-xr-x 2 root     root       40 Jul 18 15:05 .
drwxr-xr-x 3 postgres postgres 4096 Jul 18 15:09 ..

When postgres tries to create data there it fails with mkdir: cannot create directory ‘/var/lib/postgresql/data/pg_xlog’: Permission denied. No chmod or chown on the host system seems to make any difference. There's plenty of official and non-official Docker images that run a non-privileged user. Postgres prepends its start command with gosu postgres.

root@postgres-0:/# id     
uid=0(root) gid=0(root) groups=0(root)
root@postgres-0:/# gosu postgres id
uid=999(postgres) gid=999(postgres) groups=999(postgres),109(ssl-cert)

I've tested with both virtualbox and xhyve minikube, on Mac.

[dlorenc]

The existing vboxfs system isn't great. We should figure out whether to use NFS or 9p, and make this work for all drivers.

[notmaxx]

i had the same problem with mount nfs as @gjcarneiro
finally command below worked for me:
sudo busybox mount -t nfs <host.ip.addre.ss>:/host_path_to_share /mnt/host -o nolock,tcp,rw

Command below
sudo mount -t nfs ... gave the same result as yours

[dlorenc]

Of course, HostPath will work right now, but it won't mount things into the VM from the outside host.

For that to work we might need to do dynamic mounting/provisioning from the laptop into the VM.

[pmatety]

I am using ubuntu, KVM driver, my mounted directory is empty in MinikubeVM
Update: I was using minikube 0.18, updating to 0.19 mount works now :)

[dlorenc]

One possibility here might be to support an NFS-based persistent volume claim from the guest to the host: http://kubernetes.io/docs/user-guide/persistent-volumes/

[rata]

On Mon, May 16, 2016 at 05:31:18PM -0700, dlorenc wrote:

One possibility here might be to support an NFS-based persistent volume claim from the guest to the host: http://kubernetes.io/docs/user-guide/persistent-volumes/

NFS persistent volume claim? What? Why?

Why not HostPath, to use a local directory, and that's all regarding the volume?

[rata]

On Mon, May 16, 2016 at 05:42:00PM -0700, dlorenc wrote:

Of course, HostPath will work right now, but it won't mount things into the VM from the outside host.

Oh, I see what you mean now. But I'm not sure, as @vishh said in the original
report: "most hypervisors can already support mapping host directories". So,
maybe a host dir can be mounted in the VM and then in the container.

NFS is another option, but I'm not sure how that would work. A host dir in the
VM, that dir exported as NFS, and mounted by k8s as an NFS volume? Or what were
you thinking?

[rata]

Also, if docker is used, it can mount from host directly on mac/windows with docker "native" beta. One option is to use that.

Another is to do the same/similar trick that docker is using (probably the one @vishh said and the one I said in my previous post)

[vishh]

If we can have kubelet take in a configurable root directory for hostPath volumes, then we can have the kubelet map in directories from inside the VM that mirror the host root.

[Ehekatl]

this feature is super important for us, is there any way to let me add a shared folder before this come out ?

[Ehekatl]

@dlorenc thx, I add mount -t command at boot2docker init script now, but hostpath volume will facing some permission issue (e.g. postgres container), seems nfs is the only workaround

[rata]

@solsson: maybe it's an option to add permission to the volume API in kubernetes. I'm doing that for others volumes: kubernetes/kubernetes#28733 and kubernetes/kubernetes#28936

[solsson]

I noticed that kubernetes/kubernetes#28733 is merged now. Interesting.

Before Docker for Mac, with Docker Toolbox and boot2docker, I had this kind if issue often with plain docker containers. Hence I'm not so sure the issue is with Kubernetes. Maybe it's inherent with docker + non-privileged user + virtual machine mount.

If I use /tmp/somevolume as hostPath I do get the same issue at postgres start. But the difference is that I can chmod go+ws /tmp/somevolume to solve it (in virtualbox minikube).

This leads me to believe that the problem is with minikube. After all hostPath works fine on for example GKE. Maybe it's with how minikube mounts the shared folder?

[solsson]

This boot2docker issue describes the problem pretty accurately.

I think I found a workaround for virtualbox. After minikube start open the vm in virtualbox (or ssh) and run

umount /Users
mount -t vboxsf -o rw,nodev,relatime,suid,dmode=777,fmode=777 Users /Users

The first three options are from the default mount. Probably only one of the added options are needed.

With postgres howerver the next issue is:

LOG:  could not link file "pg_xlog/xlogtemp.20" to "pg_xlog/000000010000000000000001": Operation not permitted
FATAL:  could not open file "pg_xlog/000000010000000000000001": No such file or directory

which I think confims my suspicion that docker and vm shared folders is not a good fit.

[solsson]

In conclusion, an added mount option (see above) might improve minikube. But until native docker is supported, including Docker for Mac, I think that to stay sane one should realize that some services demand a lot from the file system and hostPath should really be the linux host's folder.

With /var/kubedata/pgdata-postgres-0 postgres runs fine. I noticed that /var has more liberal permissions than /tmp that I tested above.

So to access this data directly I'd have to share it from the VM to my physical host.

[rata]

On Mon, Jul 18, 2016 at 10:49:17AM -0700, solsson wrote:

This boot2docker issue describes the issue pretty accurately.

I think I found a workaround for virtualbox. After minikube start open the vm in virtualbox (or ssh) and run

umount /Users
mount -t vboxsf -o rw,nodev,relatime,suid,dmode=777,fmode=777 Users /Users

The first three options are from the default mount. Probably only one of the added options are needed.

With postgres howerver the next issue is:

LOG:  could not link file "pg_xlog/xlogtemp.20" to "pg_xlog/000000010000000000000001": Operation not permitted
FATAL:  could not open file "pg_xlog/000000010000000000000001": No such file or directory

which I think confims my suspicion that docker and vm shared folders is not a good fit.

Why it is not a good fit? Why do those errors happen?

[dlorenc]

We should document how to setup NFS in the meantime.

[dlorenc]

@gjcarneiro thanks for the report. It looks like something got messed up in the new ISO for vbox folder sharing. It's supposed to work using the vbox guest additions here: https://github.com/kubernetes/minikube/tree/master/deploy/iso/minikube-iso/package/vbox-guest

But logs on my VM show:

$ systemctl status vboxservice.service
● vboxservice.service - VirtualBox Guest Service
   Loaded: loaded (/lib/systemd/system/vboxservice.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2017-01-11 17:11:05 UTC; 11min ago
 Main PID: 2792 (code=exited, status=1/FAILURE)

Jan 11 17:11:05 minikube systemd[1]: Starting VirtualBox Guest Service...
Jan 11 17:11:05 minikube modprobe[2503]: modprobe: ERROR: could not insert 'vboxguest': No such device
Jan 11 17:11:05 minikube modprobe[2620]: modprobe: ERROR: could not insert 'vboxvideo': No such device
Jan 11 17:11:05 minikube modprobe[2759]: modprobe: ERROR: could not insert 'vboxsf': No such device
Jan 11 17:11:05 minikube systemd[1]: Started VirtualBox Guest Service.
Jan 11 17:11:05 minikube VBoxService[2792]: VBoxService: error: VbglR3Init failed with rc=VERR_FILE_NOT_FOUND
Jan 11 17:11:05 minikube systemd[1]: vboxservice.service: Main process exited, code=exited, status=1/FAILURE
Jan 11 17:11:05 minikube systemd[1]: vboxservice.service: Unit entered failed state.
Jan 11 17:11:05 minikube systemd[1]: vboxservice.service: Failed with result 'exit-code'.

We'll take a look ASAP.

[r2d4]

@gjcarneiro sorry about that -- the image has been fixed. You can get the new one simply by removing the old one

$ minikube delete
$ rm ~/.minikube/iso/cache/minikube-v1.0.2.iso

Then minikube will download the fix in the next minikube start

[keithharvey]

Upgraded docker for mac to 0.12.6 and minikube to 0.15 around the same time (and I deleted the image after the iso was fixed following the instructions above)

minikube ssh -- sudo /usr/local/etc/init.d/nfs-client start

That folder no longer exists. /usr/local/bin exists

[notmaxx]

@keithharvey same for me, that's why I just used busybox mount

[gjcarneiro]

@r2d4 it works fine now, thanks for the quick fix!

[keithharvey]

@notmaxx That worked, thanks!

[n-sviridenko]

@boj I a little bit changed your solution: https://gist.github.com/n-sviridenko/43e9e6bcaab2be356ae3af128c0e6d0b . Is there anything that can improve it?

[pietervogelaar]

I create a blog post for setting up a Minikube NFS mount. This way containers can mount source code on the host machine with great performance! http://pietervogelaar.nl/minikube-nfs-mounts

[pchico83]

It is more performance to synchronize the local and the minikube filesystem using syncthing than using shared folders. We built an open source tool that follows this approach:
https://github.com/okteto/cnd