Problem with apk, dns and boot2docker

[gliderlabs]

Problem with apk, dns and boot2docker

[jumanjiman]

@JonnyBGod perhaps check your network settings.

example 1: network is good

user@devenv:~$ docker run --rm -it alpine:latest sh
/ # apk update
fetch http://dl-4.alpinelinux.org/alpine/v3.1/main/x86_64/APKINDEX.tar.gz
v3.1.3-44-g7991b61 [http://dl-4.alpinelinux.org/alpine/v3.1/main]
OK: 4868 distinct packages available

example 2: network is bad or no default route to fetch repo data; output matches your pastebin console log

user@devenv:~$ docker run --rm -it --net=none alpine:latest sh
/ # apk update
fetch http://dl-4.alpinelinux.org/alpine/v3.1/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-4.alpinelinux.org/alpine/v3.1/main: IO ERROR
WARNING: Ignoring APKINDEX.689bb31a.tar.gz: No such file or directory
OK: 15 distinct packages available

hth!

[jumanjiman]

iow, if your boot2docker environment doesn't have a route through your physical host to reach the internet or doesn't have a proxy configured (but needs one), your build returns the same error that i mimic'd with example 2 (--net=none).

[JonnyBGod]

will check it, but it works well with other linux distributions such as ubuntu and debian

[JonnyBGod]

$ docker run --rm -it alpine:latest sh
/ # apk update
fetch http://dl-4.alpinelinux.org/alpine/v3.1/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-4.alpinelinux.org/alpine/v3.1/main: IO ERROR
WARNING: Ignoring APKINDEX.689bb31a.tar.gz: No such file or directory
OK: 15 distinct packages available
/ #

[JonnyBGod]

$ boot2docker config
# boot2docker profile filename: /Users/joaoribeiro/.boot2docker/profile
Init = false
Verbose = false
Driver = "virtualbox"
Clobber = true
ForceUpgradeDownload = false
SSH = "ssh"
SSHGen = "ssh-keygen"
SSHKey = "/Users/joaoribeiro/.ssh/id_boot2docker"
VM = "boot2docker-vm"
Dir = "/Users/joaoribeiro/.boot2docker"
ISOURL = "https://api.github.com/repos/boot2docker/boot2docker/releases"
ISO = "/Users/joaoribeiro/.boot2docker/boot2docker.iso"
DiskSize = 20000
Memory = 2048
SSHPort = 2022
DockerPort = 0
HostIP = "192.168.59.3"
DHCPIP = "192.168.59.99"
NetMask = [255, 255, 255, 0]
LowerIP = "192.168.59.103"
UpperIP = "192.168.59.254"
DHCPEnabled = true
Serial = false
SerialFile = "/Users/joaoribeiro/.boot2docker/boot2docker-vm.sock"
Waittime = 300
Retries = 75

[JonnyBGod]

further tests:

$ docker run --rm -it --net=host alpine:latest sh
/ # ping www.google.com
PING www.google.com (216.58.208.4): 56 data bytes
64 bytes from 216.58.208.4: seq=0 ttl=63 time=10.051 ms
64 bytes from 216.58.208.4: seq=1 ttl=63 time=11.970 ms
64 bytes from 216.58.208.4: seq=2 ttl=63 time=15.273 ms

$ docker run --rm -it alpine:latest sh
/ # ping www.google.com
ping: bad address 'www.google.com'
/ # exit

$ docker run --rm -it --net=host alpine:latest sh
/ # ping www.google.com
ping: bad address 'www.google.com'
/ # exit

For some reason the first time I used --net=host works but the second time did not work.

This happened to me before while building my Dockerfile. For the latest "20 something" tests for some reason two of them worked fine.

[andyshinn]

Try disabling the VirtualBox DNS NAT proxy per comment at boot2docker/boot2docker#451 (comment).

[JonnyBGod]

Joaos-MacBook-Pro:nginx joaoribeiro$ boot2docker down
Joaos-MacBook-Pro:nginx joaoribeiro$ VBoxManage modifyvm boot2docker-vm --natdnsproxy1 off --natdnshostresolver1 off
Joaos-MacBook-Pro:nginx joaoribeiro$ boot2docker up
Waiting for VM and Docker daemon to start...
...........ooo
Started.
Writing /Users/joaoribeiro/.boot2docker/certs/boot2docker-vm/ca.pem
Writing /Users/joaoribeiro/.boot2docker/certs/boot2docker-vm/cert.pem
Writing /Users/joaoribeiro/.boot2docker/certs/boot2docker-vm/key.pem
Your environment variables are already set correctly.

Joaos-MacBook-Pro:nginx joaoribeiro$ docker run --rm -it alpine:latest sh
/ # ping www.google.com
ping: bad address 'www.google.com'
/ # exit
Joaos-MacBook-Pro:nginx joaoribeiro$ docker run --rm -it --net=host alpine:latest sh
/ # ping www.google.com
PING www.google.com (216.58.208.4): 56 data bytes
64 bytes from 216.58.208.4: seq=0 ttl=63 time=8.780 ms
64 bytes from 216.58.208.4: seq=1 ttl=63 time=13.494 ms
64 bytes from 216.58.208.4: seq=2 ttl=63 time=9.012 ms
64 bytes from 216.58.208.4: seq=3 ttl=63 time=10.723 ms
64 bytes from 216.58.208.4: seq=4 ttl=63 time=10.632 ms
64 bytes from 216.58.208.4: seq=5 ttl=63 time=6.895 ms
64 bytes from 216.58.208.4: seq=6 ttl=63 time=7.814 ms
^C
--- www.google.com ping statistics ---
7 packets transmitted, 7 packets received, 0% packet loss
round-trip min/avg/max = 6.895/9.621/13.494 ms
/ # exit
Joaos-MacBook-Pro:nginx joaoribeiro$ docker run --rm -it --net=host alpine:latest sh
/ # ping www.google.com
ping: bad address 'www.google.com'
/ # exit
Joaos-MacBook-Pro:nginx joaoribeiro$ docker run --rm -it --net=host alpine:latest sh
/ # ping www.google.com
ping: bad address 'www.google.com'
/ # exit
Joaos-MacBook-Pro:nginx joaoribeiro$ boot2docker ssh cat /etc/resolv.conf
search lan
nameserver 192.168.1.254
nameserver 192.168.1.254
Joaos-MacBook-Pro:nginx joaoribeiro$ docker run --rm -it --net=host alpine:latest sh
/ # apk update
fetch http://dl-4.alpinelinux.org/alpine/v3.1/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-4.alpinelinux.org/alpine/v3.1/main: IO ERROR
WARNING: Ignoring APKINDEX.689bb31a.tar.gz: No such file or directory
OK: 15 distinct packages available
/ # exit

[JonnyBGod]

Updated docker and boot2docker to 1.6.0 and problem persists.

[andyshinn]

What is the output of docker run alpine cat /etc/resolve.conf? Can you ping the name server listed in the container resolv.conf?

[JonnyBGod]

Joaos-MacBook-Pro:nginx joaoribeiro$ docker run --rm -it alpine:latest sh
/ # cat etc/resolv.conf
search lan
nameserver 192.168.1.254
nameserver 192.168.1.254
/ # ping 192.168.1.254
PING 192.168.1.254 (192.168.1.254): 56 data bytes
64 bytes from 192.168.1.254: seq=0 ttl=61 time=4.289 ms
64 bytes from 192.168.1.254: seq=1 ttl=61 time=30.006 ms
64 bytes from 192.168.1.254: seq=2 ttl=61 time=11.134 ms
64 bytes from 192.168.1.254: seq=3 ttl=61 time=2.201 ms
64 bytes from 192.168.1.254: seq=4 ttl=61 time=12.041 ms
64 bytes from 192.168.1.254: seq=5 ttl=61 time=4.230 ms
64 bytes from 192.168.1.254: seq=6 ttl=61 time=11.325 ms
^C
--- 192.168.1.254 ping statistics ---
7 packets transmitted, 7 packets received, 0% packet loss
round-trip min/avg/max = 2.201/10.746/30.006 ms
/ # apk update
fetch http://dl-4.alpinelinux.org/alpine/v3.1/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-4.alpinelinux.org/alpine/v3.1/main: IO ERROR
WARNING: Ignoring APKINDEX.689bb31a.tar.gz: No such file or directory
OK: 15 distinct packages available
/ # exit
Joaos-MacBook-Pro:nginx joaoribeiro$ ping 192.168.1.254
PING 192.168.1.254 (192.168.1.254): 56 data bytes
64 bytes from 192.168.1.254: icmp_seq=0 ttl=64 time=2.869 ms
64 bytes from 192.168.1.254: icmp_seq=1 ttl=64 time=2.430 ms
64 bytes from 192.168.1.254: icmp_seq=2 ttl=64 time=12.737 ms
64 bytes from 192.168.1.254: icmp_seq=3 ttl=64 time=8.959 ms
64 bytes from 192.168.1.254: icmp_seq=4 ttl=64 time=2.469 ms
^C
--- 192.168.1.254 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 2.430/5.893/12.737/4.221 ms

[andyshinn]

Hmm. I know musl doesn't support domain or search in resolv.conf. Can
you try running it by manually specifying --dns 192.168.1.254?
On Apr 27, 2015 6:29 PM, "João Ribeiro" [email protected] wrote:

Joaos-MacBook-Pro:nginx joaoribeiro$ docker run --rm -it alpine:latest sh
/ # cat etc/resolv.conf
search lan
nameserver 192.168.1.254
nameserver 192.168.1.254
/ # ping 192.168.1.254
PING 192.168.1.254 (192.168.1.254): 56 data bytes
64 bytes from 192.168.1.254: seq=0 ttl=61 time=4.289 ms
64 bytes from 192.168.1.254: seq=1 ttl=61 time=30.006 ms
64 bytes from 192.168.1.254: seq=2 ttl=61 time=11.134 ms
64 bytes from 192.168.1.254: seq=3 ttl=61 time=2.201 ms
64 bytes from 192.168.1.254: seq=4 ttl=61 time=12.041 ms
64 bytes from 192.168.1.254: seq=5 ttl=61 time=4.230 ms
64 bytes from 192.168.1.254: seq=6 ttl=61 time=11.325 ms
^C
--- 192.168.1.254 ping statistics ---
7 packets transmitted, 7 packets received, 0% packet loss
round-trip min/avg/max = 2.201/10.746/30.006 ms
/ # exit
Joaos-MacBook-Pro:nginx joaoribeiro$ ping 192.168.1.254
PING 192.168.1.254 (192.168.1.254): 56 data bytes
64 bytes from 192.168.1.254: icmp_seq=0 ttl=64 time=2.869 ms
64 bytes from 192.168.1.254: icmp_seq=1 ttl=64 time=2.430 ms
64 bytes from 192.168.1.254: icmp_seq=2 ttl=64 time=12.737 ms
64 bytes from 192.168.1.254: icmp_seq=3 ttl=64 time=8.959 ms
64 bytes from 192.168.1.254: icmp_seq=4 ttl=64 time=2.469 ms
^C
--- 192.168.1.254 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 2.430/5.893/12.737/4.221 ms

—
Reply to this email directly or view it on GitHub
#28 (comment)
.

[JonnyBGod]

Works with --dns argument:

$ docker run --rm --dns 192.168.1.254 -it alpine:latest sh
/ # ping www.google.com
PING www.google.com (216.58.210.100): 56 data bytes
64 bytes from 216.58.210.100: seq=0 ttl=61 time=9.245 ms
64 bytes from 216.58.210.100: seq=1 ttl=61 time=7.852 ms
64 bytes from 216.58.210.100: seq=2 ttl=61 time=8.978 ms
64 bytes from 216.58.210.100: seq=3 ttl=61 time=17.184 ms
64 bytes from 216.58.210.100: seq=4 ttl=61 time=12.777 ms
^C
--- www.google.com ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 7.852/11.207/17.184 ms
/ # apk update
fetch http://dl-4.alpinelinux.org/alpine/v3.1/main/x86_64/APKINDEX.tar.gz
v3.1.3-44-g7991b61 [http://dl-4.alpinelinux.org/alpine/v3.1/main]
OK: 4868 distinct packages available

Should this be considered a bug/needed improvement or should we leave as is?

[JonnyBGod]

tried multiple times always with the same result. Also never had such problem while using any ubuntu or debian dist.

$ docker run --rm -it ubuntu sh
# ping www.google.com
PING www.google.com (216.58.210.100) 56(84) bytes of data.
64 bytes from lis01s14-in-f4.1e100.net (216.58.210.100): icmp_seq=1 ttl=61 time=13.7 ms
64 bytes from lis01s14-in-f4.1e100.net (216.58.210.100): icmp_seq=2 ttl=61 time=10.0 ms
64 bytes from lis01s14-in-f4.1e100.net (216.58.210.100): icmp_seq=3 ttl=61 time=17.2 ms
64 bytes from lis01s14-in-f4.1e100.net (216.58.210.100): icmp_seq=4 ttl=61 time=9.93 ms
64 bytes from lis01s14-in-f4.1e100.net (216.58.210.100): icmp_seq=5 ttl=61 time=7.16 ms
64 bytes from lis01s14-in-f4.1e100.net (216.58.210.100): icmp_seq=6 ttl=61 time=12.4 ms
^C
--- www.google.com ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5014ms
rtt min/avg/max/mdev = 7.162/11.786/17.260/3.219 ms
# apt-get update
Ign http://archive.ubuntu.com trusty InRelease
Ign http://archive.ubuntu.com trusty-updates InRelease
Ign http://archive.ubuntu.com trusty-security InRelease
Hit http://archive.ubuntu.com trusty Release.gpg
Get:1 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]
Get:2 http://archive.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://archive.ubuntu.com trusty Release
Get:3 http://archive.ubuntu.com trusty-updates Release [63.5 kB]
Get:4 http://archive.ubuntu.com trusty-security Release [63.5 kB]
Get:5 http://archive.ubuntu.com trusty/main Sources [1335 kB]
Get:6 http://archive.ubuntu.com trusty/restricted Sources [5335 B]
Get:7 http://archive.ubuntu.com trusty/universe Sources [7926 kB]
Get:8 http://archive.ubuntu.com trusty/main amd64 Packages [1743 kB]
Get:9 http://archive.ubuntu.com trusty/restricted amd64 Packages [16.0 kB]
Get:10 http://archive.ubuntu.com trusty/universe amd64 Packages [7589 kB]
Get:11 http://archive.ubuntu.com trusty-updates/main Sources [248 kB]
Get:12 http://archive.ubuntu.com trusty-updates/restricted Sources [2310 B]
Get:13 http://archive.ubuntu.com trusty-updates/universe Sources [141 kB]
Get:14 http://archive.ubuntu.com trusty-updates/main amd64 Packages [636 kB]
Get:15 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [15.1 kB]
Get:16 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [356 kB]
Get:17 http://archive.ubuntu.com trusty-security/main Sources [97.9 kB]
Get:18 http://archive.ubuntu.com trusty-security/restricted Sources [1874 B]
Get:19 http://archive.ubuntu.com trusty-security/universe Sources [23.3 kB]
Get:20 http://archive.ubuntu.com trusty-security/main amd64 Packages [329 kB]
Get:21 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [14.8 kB]
Get:22 http://archive.ubuntu.com trusty-security/universe amd64 Packages [128 kB]
Fetched 20.7 MB in 7s (2624 kB/s)
Reading package lists... Done
# cat etc/resolv.conf
search lan
nameserver 192.168.1.254
nameserver 192.168.1.254

[andyshinn]

I don't have boot2docker and can't reproduce it on my VM. I am suspecting it has something to do with musl libc querying the same name server in parallel. Can you edit the resolv.conf on the boot2docker host to only have one name server and try again?

[delitescere]

I think it's IPv6 preferred over IPv4 issue. See #11 (comment)

And guess what uses glibc?...

$ docker run --rm -ti alpine sh
/ # cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.0.2.3
search canva.com
options single-request-reopen
/ # 

[andyshinn]

Doesn't that resolv.conf just come from the boot2docker host (which is glibc based)?

[andyshinn]

@delitescere have you tried out boot2docker/boot2docker#451 (comment)? FWIW, I've had intermittent issues like this on other non-Alpine images before as well. They ended up being related to VirtualBox DNS.

[delitescere]

Not using boot2docker (but am on Ubuntu)

---

From the phone

On 1 May 2015, at 01:50, Andy Shinn [email protected] wrote:

Doesn't that resolv.conf just come from the boot2docker host (which is glibc based)?

—
Reply to this email directly or view it on GitHub.

[delitescere]

Will take a look.

This only happens on this latest Alpine base box Docker image.

BTW don't throw away old versions and use good tags for maximum adoption.

---

From the phone

On 1 May 2015, at 02:09, Andy Shinn [email protected] wrote:

@delitescere have you tried out boot2docker/boot2docker#451 (comment)? FWIW, I've had intermittent issues like this on other non-Alpine images before ass well. They ended up being related to VirtualBox DNS.

—
Reply to this email directly or view it on GitHub.

[delitescere]

Yes, boot2docker/boot2docker#451 (comment) works totally works fine when I'm not using a DNS proxy in VirtualBox (even when not using boot2docker).

I wonder why Alpine only experiences the issue. 😕

Thanks for the pointer!

[andyshinn]

@delitescere @JonnyBGod is this still an issue? If so, can you try on alpine:edge? It has a newer musl libc version. I think this is just a bug with the way musl and VirtualBox work together. I'm not sure if there is anything we can do about it. I've had the same issues on other containers (Ubuntu) in boot2docker as well.

[delitescere]

It's good for me if I disable VirtualBox's DNS shenanigans.

This only happens to me with Alpine. Not BusyBox and not Ubuntu.

Still, a big NB in the docs for VirtualBox / boot2docker users might be a good idea.

[jckimble]

by any chance have you tried adding
echo "ipv6" >> /etc/modules
before you run apk update, I added it to my dockerfile and it started working, If it fixes it I'll do a pull request for the Dockerfile adding it

[delitescere]

I just retried with alpine:3.2 and all still seems well. I have still disabled the VirtualBox DNS intermediary, and I'm on VirtualBox 5.0.2 now.

@jckimble I haven't added that but seems like a decent idea. /etc/modules contained just af_packet.

[jckimble]

@delitescere I know adding ipv6 to /etc/modules fixed mine and why it did it, on mine docker0 is given a ipv6 address if I rmmod ipv6 docker0 is given a ipv4 address and it works with no problems, then I loaded the apline image using 'docker run -it gliderlabs/alpine ping www.google.com' and it couldn't resolve it, after that I tried 'docker run -it gliderlabs/alpine /bin/sh' and did 'echo "ipv6" > /etc/modules && ping www.google.com' and it worked. I also did a shell and ran the ping before and after adding it. So I'm about 90% sure thats the dns resolv problem with this image but I want to make sure i'm not wrong and its not a fluke from me using archlinux.

[jumanjiman]

A couple months ago we started noticing odd behavior in our infrastructure related to DNS.
We use BIND at the edge of the network, and these BIND servers forward queries for external names.

Recently we started receiving CNAME answers where the corresponding answer contained an ipv6 address. We don't route outbound ipv6, so our internal hosts were unable to "talk" to these external services.

We tried two things:

• start named -4 to limit to ipv4, but that only limits bind network traffic, not the query results content
• add to named.conf these lines, which resolved the problem for us:

// Strip AAAA records from forwarded queries, even if it breaks dnssec.
filter-aaaa-on-v4 break-dnssec;
filter-aaaa { any; };

This may or may not be related to the issues you're seeing.
dig can help you investigate further (as opposed to ping).

[jckimble]

@jumanjiman what your describing is at the network between a docker host and its network, the bug that I found the fix for is for dockers internal networking with alpine which in this case the docker host is playing router with docker0 and if docker0 doesn't have an ipv4 address its like connecting a ipv4 computer to a router that only accepts ipv6. full distrobutions have autodetection for problems like this while minmalist distros don't have autodetection like udev cause it adds alot of space.

[jumanjiman]

fair enough. thx for the clarification.

[andyshinn]

I've gone over this issue again and am not sure if there is any actionable item for the Alpine Docker image itself. Can someone clarify if there was a needed changed to the image to prevent this? The echo "ipv6" >> /etc/modules command is the only thing I can see here that may or may not be fix.

If this is just an intermittent VirtualBox issue, do we all think this can be closed for now? The issue is a little convoluted and I'd rather start a new one if this is still an issue.

[delitescere]

That sounds fine Andy, thanks.

Everything seems to be working without workarounds now. No change to /etc/modules and no change to VirtualBox DNS NAT settings.

Virtualbox: 5.0.2-102096
Ubuntu: Linux vagrant-ubuntu-trusty-64 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Docker: 1.8.1, build d12ea79
Alpine docker image: gliderlabs/alpine 3.2 5bd56d81884

Software, how does it even work?? ¯\_(ツ)_/¯