Offer Version of Boxes without Hardcoded DNS?

[lavabit]

Offer Version of Boxes without Hardcoded DNS?

[ladar]

@lberk I'd be happy to merge in improvements. My philosophy is that the default values I specify in the various scripts should be used in the absence of a user defined value. The trick will be figuring a mechanism which makes it easy to override those values, and then ensuring the box config defers appropriately. Ideally someone can override the values using the Vagrantfile, and we add documentation to the roboxes.org website, which shows people how to override features.

Like I said, I haven't been able to test every combination, so I admittedly deviate from this ideal.

As for the DNS servers specifically, I'm on the fence about whether DHCP values qualify as "user provided input" ... since in many cases DHCP values are outside the user's control.

As I mentioned, I went the current route, because I needed to ensure the magma boxes booted with sane DNS servers. If they didn't various unit tests (curl in particular) would fail because,aand I'd waste hours on failed builds, and time debugging, only to realize the failure was because the VM used the DNS servers for XYZ ISP, or coffee shop.

That said, I could loosen the rules for the generic boxes, and include extra rules for the magma specific builds.

Thoughts?

[ladar]

I keep coming back to the idea that this should be configurable via the Vagrantfile, using something like:

https://www.vagrantup.com/docs/networking/public_network.html#dhcp

If it isn't, then perhaps the solution is to mimic the feature using an inline provisioning script, using the embedded Vagrantfile? The script could use environment variables, or Vagrantfile file values, to configure things like the DNS servers during the box provisioning stage... If someone disables the provisioning step, or doesn't provide values, then the defaults would be applied...

On my long term to do list is writing a generic bash script that does something along these lines. Specifically I want to embed a script which detects how Vagrant (or the user) connects. If the first login is connected via SSH, using an ephemeral key pair, then the script would either disable password logins via ssh, and/or, randomize the vagrant user password. If Vagrant or a user connects for the first time using the console, and/or with the default password, then the script could require a password change, and/or, remove the default vagrant ssh key.

I fear people don't realize that by convention, these credentials have known defaults, and might be leaving them intact. Which could be a problem, if a box happens to be get provisioned with an IP that isn't NAT'ed.

Long story short, if we went this route, then other things could also be easily configured, like the DNS servers, I'd welcome a PR along thse lines... thoughts?

[ladar]

I should add, IPv6 enable/disable should be easy to control via the same mechanism.

[lberk]

Thanks for your input. I think I agree in general with your approach and the underlying principles. But I keep hitting a road block (this is probably a case of my ruby/vagrant/networking knowledge being slightly too basic for this).

I've taken a bit of time to try and add the option in the underlying robox/tpl/$provider-$distro.rb vagrant template as you've suggested, something similar to;

if ENV.has_key?("robox_dns")
  config.vm.provision "shell", run: "always", inline: <<-SHELL
  sed -i -e "s/#DNS=.*/DNS="+ENV['robox_dns']+"/g" /etc/systemd/resolved.conf
  SHELL
else
  config.vm.provision "shell", run: "always", inline: <<-SHELL
  sed -i -e "s/#DNS=.*/DNS=4.2.2.1 4.2.2.2 208.67.220.220/g" /etc/systemd/resolved.conf
  SHELL
end

That way, if the environment variable isn't declared, we default to your known-sane value. If it is declared, change to whatever is specified (it'd be worthwhile for my use-case to perhaps have a 'none' special case where no extra sed line happens.

Is that what you were envisioning?

The issue I'm running in to with this approach (and again, if you have any pointers, I'd appreciate it and am willing to try incorporating them!) is that this kind of inline provisioning script seems to be run after the box is provisioned with default values. This means, when I run vagrant up on the box with altered dns, it still fails at
==> default: Waiting for domain to get an IP address....
which I assume (could be wrong) that it's related to the dns issue?

This is why I come back to having the 'generic' namespaces as slightly more permissive for the networking; just so I can stand the box up and provision them at all. That being said, I'm aware we've got huge matrix of potential config options people will be wanting to tweak, and its a balancing act trying to make these options configurable, yet testable to ensure functionality.

Thoughts?

[ladar]

You're script looks right. I'd go with a environment variables of NS1, NS2, and NS3. If any are defined, they're configured. Presumably we'll need to adapt the basic strategy to the different distros, and having the values separate should make that easier.

As for your issue. Which hypervisor are you using? Is it libvirt? I believe all of the boxes are configured, by default to bring up a preconfigured NIC that uses DHCP. Based on the error, it feels like the box is stuck waiting for a DHCP address?

In the world of libvirt, a domain is synonymous with guest virtual machine.

[ladar]

@lberk can you do a little test for me. When you see the error, can you still connect to the guest console, and login using vagrant:vagrant. Once you do that, you can run a few tests. The ifconfig command will show what NICs are configured, and if they have an IP address assigned. The ping 4.2.2.1 will tell if the machines have internet access, and ping lavabit.com will tell you if the resolver is working. (Depending on the distro, nslookup or dig should also work.)

Those commands should be installed, but I haven't verified it with every variant, so if they aren't let me know and I'll add the required package to the baseline configuration for that distro.

[lberk]

@ladar oops, wrong error message I reported (wrong box)

for example, on generic/ubuntu1804

$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.121.195  netmask 255.255.255.0  broadcast 192.168.121.255
        inet6 fe80::5054:ff:fea5:7e71  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:a5:7e:71  txqueuelen 1000  (Ethernet)
        RX packets 15468  bytes 178887599 (178.8 MB)
        RX errors 0  dropped 11  overruns 0  frame 0
        TX packets 9136  bytes 926253 (926.2 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.100.10.24  netmask 255.255.255.0  broadcast 10.100.10.255
        ether 52:54:00:99:e1:e3  txqueuelen 1000  (Ethernet)
        RX packets 507  bytes 43502 (43.5 KB)
        RX errors 0  dropped 10  overruns 0  frame 0
        TX packets 1088  bytes 119879 (119.8 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 1444  bytes 103727 (103.7 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1444  bytes 103727 (103.7 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

$ ping -c 3 4.2.2.1
PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data.
64 bytes from 4.2.2.1: icmp_seq=1 ttl=52 time=18.9 ms
64 bytes from 4.2.2.1: icmp_seq=2 ttl=52 time=27.2 ms
64 bytes from 4.2.2.1: icmp_seq=3 ttl=52 time=18.7 ms

--- 4.2.2.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 18.792/21.657/27.279/3.977 ms

$ ping -c 3 lavabit.com
ping: lavabit.com: Temporary failure in name resolution

I can change the ruby script to use those env var names instead. Is there a better way to include them? than by adding them directly to tpl/$provider-$distro.rb ? I'd prefer to have them organised similar to how you have the scripts/ directory right now that create giant, monolithic base vagrantfiles.

[ladar]

What you posted makes very little sense to me. All I can think of, is that perhaps the additional NIC (aka eth1 is getting created by Vagrant (or some other rule) and that is what is causing problems with your DNS resolution.

Specifically, your post shows you can access the box, and the box can access NS1 (aka 4.2.2.1), but that it can't resolve the DNS name lavabit.com. Can you run two more commands:

systemd-resolve --status

and

nslookup lavabit.com

You should see something like the following:

[ladar@darla vagrant]$ ./connect.sh generic ubuntu1804 libvirt
Installing the 'vagrant-libvirt' plugin. This can take a few minutes...
Fetching: formatador-0.2.5.gem (100%)
Fetching: excon-0.62.0.gem (100%)
Fetching: fog-core-1.43.0.gem (100%)
Fetching: mini_portile2-2.4.0.gem (100%)
Fetching: nokogiri-1.10.1.gem (100%)
Building native extensions.  This could take a while...
Fetching: ruby-libvirt-0.7.1.gem (100%)
Building native extensions.  This could take a while...
Fetching: fog-xml-0.1.3.gem (100%)
Fetching: fog-json-1.2.0.gem (100%)
Fetching: fog-libvirt-0.6.0.gem (100%)
Fetching: vagrant-libvirt-0.0.45.gem (100%)
Installed the plugin 'vagrant-libvirt (0.0.45)'!
==> box: Loading metadata for box 'generic/ubuntu1804'
    box: URL: https://vagrantcloud.com/generic/ubuntu1804
==> box: Adding box 'generic/ubuntu1804' (v1.8.60) for provider: libvirt
    box: Downloading: https://vagrantcloud.com/generic/boxes/ubuntu1804/versions/1.8.60/providers/libvirt.box
    box: Download redirected to host: vagrantcloud-files-production.s3.amazonaws.com
==> box: Successfully added box 'generic/ubuntu1804' (v1.8.60) for 'libvirt'!
Bringing machine 'default' up with 'libvirt' provider...
==> default: Checking if box 'generic/ubuntu1804' version '1.8.60' is up to date...
==> default: Uploading base box image as volume into libvirt storage...
==> default: Creating image (snapshot of base box volume).
==> default: Creating domain with the following settings...
==> default:  -- Name:              ubuntu1804-libvirt_default
==> default:  -- Domain type:       kvm
==> default:  -- Cpus:              2
==> default:  -- Feature:           acpi
==> default:  -- Feature:           apic
==> default:  -- Feature:           pae
==> default:  -- Memory:            2048M
==> default:  -- Management MAC:    
==> default:  -- Loader:            
==> default:  -- Nvram:             
==> default:  -- Base box:          generic/ubuntu1804
==> default:  -- Storage pool:      default
==> default:  -- Image:             /var/lib/libvirt/images/ubuntu1804-libvirt_default.img (32G)
==> default:  -- Volume Cache:      default
==> default:  -- Kernel:            
==> default:  -- Initrd:            
==> default:  -- Graphics Type:     vnc
==> default:  -- Graphics Port:     -1
==> default:  -- Graphics IP:       127.0.0.1
==> default:  -- Graphics Password: Not defined
==> default:  -- Video Type:        cirrus
==> default:  -- Video VRAM:        256
==> default:  -- Sound Type:	
==> default:  -- Keymap:            en-us
==> default:  -- TPM Path:          
==> default:  -- INPUT:             type=mouse, bus=ps2
==> default: Creating shared folders metadata...
==> default: Starting domain.
==> default: Waiting for domain to get an IP address...
==> default: Waiting for SSH to become available...
    default: 
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default: 
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Configuring and enabling network interfaces...
vagrant@ubuntu1804:~$ nslookup lavabit.com
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	lavabit.com
Address: 38.107.241.66

vagrant@ubuntu1804:~$ cat /etc/resolv.conf 
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
vagrant@ubuntu1804:~$ systemd-resolve --status
Global
         DNS Servers: 4.2.2.1
                      4.2.2.2
                      208.67.220.220
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 2 (eth0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: yes
    DNSSEC supported: yes
         DNS Servers: 4.2.2.1
                      4.2.2.2
                      208.67.220.220
                      192.168.121.1
vagrant@ubuntu1804:~$ ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.121.128  netmask 255.255.255.0  broadcast 192.168.121.255
        inet6 fe80::5054:ff:feed:6a84  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:ed:6a:84  txqueuelen 1000  (Ethernet)
        RX packets 604  bytes 101891 (101.8 KB)
        RX errors 0  dropped 49  overruns 0  frame 0
        TX packets 614  bytes 104313 (104.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 185  bytes 12281 (12.2 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 185  bytes 12281 (12.2 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[ladar]

@lberk in reply to your comment... if we're speaking ideally, then users should be to set a value like:

config.vm.nameservers = ['1.1.1.1','2.2.2.2']

or

config.vm.nameservers = auto

Or dhcp if they want to use the DHCP provided nameservers, as you suggested:

config.vm.nameservers = none

For no DNS servers. Of course the absence of a value, or setting:

config.vm.nameservers = default

Would use the existing values. I just don't have know if a provisioning script, inside the bundled Vagrantfile could read/react using those settings. Ideally we should open up a ticket with Vagrant, and get them to add support for that. Short term we might be able to fake support for something like this by using inline script like you proposed, inside the bundled Vagrantfile.

The scripts/ files get executed at build time, which wouldn't help you, right? What you want is to be able download the box, and have it use custom NS values during the provisioning step, right?

The only way to do that is by adding scripts to the tpl/ files. And the hard part, once we get it working, will be creating variants for the 5-8 different DNS config schemes used by the various distros.

[lberk]

Would be happy to try and paste the output of any other commands (is there a better place to go over this stuff than github?)

$ nslookup lavabit.com
;; connection timed out; no servers could be reached

$ nslookup lavabit.com

Server:         127.0.0.53
Address:        127.0.0.53#53

** server can't find lavabit.com: SERVFAIL

$ cat /etc/resolv.conf 
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
$ systemd-resolve --status
Global
         DNS Servers: 4.2.2.1
                      4.2.2.2
                      208.67.220.220
                      192.168.121.1
                      10.100.10.1
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 3 (eth1)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: yes
    DNSSEC supported: yes

Link 2 (eth0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: yes
    DNSSEC supported: yes
         DNS Servers: 4.2.2.1
                      4.2.2.2
                      208.67.220.220
                      192.168.121.1
$ ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.121.27  netmask 255.255.255.0  broadcast 192.168.121.255
        inet6 fe80::5054:ff:fe11:225d  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:11:22:5d  txqueuelen 1000  (Ethernet)
        RX packets 15233  bytes 178873309 (178.8 MB)
        RX errors 0  dropped 11  overruns 0  frame 0
        TX packets 8756  bytes 931727 (931.7 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.100.10.24  netmask 255.255.255.0  broadcast 10.100.10.255
        ether 52:54:00:3b:ac:44  txqueuelen 1000  (Ethernet)
        RX packets 300  bytes 27412 (27.4 KB)
        RX errors 0  dropped 12  overruns 0  frame 0
        TX packets 626  bytes 69234 (69.2 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 920  bytes 67758 (67.7 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 920  bytes 67758 (67.7 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[ladar]

@lberk just to confirm. The box is provisioning properly, you're just concerned that DNS doesn't work properly inside the guest, correct?

The resolv.conf, nslookup results look sane. The guest is setting up a resolver proxy, as it should, so using 127.0.0.1 is fine.

The the source of the your troubles appears to be the systemd-resolve daemon. My guess, the daemon has the default resolvers loaded first, and something on your network is blocking UDP access over port 53, but lets confirm that theory:

ping 4.2.2.1
nslookup lavabit.com 4.2.2.1

Will confirm IP connectivity, and then try using that name server to resolve the query. Let's also confirm that the DHCP provided name servers are both correct:

ping 192.168.121.1
nslookup lavabit.com 192.168.121.1

and...

ping 10.100.10.1
nslookup lavabit.com 10.100.10.1

My guess is based on the idea that usually resolvers only search the first 3 nameservers, and ignore the rest. If that's the case, then this should be an easy fix.

[lberk]

@ladar ah awesome, thanks!
As for the output;

$ ping 4.2.2.1
PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data.
64 bytes from 4.2.2.1: icmp_seq=1 ttl=52 time=18.9 ms
64 bytes from 4.2.2.1: icmp_seq=2 ttl=52 time=18.9 ms

--- 4.2.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 18.932/18.935/18.939/0.137 ms

$ nslookup lavabit.com 4.2.2.1
;; connection timed out; no servers could be reached

$ ping -c 1 192.168.121.1
PING 192.168.121.1 (192.168.121.1) 56(84) bytes of data.
64 bytes from 192.168.121.1: icmp_seq=1 ttl=64 time=0.151 ms

--- 192.168.121.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.151/0.151/0.151/0.000 ms

$ nslookup lavabit.com 192.168.121.1
Server:         192.168.121.1
Address:        192.168.121.1#53

Non-authoritative answer:
Name:   lavabit.com
Address: 38.107.241.66

$ ping -c1 10.100.10.1
PING 10.100.10.1 (10.100.10.1) 56(84) bytes of data.
64 bytes from 10.100.10.1: icmp_seq=1 ttl=64 time=0.195 ms

--- 10.100.10.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.195/0.195/0.195/0.000 ms

$ nslookup lavabit.com 10.100.10.1
Server:         10.100.10.1
Address:        10.100.10.1#53

Non-authoritative answer:
Name:   lavabit.com
Address: 38.107.241.66

If this is something I can fix solely on our end things, that would be happy to implement if you have any suggestions.

[ladar]

I was correct. Your network is blocking access to outside nameservers. Because the first three entries in the list are outside your network, and thus blocked, DNS is failing. You either need to unblock access to outside DNS servers (UDP port 53), or update the guest so it doesn't use pre-configured values. I can't easily test a fix where I am, but if I were to guess, something like the following might work (for Ubuntu 18.04):

sudo sed -i "/nameservers/d" /etc/netplan/01-netcfg.yaml
sudo sed -i "/addresses/d" /etc/netplan/01-netcfg.yaml
sudo netplan generate
sudo systemctl restart systemd-networkd.service

As discussed above, it would be nice if there was a standardized way of setting nameservers via the Vagrantfile, but that will take some work.

[brokep]

@lberk I'd be happy to merge in improvements. My philosophy is that the default values I specify in the various scripts should be used in the absence of a user defined value. The trick will be figuring a mechanism which makes it easy to override those values, and then ensuring the box config defers appropriately. Ideally someone can override the values using the Vagrantfile, and we add documentation to the roboxes.org website, which shows people how to override features.

Like I said, I haven't been able to test every combination, so I admittedly deviate from this ideal.

As for the DNS servers specifically, I'm on the fence about whether DHCP values qualify as "user provided input" ... since in many cases DHCP values are outside the user's control.

As I mentioned, I went the current route, because I needed to ensure the magma boxes booted with sane DNS servers. If they didn't various unit tests (curl in particular) would fail because,aand I'd waste hours on failed builds, and time debugging, only to realize the failure was because the VM used the DNS servers for XYZ ISP, or coffee shop.

That said, I could loosen the rules for the generic boxes, and include extra rules for the magma specific builds.

Thoughts?

I think that would be a great idea. I personally take advantage of the generic boxes for nearly all my Evilpot / Honeypot fork.

[ladar]

@lberk did adding those commands to your Vagrantfile solve your problem?

[lberk]

@ladar Unfortunately not, output of the commands in #35 (comment) were the exact same as well. Any chance there should be some changes in resolved as well?

[ladar]

@lberk the following seemed to work for me:

sudo sed -i "/nameservers/d" /etc/netplan/01-netcfg.yaml
sudo sed -i "/addresses/d" /etc/netplan/01-netcfg.yaml
sudo sed -i "s/DNS=.*/DNS=/g" /etc/systemd/resolved.conf
sudo netplan generate
sudo netplan apply
sudo systemctl restart systemd-networkd.service systemd-resolved.service

[lberk]

@ladar unfortunately that still doesn't seem to work for me (either running as a provisioning script to the box, or running by hand). The ping/nslookup command sequence result was the exact same as before as well.

[lberk]

@ladar fwliw, this is partly why I was originally motivated to create our own boxes without the dns/network modifications. I'm still open to working on this particular issue, but could we also approach changing how we modify the base vagrantboxes in parallel to this so that different/specific dns servers could be specified at runtime instead of build time?

[ladar]

could we also approach changing how we modify the base vagrantboxes in parallel to this so that different/specific dns servers could be specified at runtime instead of build time?

@lberk yes... there are several ways of making this work, but if you have the time, then please submit a PR which does this.

I'm not quite sure why you are having so much trouble getting the resolver setup properly... but the key piece of data I need isn't whether ping works. It's the output from systemd-resolve --status after you run those commands, so I can see how the config was modified. And if it fails, any subsequent errors which might shed a clue, from your /var/log/syslog file.

Try run the following, and if it doesn't work, send the output:

vagrant up && vagrant ssh -c 'sudo sed -i "/nameservers/d" /etc/netplan/01-netcfg.yaml ; sudo sed -i "/addresses/d" /etc/netplan/01-netcfg.yaml ; sudo sed -i "s/DNS=.*/DNS=/g" /etc/systemd/resolved.conf ; sudo sed -i "s/DNSSEC=.*/DNSSEC=no/g" /etc/systemd/resolved.conf ; sudo netplan generate ; sudo netplan apply ; sudo systemctl restart systemd-networkd.service systemd-resolved.service ; systemd-resolve --status --no-pager ; ping -c1 lavabit.com || sudo tail -30 /var/log/syslog'

Those commands should remove the global resolvers, and switch to using the DHCP provided values, which is what I think you want. On my system the output is:

Bringing machine 'default' up with 'libvirt' provider...
==> default: Checking if box 'generic/ubuntu1804' version '1.8.60' is up to date...
==> default: Creating image (snapshot of base box volume).
==> default: Creating domain with the following settings...
==> default:  -- Name:              ubuntu1804-libvirt_default
==> default:  -- Domain type:       kvm
==> default:  -- Cpus:              2
==> default:  -- Feature:           acpi
==> default:  -- Feature:           apic
==> default:  -- Feature:           pae
==> default:  -- Memory:            2048M
==> default:  -- Management MAC:    
==> default:  -- Loader:            
==> default:  -- Nvram:             
==> default:  -- Base box:          generic/ubuntu1804
==> default:  -- Storage pool:      default
==> default:  -- Image:             /var/lib/libvirt/images/ubuntu1804-libvirt_default.img (32G)
==> default:  -- Volume Cache:      default
==> default:  -- Kernel:            
==> default:  -- Initrd:            
==> default:  -- Graphics Type:     vnc
==> default:  -- Graphics Port:     -1
==> default:  -- Graphics IP:       127.0.0.1
==> default:  -- Graphics Password: Not defined
==> default:  -- Video Type:        cirrus
==> default:  -- Video VRAM:        256
==> default:  -- Sound Type:	
==> default:  -- Keymap:            en-us
==> default:  -- TPM Path:          
==> default:  -- INPUT:             type=mouse, bus=ps2
==> default: Creating shared folders metadata...
==> default: Starting domain.
==> default: Waiting for domain to get an IP address...
==> default: Waiting for SSH to become available...
    default: 
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default: 
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Configuring and enabling network interfaces...
Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 2 (eth0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.121.1
PING lavabit.com (38.107.241.66) 56(84) bytes of data.
64 bytes from lavabit.com (38.107.241.66): icmp_seq=1 ttl=54 time=8.10 ms

--- lavabit.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 8.103/8.103/8.103/0.000 ms
Connection to 192.168.121.103 closed.

The syslog file only gets printed if ping fails.

Also, I noticed while investigating this further that when I switched to the DHCP provided nameserver, I had to also disable DNSSEC, or it lookups would fail. Disabling DNSSEC wasn't part of my original script, and I suspect this could be the source of your problem. It seems that the default libvirt scheme is to use a DNS proxy, which doesn't work when DNSSEC is enabled.

If those commands still don't work, it might help to look at your Vagrantfile. If it isn't sensitive, can you share it with me via email?

[ladar]

@lberk were you able to test my commands, and confirm that it was actually the DNSSEC requirement that was causing issues?

[lberk]

@ladar thanks for the prod. that worked! I'd be happy to to share the vagrantfile, I'm just not sure which email you'd like it forwarded to?

[lberk]

Actually, that's not needed, the vagrantfile is completely public; https://github.com/performancecopilot/pcp/blob/master/Vagrantfile
the provisioning scripts are under; https://github.com/performancecopilot/pcp/tree/master/provisioning

[ladar]

@lberk to summarize then, the issue had nothing to do with which resolvers were being used... but rather, when using the default libvirt DNS configuration (which seems to be a local proxy), DNS queries fail because the DNSSEC data isn't being provided to the guest... which is completely separate from making it easier for end users to override the pre-configured default networking values (DNS/IPv6).

Any suggestions for a permanent fix? I don't think disabling DNSSEC validation is a good choice...

If I don't hear anything, I'll close this issue until someone decides to take up the challenge.

[lberk]

@ladar thanks for helping diagnose it.

I'll have to dig into my libvirt setup a bit more then to see if there's a more surgical/local fix I can apply.

Until then, I'll close the issue, thanks again for the help!

[ladar]

You might want to dig into the man page for dnsmasq ... which I believe, is what libvirt uses to handle DHCP requests, and proxy/relay DNS queries. At least on my system, there are various DNSSEC options. It might also be worth looking at your /etc/dnsmasq.conf file (on CentOS/RHEL).

It isn't immediately clear whether dnsmasq is validating results, and setting the authenticated bit, nor why it would fail when being queried for DNSSEC record types. My best guess is the dnsmasq DNS proxy doesn't understand the record types used by DNSSEC records, so the queries are failing. And it does this because it expects something higher in the resolver stack to validate the query.

[ekohl]

I have the same issue. The network firewall doesn't allow connecting external nameservers and you have to go through the provided nameservers. My vagrant (with libvirt) provides the correct nameservers via DHCP and with the generic image I expected it to honor my settings.

Another downside is that my local dnsmasq can resolve all the hostnames for all VMs I'm running. That means I can use DNS to set up connections to other VMs. Using external nameservers breaks that feature.

As a user of the generic image I expected a very vanilla Ubuntu. I'm currently using this because Ubuntu only offers Virtualbox images.

Note I tested this with with the 16.04 image now. The workaround I use:

sed -i 's/allow-hotplug eth0/auto eth0/ ; /^dns-nameserver/d ; /^pre-up sleep 2$/d' /etc/network/interfaces && systemctl restart networking

Edit: in 18.04 I have no issue because systemd-resolved detects it can't reach the external nameservers and uses the DHCP provided ones.

[ladar]

@ekohl the best solution to this problem is to ask Vagrant to add the ability to configure nameservers via the Vagrantfile, and then have the guest plugin make the configuration changes.

In lieu of that, there really is no perfect answer. Ideally the default nameservers should get used unless the user overrides them. Whether the DHCP provided values consist of an override is debatable.

The biggest issue is that the default values don't work by default on some systems/platforms. I think this is primarily a libvirt concern, since it seems libvirt uses dnsmasq to proxy requests, and I suspect block queries directly to default servers, by default. I've considered opening an issue on the vagrant-libvirt repo to discuss, and find out more. If that's the case, then at least for the libvirt images it would make sense to prioritize the DHCP provided values.

If you read the ticket above, the problem wasn't requests being blocked, it was the proxied requests not honoring the dnssec requirement. That needs further investigation as well.

I'm open to suggestions for fixing this. I'm just not sure building a separate box for every distro without default DNS server values is viable. It simply takes too long to build all the images once (48+ hours).

[ladar]

P.S. @ekohl I find it curious that for you, 16.04 doesn't work, but 18.04 does. If you look at the original post, that was for an 18.04 configuration. Again, I think this speaks to this issue being platform dependent.

[ekohl]

In lieu of that, there really is no perfect answer. Ideally the default nameservers should get used unless the user overrides them. Whether the DHCP provided values consist of an override is debatable.

IMHO default nameservers are what DHCP provides. If your company uses DNS nameservers that are only reachable internally then external nameservers can't resolve those either. It can also be a privacy concern.

The biggest issue is that the default values don't work by default on some systems/platforms. I think this is primarily a libvirt concern, since it seems libvirt uses dnsmasq to proxy requests, and I suspect block queries directly to default servers, by default. I've considered opening an issue on the vagrant-libvirt repo to discuss, and find out more. If that's the case, then at least for the libvirt images it would make sense to prioritize the DHCP provided values.

In my experience libvirt just works by default if you use DHCP provided values. Also, in my case it's not the my host blocking it. From my host without any virtualization I can't do any DNS queries to port 53 either.

I'm open to suggestions for fixing this. I'm just not sure building a separate box for every distro without default DNS server values is viable. It simply takes too long to build all the images once (48+ hours).

I guess this comes down to the choice you have to make: do you assume broken-by-default or working-by-default. The current model is broken-by-default but I'd argue this punished users who have a working environment. I can respect that you have a difficult decision to make and you'll never please all users.

P.S. @ekohl I find it curious that for you, 16.04 doesn't work, but 18.04 does. If you look at the original post, that was for an 18.04 configuration. Again, I think this speaks to this issue being platform dependent.

This can be explained. 16.04 uses resolv.conf. By default that only reads the first 3 nameserver entries (from man resolv.conf: Up to MAXNS (currently 3, see <resolv.h>) name servers may be listed, one per keyword). Since the images have 3 hardcoded DNS servers, the DHCP provided servers are never consulted. systemd-resolved as used in 18.04 doesn't have this limit so it uses the DHCP provided servers again.

Note that when using systemd-resolved it may be cleaner to use FallbackDNS in /etc/systemd/resolved.conf so you still prefer the on-network provided DNS servers.

[ladar]

IMHO default nameservers are what DHCP provides. If your company uses DNS nameservers that are only reachable internally then external nameservers can't resolve those either. It can also be a privacy concern.

Privacy is one of the concerns. But that's why I picked the Level 3 resolvers, with OpenDNS as a backup. As opposed to whatever DNS servers might be handed out via DHCP at the coffee shop, hotel wifi, etc I might be at.

In my experience libvirt just works by default if you use DHCP provided values. Also, in my case it's not the my host blocking it. From my host without any virtualization I can't do any DNS queries to port 53 either.

Again, it seems your environment is different from the original poster, and both are different than me.

I guess this comes down to the choice you have to make: do you assume broken-by-default or working-by-default. The current model is broken-by-default but I'd argue this punished users who have a working environment. I can respect that you have a difficult decision to make and you'll never please all users.

The goal is most certainly to work by default. These images were all originally created to testing magmad ... and part of that testing process included testing the bundled dependencies. Some of those dependencies have unit tests which ensure non-existent domains fail properly. I found that when I didn't specify the name servers, I'd freqently find myself pointed at resolvers that take non-existent domains and redirect them to marketing websites, thus breaking the test cases, and triggering a failure which required investigation, etc.

This can be explained. 16.04 uses resolv.conf. By default that only reads the first 3 nameserver entries (from man resolv.conf: Up to MAXNS (currently 3, see <resolv.h>) name servers may be listed, one per keyword). Since the images have 3 hardcoded DNS servers, the DHCP provided servers are never consulted. systemd-resolved as used in 18.04 doesn't have this limit so it uses the DHCP provided servers again.

One of several ideas I experimented with when investigating the original report was whether providing just 2 DNS servers would fix this issue. It didn't quite work as expected, but that was with systemd-resolved and as it turns out, was related to the dnssec requirement.

Note that when using systemd-resolved it may be cleaner to use FallbackDNS in /etc/systemd/resolved.conf so you still prefer the on-network provided DNS servers.

I noticed that option, but wasn't quite sure how it work in practice, and haven't had the time to investigate/experiment.

In my view, the BIGGEST, problem is that every combination of distro/provider/platform has a different ideal solution. Which brings me back to trying to make things work out of the box with the default resolvers, and making it easy to explicitly set nameservers regardless of distro/provider via the config.vm.network namespace. To that end I've been meaning to propose a vagrant feature request, for a "generic" guest agent, and/or, support for embedding a distro specific config script in each image that could be called to set values like hostname, nameserver, etc. I just haven't had the time.

[ekohl]

I can appreciate the lack of time and desire for a unified config to reduce the time spent. 👍 for solving this at the Vagrant layer rather than at the image layer.

[kawaway]

#11 (comment)
vagrant-dnsconf, a vagrant plugin that set guest dns configuration