cirruslabs/softnet

Software networking with isolation for Tart

Softnet

Softnet is a software networking for Tart which provides better network isolation and alleviates DHCP shortage on production systems. Please check out this blog post for backstory.

Working model

Softnet solves two problems:

1. VM network isolation

• VZNATNetworkDeviceAttachment (the default networking in Tart) enables vmnet's bridge isolation by default and prevents cross-VM traffic, however it's still possible for any VM to spoof the host's ARP-table and capture other VMs traffic

2. DHCP exhaustion

• macOS built-in DHCP-server allocates a /24 subnet with 86400 seconds lease time by default, which only allows for ~253 VMs a day (or 1 VM every ~6 minutes) to be spawned without causing a denial-of-service, which is pretty limiting for CI services like Cirrus CI

And assumes that:

1. Tart gives it's VMs unique MAC-addresses
2. macOS built-in DHCP-server won't re-use the IP-addresses from it's pool until their lease expire

...otherwise it's possible for two VMs to receive an identical IP-address from the macOS built-in DHCP-server (even in the presence of Softnet's packet filtering) and thus bypass the protections offered by Softnet.

Installing

For proper functioning, Softnet binary requires two things:

• a SUID-bit to be set on the binary or a passwordless sudo to be configured, which effectively gives the binary root privileges
  • these privileges are needed to create vmnet.framework interface and perform DHCP-related system tweaks
  • the privileges will be dropped automatically to that of the calling user (or those represented by the --user and --group command-line arguments) once all of the initialization is completed
• the binary to be available in PATH
  • so that the Tart will be able to find it

Running

Softnet is started and managed automatically by Tart if --net-softnet flag is provided when calling tart run.

softnet's Issues

Thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: SendError { .. }', lib/host.rs:110:48

Hi,

Occasionally we are seeing this error:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: SendError { .. }', lib/host.rs:110:48

not sure how to reproduce it yet. If anyone has any insights...?

Escalate privileges via passwordless sudo as an alternative to manually setting a SUID-bit

This way the Softnet will work automagically when installed via Homebrew, assuming that the passwordless sudo is configured on the user's machine.

Publish binary and Homebrew releases

So that Softnet can be installed via brew and added to PATH automatically, which is an installation requirement.

Network provisioning delay when creating 2 VMs simultaneously with softnet

This bug appears on both an AWS MacOS instance as well as my Macbook Pro.

I have a cloned image called testing. The name or MacOS image version are not relevant

Reproduce

I have a script that:

• creates a new clone of the testing image
• runs the image with softnet and puts it to the background
• tries to SSH to the VM to run a command
• keeps the VM around for 20s
• deletes the VM

vm_start.sh:

#!/bin/bash

IMAGE_NAME=testing

tart clone $IMAGE_NAME $1
tart run $1 --no-graphics --net-softnet &
killpid=$!
sshpass -padmin ssh -o ConnectTimeout=3600 -o StrictHostKeyChecking=no admin@$(tart ip $1 --wait 300) echo "$1 is done"
sleep 20
kill $killpid
wait $killpid 2>/dev/null
tart delete $1

• ./vm_start.sh a: works fine, as only bringing up one VM
• ./vm_start.sh a &; sleep 5; ./vm_start.sh b;: works fine, because although I'm bringing up 2 VMs, I'm stagging the creation with a sleep
• ./vm_start.sh a &; ./vm_start.sh b: one VM is successful, but the other one typically fails because the SSH command times out.

Expected behaviour would be seeing the output a is done and b is done each time.

Allow SSH connections from host

So that the Cirrus CLI's persistent_worker: isolation with tart: would work.

Originally discussed in https://github.com/cirruslabs/tart/discussions/178.