lima-vm/socket_vmnet

vmnet.framework support for unmodified rootless QEMU (no dependency on VDE)

socket_vmnet's Issues

ERROR: Failed to connect to "/var/run/socket_vmnet": Permission denied

I've been using socket_vmnet without issue for a couple months, but every time I try using it now I get:
Failed to connect to "/var/run/socket_vmnet": Permission denied

$ /opt/socket_vmnet/bin/socket_vmnet_client /var/run/socket_vmnet qemu-system-aarch64
Failed to connect to "/var/run/socket_vmnet": Permission denied

I recently updated to macOS 12.6.1, may have been a coincidence (I tried on a machine with 13.0 and it works). A coworker has also been experiencing this for a little while (before 12.6.1).

I ran make uninstall and manually removed /var/run/socket_vmnet, restarted my computer, reinstalled, but still the same error.

$ ls -l /var/run
srwxrwx---   1 root          staff           0 Oct 28 10:43 socket_vmnet
-rw-r--r--   1 root          daemon          0 Oct 28 10:43 socket_vmnet.stderr
-rw-r--r--   1 root          daemon          0 Oct 28 10:43 socket_vmnet.stdout

Support `VZFileHandleNetworkDeviceAttachment`

Virtualization.framework provides built-in support for vmnet.framework (without root, for the NAT (shared) mode).

However, it has some drawbacks compared to socket_vmnet:

• VZNATNetworkDeviceAttachment doesn't support custom IP range
• VZBridgedNetworkDeviceAttachment requires the com.apple.vm.networking entitlement, which requires contacting to an Apple representative.

So, it still makes sense to support socket_vmnet for VZ.

VZFileHandleNetworkDeviceAttachment can be used for this, but its protocol is incompatible with QEMU. (DGRAM sock, without the uint32be length header)

Context:

• lima-vm/lima#1206 (comment)

---

Probably, CLI will be like socket_vmnet unix:///var/run/socket_vmnet unixgram://var/run/socket_vmnet.unixgram to cover both QEMU socket and VZ socket in a single process (so as to avoid complicating the sudoers file too much)

Provide install binary or package for end users?

Hi I would like to know if there is a plan to create a binary or a package so the minikube users could install this or minikube could auto-install it, currently our process requires people with knowledge of using "make" and install build tools to make it work?

or is it possible to install/update it using brew install ?

https://minikube.sigs.k8s.io/docs/drivers/qemu/#requirements

Multiple VMs Might Break Network Connections

OS / App Versions

• macOS 13.5 (x86-64)
• QEMU 7.2.1
• socket_vmnet 1.1.2 built from source

Problem

The network of currently started QEMU VM stops working (symptom: ping 1.1.1.1 times out) when another VM (with unique MAC address) boots up but DOES NOT initiate any DHCP request.

However, after the second VM finishes its DHCP request, the network on first VM resumes to normal.

Steps to Reproduce

1. start socket_vmnet

sudo env DEBUG=true /usr/local/bin/socket_vmnet \
	--socket-group=admin \
	--vmnet-mode=shared \
	--vmnet-gateway=10.233.2.1 \
	--vmnet-dhcp-end=10.233.2.250 \
	--vmnet-mask=255.255.255.0 \
	/run/vmnet.sock

2. start VM `vmnet-test-01`

# Alpine Live ISOs available at https://www.alpinelinux.org/downloads/
# The "Virtual" ones are smallest

/usr/local/bin/socket_vmnet_client \
    /run/vmnet.sock \
    qemu-system-x86_64 \
        -name "vmnet-test-01" \
        -machine type=q35,accel=hvf \
        -cpu host \
        -smp cpus=1,sockets=1,cores=1,threads=1 \
        -m 1024 \
        -cdrom "$HOME/Downloads/alpine-virt-3.18.2-x86_64.iso" \
        -netdev socket,id=net0,fd=3 \
        -device "virtio-net-pci,netdev=net0,mac=de:ad:be:ef:00:01" \
        -object rng-random,filename=/dev/urandom,id=rng0 \
        -device virtio-rng-pci,rng=rng0 \
        -device virtio-gpu-pci \
        -nographic

3. acquire IP via DHCP for vmnet-test-01

# login with user "root"

cat > /etc/network/interfaces <<-END
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp
END

rc-service networking start

# it pings successfully
ping -c 5 -w 1 1.1.1.1

4. start VM `vmnet-test-02`

/usr/local/bin/socket_vmnet_client \
    /run/vmnet.sock \
    qemu-system-x86_64 \
        -name "vmnet-test-02" \
        -machine type=q35,accel=hvf \
        -cpu host \
        -smp cpus=1,sockets=1,cores=1,threads=1 \
        -m 1024 \
        -cdrom "$HOME/Downloads/alpine-virt-3.18.2-x86_64.iso" \
        -netdev socket,id=net0,fd=3 \
        -device "virtio-net-pci,netdev=net0,mac=de:ad:be:ef:00:02" \
        -object rng-random,filename=/dev/urandom,id=rng0 \
        -device virtio-rng-pci,rng=rng0 \
        -device virtio-gpu-pci \
        -nographic

5. after 1-2 minutes, network on vmnet-test-01 stops working

# timeout
ping -c 5 -w 1 1.1.1.1

# DHCP cannot renew lease
rc-service networking restart

6. repeat step 3 for vmnet-test-02

# login with user "root"

cat > /etc/network/interfaces <<-END
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp
END

rc-service networking start

# it pings successfully
ping -c 5 -w 1 1.1.1.1

7. surprisingly, now networks on both VMs resume to normal

Debug Log

https://gist.github.com/iwinux/305a9bea6b77c5c44494bb1883354f72

socket_vmnet failing on M1 (`start(): vmnet_return_t VMNET_FAILURE`)

I've now observed the error from lima-vm/lima#1049 two more times (qemu failing to start up because fd_connect throws an error). Both times have been on an M1 mini; I cannot remember if the bug report on the lima repo was also based on a failure on M1, or if it was Intel.

Unfortunately I've been running with lima 0.12.0, which doesn't have the error reporting fix. However, I can see errors in the daemon logs (after qemu failed):

jan@zilicon _networks % cat rancher-desktop-shared_socket_vmnet.stderr.log
start(): vmnet_return_t VMNET_FAILURE
start: Undefined error: 0
jan@zilicon _networks % cat rancher-desktop-shared_socket_vmnet.stdout.log
Initializing vmnet.framework (mode 1001)
jan@zilicon _networks % cat rancher-desktop-bridged_en0_socket_vmnet.stderr.log
on_accept(): vmnet_return_t VMNET_INVALID_ARGUMENT
vmnet_write: Undefined error: 0

The bridged network was running, but the shared network was not.

The only way I found to get things working again was by rebooting the machine.

Drop the root privilege after the initialization

Related: https://github.com/cirruslabs/softnet/tree/main#installing

DHCP leases file not getting updated

We've been getting issues on our personal machines where our DHCP leases file is not being updated after starting an instance with socket_vmnet. The fix so far has been uninstall socket_vmnet, reboot machine, reinstall, which resolves the issue. However I'm trying to integrate GitHub Action tests for QEMU/socket_vmnet and I'm getting the same thing (logs below).

Is there a command we could run that would resolve this issue? I see in the README there's a command to Reload the DHCP daemon

sudo /bin/launchctl kickstart -kp system/com.apple.bootpd

I'm not sure if that should resolve the issue, however trying to run it in GitHub Action resulted in Could not find service "com.apple.bootpd" in domain for system

Below are to logs from the GitHub Action machine with the main error being StartHost failed, but will try again: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: open /var/db/dhcpd_leases: no such file or directory

We experienced the same thing on our personal machines, we tried manually creating the file just incase the there were some permission errors but that didn't resolve the issue, plus I'm sure it's more than capable of creating the file itself.

Any help would be appreciated, thanks!

functional_test.go:2161: (dbg) Run:  ./minikube-darwin-amd64 start -p func-184505 --memory=4000 --apiserver-port=8441 --wait=all --driver=qemu
    functional_test.go:2161: (dbg) Non-zero exit: ./minikube-darwin-amd64 start -p func-184505 --memory=4000 --apiserver-port=8441 --wait=all --driver=qemu: exit status 80 (2m23.88093201s)
        
        -- stdout --
        	* [func-184505] minikube v1.28.0 on Darwin 12.6.1
        	  - MINIKUBE_HOME=/Users/runner/work/minikube/minikube/minikube_binaries/testhome
        	  - KUBECONFIG=/Users/runner/work/minikube/minikube/minikube_binaries/testhome/kubeconfig
        	* Using the qemu2 (experimental) driver based on user configuration
        	* Automatically selected the socket_vmnet network
        	* Downloading VM boot image ...
        	* Starting control plane node func-184505 in cluster func-184505
        	* Downloading Kubernetes v1.25.3 preload ...
        	* Creating qemu2 VM (CPUs=2, Memory=[40](https://github.com/spowelljr/minikube/actions/runs/3604165929/jobs/6073444435#step:12:41)00MB, Disk=20000MB) ...
        	* Deleting "func-184505" in qemu2 ...
        	* Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
        	
        	
        
        -- /stdout --
        ** stderr ** 
        	! Local proxy ignored: not passing HTTP_PROXY=localhost:49253 to docker env.
        	! Local proxy ignored: not passing HTTP_PROXY=localhost:49253 to docker env.
        	! StartHost failed, but will try again: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: open /var/db/dhcpd_leases: no such file or directory
        	! Local proxy ignored: not passing HTTP_PROXY=localhost:49253 to docker env.
        	* Failed to start qemu2 VM. Running "minikube delete -p func-18[45](https://github.com/spowelljr/minikube/actions/runs/3604165929/jobs/6073444435#step:12:46)05" may fix it: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: open /var/db/dhcpd_leases: no such file or directory
        	X Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: open /var/db/dhcpd_leases: no such file or directory

brew instruction to install launchd service is wrong

Thank you for adding socket_vmnet to brew, I just tried installing it and seems to be installed but the instructions to add launchd service is wrong (when I run with sudo it dosnt work) I think it needs to be run without sudo

==> Caveats
To install an optional launchd service, run the following command (sudo is necessary):
sudo brew services start socket_vmnet

socket_vmnet is keg-only, which means it was not symlinked into /opt/homebrew,
because /opt/homebrew/bin is often writable by a non-admin user.

If you need to have socket_vmnet first in your PATH, run:
  echo 'export PATH="/opt/homebrew/opt/socket_vmnet/bin:$PATH"' >> /Users/medya/.bash_profile


To restart socket_vmnet after an upgrade:
  brew services restart socket_vmnet

here is when I try to it with an without sudo

$ sudo brew services start socket_vmnet
sudo: brew: command not found


$ brew services start socket_vmnet
==> Tapping homebrew/services
Cloning into '/opt/homebrew/Library/Taps/homebrew/homebrew-services'...
remote: Enumerating objects: 2233, done.
remote: Counting objects: 100% (167/167), done.
remote: Compressing objects: 100% (62/62), done.
remote: Total 2233 (delta 120), reused 112 (delta 105), pack-reused 2066
Receiving objects: 100% (2233/2233), 614.54 KiB | 1.84 MiB/s, done.
Resolving deltas: 100% (1000/1000), done.
Tapped 1 command (45 files, 773KB).
==> Successfully started `socket_vmnet` (label: homebrew.mxcl.socket_vmnet)
10:45:20 medya/workspace/minikube

This or VDE?

It would be nice if there was some more documentation about the differences between this and vde_vmnet. Obviously one depends on VDE and one doesn't, but (assuming I can compile VDE without too much trouble) that doesn't really tell me anything about which to prefer as a normal user.

Are there any other differences in functionality between the two? Is the intent that this will replace vde_vmnet at some point? Or should I just choose one arbitrarily?

mac address in bridged mode

I'm using --vmnet-mode=bridged, and my debian bookworm guest is able to get an IP that's in the same subnet as the macos(11.7.8) host. While it "works", I see that my router's dhcp server sees both the macos host and the debian guest IPs on the same MAC address. This MAC address is the same as the bridged interface's MAC address on the macos host. Using the arp command, I've verified that other hosts on my network also see the same mac address for both the macos host IP and the debian guest IP. Am I doing something wrong? After some digging, I found that Parallels also had a similar issue with the vmnet framework for a bit before they resolved it. You can see reports here -> https://forum.parallels.com/threads/pd16-on-big-sur-vm-not-using-the-configured-mac-address.351236/

Security Policy violation SECURITY.md

This issue was automatically created by Allstar.

Security Policy Violation
Security policy not enabled.
A SECURITY.md file can give users information about what constitutes a vulnerability and how to report one securely so that information about a bug is not publicly visible. Examples of secure reporting methods include using an issue tracker with private issue support, or encrypted email with a published key.

To fix this, add a SECURITY.md file that explains how to handle vulnerabilities found in your repository. Go to https://github.com/lima-vm/socket_vmnet/security/policy to enable.

For more information, see https://docs.github.com/en/code-security/getting-started/adding-a-security-policy-to-your-repository.

---

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

find a bug in function on_vmnet_packets_available

socket_vmnet/main.c

Line 230 in 910aaef

_on_vmnet_packets_available(iface, q, max_bytes, state);

I think it may be replaced by
_on_vmnet_packets_available(iface, MAX_PACKET_COUNT_AT_ONCE, max_bytes, state);

Can't start launchd service on brew install

Reproduction:
Install socket_vmnet via brew

$ brew install socket_vmnet
Running `brew update --auto-update`...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).

==> Downloading https://ghcr.io/v2/homebrew/core/socket_vmnet/manifests/1.1.0
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/socket_vmnet/blobs/sha256:8e309781893e7dd83edadee99ab08ec8295917f3e687f9c3949c95129be9ef03
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:8e309781893e7dd83edadee99ab08ec8295917f3e687f9c3949c95129be9ef03?se=2022-12-05T21%3A05%3A00Z&sig=YQb2gAlUoTEffFyUVsXM5dPvxxBb3PjxW%2FrOoWUzmEw%3D&sp=r&spr=https&sr=b&sv=2019-12-12
######################################################################## 100.0%
==> Pouring socket_vmnet--1.1.0.arm64_ventura.bottle.tar.gz
==> Caveats
To install an optional launchd service, run the following command (sudo is necessary):
sudo brew services start socket_vmnet

socket_vmnet is keg-only, which means it was not symlinked into /opt/homebrew,
because /opt/homebrew/bin is often writable by a non-admin user.

If you need to have socket_vmnet first in your PATH, run:
  echo 'export PATH="/opt/homebrew/opt/socket_vmnet/bin:$PATH"' >> /Users/powellsteven/.bash_profile


To restart socket_vmnet after an upgrade:
  brew services restart socket_vmnet
Or, if you don't want/need a background service you can just run:
  /opt/homebrew/opt/socket_vmnet/bin/socket_vmnet --vmnet-gateway=192.168.105.1 /opt/homebrew/var/run/socket_vmnet
==> Summary
🍺  /opt/homebrew/Cellar/socket_vmnet/1.1.0: 14 files, 112.9KB
==> Running `brew cleanup socket_vmnet`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

Using the info, try starting the launchd service

$ sudo brew services start socket_vmnet
sudo: brew: command not found

brew is not available from root by default, trying again by specifying full path

$ sudo /opt/homebrew/bin/brew services start socket_vmnet
Error: Running Homebrew as root is extremely dangerous and no longer supported.
As Homebrew does not drop privileges on installation you would be giving all
build scripts full access to your system.
Error: Failure while executing; `/opt/homebrew/bin/brew tap homebrew/services` exited with 1.

Security Policy violation Branch Protection

This issue was automatically created by Allstar.

Security Policy Violation
No protection found for branch master

---

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

socket_vmnet gets stuck randomly

We have 7 qemu VMs running, having 3 virtual ethernet interface each.

socket_vmnet works most of the times but randomly stops working and the communication between the VMs is stopped.

The debug logs show the process get stuck on writev() call.
DEBUG| [Socket-to-Socket i=1815762] Sending from socket 8 to socket 5: 4 + 95 bytes

There is no log after the above log:
On the VM reboot the logs show that writev() call return -1

I suspect this is due to a race condition when multiple threads are accessing the socket to send and receive data. I don't have the exact explanation yet bet the behavior is pointing to a race condition.

Implement L2 learning to avoid flooding

socket_vmnet should learn the MAC addresses of the VMs, set the MAC addresses to the struct conn objects, and avoid flooding when possible.

socket_vmnet/main.c

Lines 85 to 89 in 8b16e51

	struct conn {
	// TODO: uint8_t mac[6];
	int socket_fd;
	struct conn *next;
	} _conn;

socket_vmnet/main.c

Lines 180 to 205 in 8b16e51

	for (struct conn *conn = conns; conn != NULL; conn = conn->next) {
	// FIXME: avoid flooding
	DEBUGF("[Handler i=%d] Sending to the socket %d: 4 + %ld bytes [Dest "
	"%02X:%02X:%02X:%02X:%02X:%02X]",
	i, conn->socket_fd, pdv[i].vm_pkt_size, dest_mac[0], dest_mac[1],
	dest_mac[2], dest_mac[3], dest_mac[4], dest_mac[5]);
	uint32_t header_be = htonl(pdv[i].vm_pkt_size);
	struct iovec iov[2] = {
	{
	.iov_base = &header_be,
	.iov_len = 4,
	},
	{
	.iov_base = pdv[i].vm_pkt_iov[0].iov_base,
	.iov_len = pdv[i].vm_pkt_size, // not vm_pkt_iov[0].iov_len
	},
	};
	ssize_t written = writev(conn->socket_fd, iov, 2);
	DEBUGF("[Handler i=%d] Sent to the socket: %ld bytes (including uint32be "
	"header)",
	i, written);
	if (written < 0) {
	perror("writev");
	goto done;
	}
	}

socket_vmnet/main.c

Lines 512 to 542 in 8b16e51

	// Flood the packet to other VMs in the same network too.
	// (Not handled by vmnet)
	// FIXME: avoid flooding
	dispatch_semaphore_wait(state->sem, DISPATCH_TIME_FOREVER);
	struct conn *conns = state->conns;
	dispatch_semaphore_signal(state->sem);
	for (struct conn *conn = conns; conn != NULL; conn = conn->next) {
	if (conn->socket_fd == accept_fd)
	continue;
	DEBUGF("[Socket-to-Socket i=%lld] Sending from socket %d to socket %d: "
	"4 + %d bytes",
	i, accept_fd, conn->socket_fd, header);
	struct iovec iov[2] = {
	{
	.iov_base = &header_be,
	.iov_len = 4,
	},
	{
	.iov_base = buf,
	.iov_len = header,
	},
	};
	ssize_t written = writev(conn->socket_fd, iov, 2);
	DEBUGF("[Socket-to-Socket i=%lld] Sent from socket %d to socket %d: %ld "
	"bytes (including uint32be header)",
	i, accept_fd, conn->socket_fd, written);
	if (written < 0) {
	perror("writev");
	continue;
	}
	}

implement --version

Hi, thank you for adding binaries to the release page, we are excited to look into using it !

#10

I would like to know if we could have a "--version" or "version" command to see what version it is installed, so we could update or tell user to update their socket_vmnet