This guide is focused on my own hardware. If you want a general guide / tutorial, go elsewhere.
- Know what GPU passthrough is and know what you are doing. I'm not explaining a lot of things about GPU passthrough because I assume that you already read many docs. Just search Google;
- Be OK with
vim
; - Adapt the following steps to your own hardware;
- Use a Keepass to store the passwords safely;
- Upgrade your BIOS to the latest version;
- Enable virtualisation features in the BIOS (IOMMU and Amd-v / VT-x);
- Be OK with the i3 shortcuts if you want to use it (optional);
- Know the Linux / networking basics (CLI, configure an IP address, mount a USB device, extract files from an archive, etc).
- MSI MPG B550 GAMING EDGE WIFI (see the
Issues
) - AMD Ryzen 5 3600XT CPU
- 32 Go RAM
- 2 GPUs
- GeForce GTX 1660 SUPER (first PCIE slot), which will be passed through to the Windows VM
- GeForce GT 730 (second PCIE slot), used by the host because my CPU hasn't integrated graphics feature
- Be Quiet Silent Base 802 Window Black case
- Dark Rock PRO 4 ventirad
- 9 Be Quiet fans
- 2 x 1 To SSDs (Samsung and SanDisk) in a ZFS mirrored pool (to store files and VMs disks)
- 1 x 1 To Samsung NVMe SSD 970 EVO Plus (Linux host OS)
- Use i3 on the host machine (right screen). I encountered some stability issues with KDE;
- Use Windows in a VM with native performances (left and middle screens);
- Use the same keyboard and mouse to manage both Windows VM and Linux host (mouse can go from the middle screen to the right screen as if it was the same machine);
- Be able to use a ZFS mirrored pool (that's why i'm using a Linux host) to store data safely, perform snapshots, create secure encrypted datasets, etc;
- Use a clean Ubuntu 20.04 Server distrib;
- Automatically start the Windows VM when the host boots;
- Be able to play games in the VM as if it was a real physical machine without latency;
- Redirect the Windows VM sound to a headset without any latency.
- We will use 12 Go RAM hugepages. This memory will be reserved for the Windows VM in order to increase performances. You can change the value if you want. If, like me, you use ZFS and like 5 or 6 VMs autostarted on boot (ELK, wiki, proxy, dev machines, etc), you will need a lot of RAM. I think 32 Go is the minimum in a setup like this.
- Here, the GPU will be passed through to play games in a Windows VM, but you can also pass it through a Linux VM for crypto mining.
Create a bootable USB key using a Ubuntu 20.04 Server ISO https://releases.ubuntu.com/20.04/. You can also start from a Lubuntu, Xubuntu, etc ISO.
Boot on the USB key and install Ubuntu.
- Choose the language;
- Choose the keyboard layout;
- Don't configure the network (
Continue without network
). The NICs may not been detected; then you will have to finish the installation without internet connection; - Don't use a proxy (
Next
); - Use the default repo (
Next
); - During the storage layout configuration, I recommend to encrypt the disk with luks. If you do so, set up a simple password (avoid azerty / qwerty typo issues), we will change it later. Select the correct device on which you want to install the OS (a NVMe SSD in my case);
- Confirm the storage layout;
- Configure the user name, the machine name and set a simple password (you can change it after);
- Install OpenSSH server, click
Next
; - Once the installation is finished (it should be fast), click
Reboot now
; - Remove the installation medium and press
Enter
.
Boot the OS. Enter the luks password if you set up one. Login. Print the network interfaces.
ip a
On a MSI MPG B550 GAMING EDGE WIFI, the 2.5G ethernet NIC may not appear because you need to install the driver. Also note that you need some packages like make
to compile and install the driver. On another Linux computer, you can :
- Plug and mount a USB key;
- Download the needed packages to build the driver in a folder on the USB key
apt-get download $(apt-cache depends --recurse --no-recommends --no-suggests --no-conflicts --no-breaks --no-replaces --no-enhances --no-pre-depends build-essential | grep "^\w")
- Go on the Realtek website https://www.realtek.com/en/component/zoo/category/network-interface-controllers-10-100-1000m-gigabit-ethernet-pci-express-software, download the
2.5G Ethernet LINUX driver r8125 for kernel up to 5.6
driver and put it also on the USB key. Then on the machine on which we just installed Ubuntu 20.04: - Plug the USB key and mount it;
- Enter the directory where we downloaded the
build-essential
packages, install them withsudo dpkg -i *.deb
(ignore the errors you may encounter); - Enter the directory where we downloaded the NIC driver, extract it (
tar -xvf xxx
) and install it withsudo ./autorun.sh
(ignore the errors you may encounter). - Run
ip a
. The ethernet NIC should be visible. Configure an IP address with the following commands (replace the IP address / netmask considering what your network address is, of course). Replace also the interface name by your own (eno1
in my case).
sudo ip addr add 192.168.1.100/24 dev eno1
sudo ip link set dev eno1 up
sudo ip route add default via 192.168.1.1
The internet access should be OK and you should be able to ping 8.8.8.8
. Edit /etc/resolv.conf
and add nameserver 8.8.8.8
at the top of the others uncommented lines (we shouldn't do that but we don't care because the /etc/resolv.conf
file will be erased automatically when we will apply updates. We will configure the network properly later). Now you should be able to ping google.com
. At this point, you can SSH to this machine from another one if you want. If the content of /etc/resolv.conf
get erased, just configure it again as we did before. Update the packages.
sudo apt update && sudo apt upgrade -y && sudo apt autoremove -y
Configure the DNS in /etc/resolv.conf
as we did before. Install useful packages (or not. You choose).
sudo apt install numlockx terminator xorg nvidia-driver-390 -y
Configure the DNS in /etc/resolv.conf
as we did before. I use i3, but you can use the desktop environment (or windows manager) you want. The following is a copy / paste of https://github.com/addy-dclxvi/i3-starterpack. Install i3 and the packages we will use. You can also use your own i3 config.
sudo apt install i3 compton hsetroot rxvt-unicode xsel rofi fonts-noto fonts-mplus xsettingsd lxappearance scrot viewnior -y
Configure X to use i3.
echo "exec i3" >> ~/.xinitrc
Start the X server.
startx
Press Enter
2 times to generate the config and use the Windows
key as super key. On the right bottom, click on the small icon and connect to the Wifi. You can open a terminal by pressing Windows
+ Enter
. Now you should be able to ping google.com
.
Get the i3 configuration.
mkdir /tmp/i3
cd /tmp/i3
git clone https://github.com/addy-dclxvi/i3-starterpack.git
cd i3-starterpack
cp -r .Xresources .config/ .fonts/ .urxvt/ .wallpaper.png .xsettingsd ~/
Refresh fonts.
fc-cache -fv
Adapts the configuration to your needs. Change the NIC names. Super
key is the Windows
key. Use Super + Shift + R
to refresh the running config using the config files.
Run the command below to print the datetime status.
timedatectl status
If the timezone is correct, you can go to the next paragraph. If not, follow the steps below. Liste the available timezeones.
timedatectl list-timezones
Set the timezone you want. The example below is for Paris time.
sudo timedatectl set-timezone Europe/Paris
I'm not talking about IOMMU groups. Just search Google about that. Print the PCI devices.
lspci -nn
You should get a result like that. Here are the devices I want to pass through.
$ lspci -nn
[...]
2b:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1660 SUPER] [10de:21c4] (rev a1)
2b:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1)
2b:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1)
2b:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] [10de:1aed] (rev a1)
[...]
I also want to pass my headset which is connected with USB. Print the USB devices.
lsusb
Here is the USB device I want to pass through. Note that if you are installing a new VM, you will need to pass a keyboard and a mouse (it means that you will need to have 2 keyboards and 2 mouses connected to your desktop).
$ lsusb
[...]
Bus 003 Device 005: ID 1038:12ad SteelSeries ApS SteelSeries Arctis 7
Bus 001 Device 008: ID 046d:c328 Logitech, Inc. Corded Keyboard K280e
Bus 001 Device 009: ID 145f:01ea Trust Trust Presenter Mouse
[...]
So we need to pass the devices 10de:21c4
, 10de:1aeb
, 10de:1aec
, 10de:1aed
, 1038:12ad
, 046d:c328
and 145f:01ea
.
Edit /etc/default/grub
.
sudo vim /etc/default/grub
Change the line GRUB_CMDLINE_LINUX_DEFAULT
to the following (change the values according to your hardware). It's not needed to specify the USB devices. We need to specify all the IOMMU group.
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt kvm.ignore_msrs=1 vfio-pci.ids=10de:21c4,10de:1aeb,10de:1aec,10de:1aed"
Update Grub.
sudo update-grub
Generate a new initramfs image.
sudo update-initramfs -u -k all
Reboot.
sudo reboot
After the reboot, print the PCI devices.
lspci -nnv
The kernel driver in use should always be vfio-pci
for the devices we passed through (your GPU).
2b:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1660 SUPER] [10de:21c4] (rev a1) (prog-if 00 [VGA controller])
[...]
Kernel driver in use: vfio-pci
[...]
2b:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1)
[...]
Kernel driver in use: vfio-pci
[...]
2b:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1) (prog-if 30 [XHCI])
[...]
Kernel driver in use: vfio-pci
[...]
2b:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] [10de:1aed] (rev a1)
[...]
Kernel driver in use: vfio-pci
[...]
Now we can use the GPU in a VM.
Install the needed packages for KVM.
sudo apt install libvirt-daemon-system virt-manager qemu-kvm ovmf bridge-utils -y
Reboot.
sudo reboot
Now we want to replace the default
bridge by another one. List the KVM bridges.
virsh net-list --all
Delete the default bridge.
virsh net-undefine default
virsh net-destroy default
Create a new config file.
sudo vim virbr0.conf
Paste the following (adapt the content).
<network>
<name>virbr0</name>
<uuid>168fc909-7c5c-4333-a3cb-f74584d26321</uuid>
<forward mode='nat'/>
<bridge name='virbr0' stp='on' delay='0'/>
<mac address='52:54:00:48:3f:00'/>
<ip address='10.85.66.1' netmask='255.255.255.0'/>
</network>
Create the bridge using the config file.
virsh net-define --file virbr0.conf
Configure the bridge to be started automatically and start it.
virsh net-autostart --network virbr0
virsh net-start virbr0
Use the following command to install the Windows VM.
- Change the RAM and vCPUs values if you want;
- Always use virtio and raw disk format for better performances;
2b:00.0
and2b:00.1
is the GPU. It's not mandatory to specify the USB controller and the Serial bus controller;1038:12ad
is my headset;046d:c328
is the keyboard we will pass to the VM (that's why you need 2 keyboards);145f:01ea
is the mouse we will pass to the VM (that's why you need 2 mouses);- Check that the file
/usr/share/OVMF/OVMF_CODE.fd
exists. - Check the ISOs paths;
018-win10
is my VM name, change it if you want;- When you will run the command below, a message will be printed (
Press any to to boot...
), then you have to be quick if you don't want to drop in the UEFI shell. If you do any mistake, just delete the VM and start again (virsh destroy 018-win10 && sleep 5 && virsh undefine 018-win10 --nvram && sudo rm /tank/vms/storage/018-win10/018-win10-disk1.raw
).
virt-install \
--virt-type kvm \
--name 018-win10 \
--os-variant=win10 \
--vcpus 8 \
--memory 12288 \
--features kvm_hidden=on \
--disk path=/path/to/018-win10-disk1.raw,format=raw,bus=virtio,size=50 \
--cdrom /path/to/Win10_20H2_v2_French_x64.iso \
--disk /path/to/virtio-win-0.1.185.iso,device=cdrom \
--network bridge=virbr0,model=virtio \
--graphics none \
--hostdev 2b:00.0,address.type=pci,address.multifunction=on \
--hostdev 2b:00.1,address.type=pci \
--hostdev 1038:12ad \
--hostdev 046d:c328 \
--hostdev 145f:01ea \
--boot loader=/usr/share/OVMF/OVMF_CODE.fd,loader.readonly=yes,loader.type=pflash
During the disk selection in the installer :
- Click on
Load a driver
, browse the virtio CD, selectamd64
,w10
, clickOK
, clickNext
to load the driver. The virtio disk should be visible. Now do the same for the NIC. - Click again on
Load a driver
, browse the virtio CD, selectNetKVM
,w10
,amd64
, clickOK
, clickNext
to load the driver. Once the installation is finished, you should see the Windows desktop with a bad resolution. Now, configure a static IP address (then you should be able toping google.com
). At this point, you still have a bad screen resolution. Shutdown the VM.
If you already own the VM disk (backup), you can use the following command. If you had barrier configured on this VM, the SSL keys fingerprints may have changed and barrier wont work. Then, connect on the VM using RDP and accept the new key fingerprint.
virt-install \
--virt-type kvm \
--name 018-win10 \
--os-variant=win10 \
--vcpus 8 \
--memory 12288 \
--features kvm_hidden=on \
--import \
--disk path=/path/to/018-win10-disk1.raw,format=raw,bus=virtio \
--network bridge=virbr0,model=virtio \
--graphics none \
--hostdev 2b:00.0,address.type=pci,address.multifunction=on \
--hostdev 2b:00.1,address.type=pci \
--hostdev 1038:12ad \
--boot loader=/usr/share/OVMF/OVMF_CODE.fd,loader.readonly=yes,loader.type=pflash
Then virsh shutdown
and perform the optimization steps below.
Edit /etc/sysctl.conf
.
sudo vim /etc/sysctl.conf
Paste the following at the end (we cant to allocate 12Go).
# Hugepages
vm.nr_hugepages = 6144
Apply the changes.
sudo sysctl -p
The following should be printed.
$ sudo sysctl -p
vm.nr_hugepages = 6144
If you run a top
, htop
or free -h
, you should see that the amount of RAM you set up as hugepages is now used.
Edit the VM config.
virsh edit 018-win10
Under </currentMemory>
, add the following.
<memoryBacking>
<hugepages/>
</memoryBacking>
Edit the VM config.
virsh edit 018-win10
Between <hyperv>
and </hyperv>
, add the following.
<vendor_id state='on' value='1234567890ab'/>
Edit the VM config.
virsh edit 018-win10
Under <vcpu placement='static'>
, add the following.
<cputune>
<vcpupin vcpu='0' cpuset='2'/>
<vcpupin vcpu='1' cpuset='8'/>
<vcpupin vcpu='2' cpuset='3'/>
<vcpupin vcpu='3' cpuset='9'/>
<vcpupin vcpu='4' cpuset='4'/>
<vcpupin vcpu='5' cpuset='10'/>
<vcpupin vcpu='6' cpuset='5'/>
<vcpupin vcpu='7' cpuset='11'/>
<emulatorpin cpuset='0-1,6-7'/>
</cputune>
Replace <cpu mode='host-model' check='partial'/>
by the following line.
<cpu mode='host-passthrough' check='none'>
<topology sockets='1' cores='4' threads='2'/>
</cpu>
Boot the Windows VM. Install all the Windows updates. The screen resolution should now be normal. You can also install the Nvidia drivers if you want. In the tasks manager, you should see your GPU.
Now we want to use the same keyboard and mouse (the host's ones) to manage the Windows VM and the Linux host. To do so, install barrier
on the Linux host.
sudo apt install barrier
Install it also on the Windows VM. Configure it on both machines. Then you should be able to use the same keyboard and mouse on both systems.
Just run the following command (replace the VM name).
virsh autostart 018-win10
If you used a weak password during the installation, change it using the commands below. The device name (/dev/nvme0n1p3
) may change for you.
sudo cryptsetup luksDump /dev/nvme0n1p3
Change the luks key.
sudo cryptsetup luksChangeKey /dev/nvme0n1p3 -S 0
Note that you can also use CRTL
+ ALT
. Edit /etc/barrier.conf
.
sudo vim /etc/barrier.conf
Add the following (adapt the content). The screen name must be the machine hostname. The Windows VM, even if it has 2 monitors (in my case), counts as one screen in the config.
section: screens
my-desktop:
halfDuplexCapsLock = false
halfDuplexNumLock = false
halfDuplexScrollLock = false
xtestIsXineramaUnaware = false
preserveFocus = false
switchCorners = none
switchCornerSize = 0
meta = altgr
altgr = shift
018-win10:
halfDuplexCapsLock = false
halfDuplexNumLock = false
halfDuplexScrollLock = false
xtestIsXineramaUnaware = false
preserveFocus = false
switchCorners = none
switchCornerSize = 0
meta = altgr
altgr = shift
end
section: links
018-win10:
right = my-desktop
my-desktop:
left = 018-win10
end
Now, load and test the config using the barrier GUI.
With i3 it's easy. Edit the i3 config file.
vim ~/.config/i3/config
Add the following (for example in the #autostart
section).
exec --no-startup-id /usr/bin/barriers -f --no-tray --debug INFO --name my-desktop --enable-crypto -c /etc/barrier.conf --address [10.85.66.1]:24800
Try to reboot your computer and see if it works.
sudo reboot
If you have ZFS pools, you can import then using the commands below.
sudo apt install zfsutils-linux -y
sudo zpool import <pool_name> -f
Note that you can't use both NVMe slots on a MSI MPG B550 GAMING EDGE WIFI when you are using both PCIE slots.
Check that the VM has autostart
enabled. Check that all the (USB) devices you redirected to this VM are connected to the host. Try to start the Windows VM manually with virsh
and see which warning message is printed.
- Configure hugepages on host https://help.ubuntu.com/community/KVM%20-%20Using%20Hugepages
- GPU passthrough guide for Ubuntu https://mathiashueber.com/windows-virtual-machine-gpu-passthrough-ubuntu/
- Download a package with dependencies https://stackoverflow.com/questions/13756800/how-to-download-all-dependencies-and-packages-to-directory
- i3 configuration https://github.com/addy-dclxvi/i3-starterpack
- Timezone configuration https://linuxize.com/post/how-to-set-or-change-timezone-on-ubuntu-18-04/
- Correct barrier alt gr issue debauchee/barrier#100