GPU-Passthrough with Qemu (without libvirt)

In the Internet there are many tutorials how to passthrough the GPU to a VM from a Linux host. Usually those instructions are based on libvirt. But I like to keep things “simple”, so I did this without libvirt.

On the other hand this tutorial requires fluent knowledge how to use Qemu.

I did this using a AMD Radeon RX 590 and a Debian-11 Linux system with a Ryzen-GPU and an ASUS TUF GAMING B550-PLUS mainboard. I have no second GPU, so I had to unbind my host from the GPU. And everything on the host had to be done via SSH from a second computer.
This works in BIOS/MBR and in (U)EFI boot mode.

Alternatively you may “virtualize” your GPU without passing it fully trough to the guest. But this is only possible with modern Intel graphics (>= 5th i-Core generation) or “professional” and expensive AMD oder NVidia GPUs.
https://wiki.archlinux.org/title/Intel_GVT-g
https://en.wikipedia.org/wiki/GPU_virtualization#Mediated_pass-through

1. extract GPU-ROM.
For NVidia GPUs additional modifications to the ROM file may be required. But I haven’t worked trough that.

echo 1 > /sys/bus/pci/devices/0000\:05\:00.0/rom
cat /sys/bus/pci/devices/0000\:05\:00.0/rom > vbios.rom
echo 0 > /sys/bus/pci/devices/0000\:05\:00.0/rom

2. Additional boot parameters.

# add to /etc/default/grub -> GRUB_CMDLINE_LINUX_DEFAULT
amd_iommu=on intel_iommu=on iommu=pt
# run:
update-grub

3a. Unbind GPU and HDMI audio drivers and bind vfio module to GPU at boot time.

/etc/initramfs-tools/modules
----------
vfio
vfio_iommu_type1
vfio-pci
vfio_virqfd
----------

/etc/modprobe.d/vfio.conf
----------
softdep amdgpu pre: vfio vfio_pci
softdep snd_hda_intel: vfio vfio_pci
options vfio_pci ids=1002:67df,1002:aaf0
blacklist amdgpu
----------

# Boot parameters via /etc/default/grub -> GRUB_CMDLINE_LINUX_DEFAULT
# (adjust PCI ids for GPU and HDMI audio device)
# Replace "amdgpu" with your GPU driver. (e.g. i915 or nouveau)
vfio-pci.ids=1002:67df,1002:aaf0
rd.driver.pre=vfio-pci
rd.driver.blacklist=amdgpu
modprobe.blacklist=amdgpu
vfio-pci.disable_idle_d3=1
# finally run:
update-initramfs -u
update-grub
reboot

3a. Unbind GPU and HDMI audio drivers and bind vfio module to GPU while the system is running. (adjust PCI ids and PCI adresses for GPU and HDMI audio)

systemctl stop display-manager.service
echo '0000:05:00.1' > /sys/bus/pci/drivers/snd_hda_intel/unbind
echo '0000:05:00.0' > /sys/bus/pci/drivers/amdgpu/unbind
modprobe vfio_pci ids=1002:67df,1002:aaf0 disable_idle_d3=1

4. Before starting Qemu (no vtcon1 and no efi-framebuffer with EFI boot):

echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

 

5. Add these options to your Qemu start script (vbios.rom file needed):

-device vfio-pci,host=05:00.0,multifunction=on,x-vga=on,id=vfio0,romfile=vbios.rom \
-device vfio-pci,host=05:00.1,id=vfio1 \
-vga none \
-display none \

 

Have fun :-)

 

See also: