PCI Passthrough in Proxmox

Important BIOS settings for getting things working properly!

Make sure your BIOS is on its latest update.
Enable VT-d
Required for passthrough to work at all; supported CPUs.
Disable CSM/Legacy Boot
Enable ACS
If present, set to Enabled (Auto doesn’t work).
Set PEG {NUMBER} ASPM to L0 or L0sL1
Set whichever amount you've got to mentioned, if L0 is present, pick that.
Enable 4G Decoding
Disable Resizable BAR/Smart Access Memory
AMD GPUSs (Vega and up) experience ‘Code 43 error’ if enabled.
Enable IOMMU
If present, mostly for AMD boards; supported CPUs.
Set Primary Display to CPU/iGPU
If your CPU has an iGPU, if not skip this.
Pre-Allocated Memory to 64M
If your CPU has an iGPU, if not skip this too.

Setting the Boot arguments

nano /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt initcall_blacklist=sysfb_init"

GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt initcall_blacklist=sysfb_init"

Hit Ctrl + X -> Y -> Enter to save changes.

Your system might not be relying on Grub, but systemd instead, this is the case when you’re using ZFS.
Ignore if Proxmox was not set up with RAID/ZFS during setup.

Edit the systemd file

nano /etc/kernel/cmdline

For Intel CPUs

root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt initcall_blacklist=sysfb_init

For AMD CPUs

root=ZFS=rpool/ROOT/pve-1 boot=zfs iommu=pt initcall_blacklist=sysfb_init

Hit Ctrl + X -> Y -> Enter to save changes.

Once you’ve made necessary changes to your system, we need to update the system file and reboot.

For GRUB systems

update-grub

For systemd systems

pve-efiboot-tool refresh

Finally, reboot

reboot

Configuring IOMMU

Once the Host is up and running again, we need to verify that IOMMU is enabled.

For Intel CPUs

dmesg | grep -e DMAR -e IOMMU

For AMD CPUs

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi

You should be seeing something like this;
“DMAR: IOMMU enabled”, if not, go back and double check changes.

Then we need to enable necessary kernel modules, edit the following file

nano /etc/modules

Add in these lines

vfio
vfio_iommu_type1
vfio_pci

Hit Ctrl + X -> Y -> Enter to save changes.

After changing anything modules related, you need to refresh your initramfs.

update-initramfs -u -k all

Now check if remapping has been enabled.

dmesg | grep remapping

Should output something like this;
For AMD CPUs
“AMD-Vi: Interrupt remapping enabled”
For Intel CPUs
“DMAR-IR: Enabled IRQ remapping in x2apic mode”
x2apic can be different on older CPUs, but should still work.
Only if you encounter issues, consider this;
You could also try adding in nox2apic to your Grub or Systemd args.

If your system doesn’t support interrupt remapping, you might want to allow unsafe interrupts.
Please be aware that this option might make your system unstable, but not necessarily.

Edit the following file

nano /etc/modprobe.d/iommu_unsafe_interrupts.conf

Add in this line

options vfio_iommu_type1 allow_unsafe_interrupts=1

Hit Ctrl + X -> Y -> Enter to save changes.

Checking IOMMU Groups

For PCI passthrough to work properly, you need to have a dedicated IOMMU group for each of the devices you want to assign to a VM.
To make sure that’s the case, enter the following command;

pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist ""

Replace {nodename} with the name of your Proxmox node.

This is an example of one of my nodes. As you can see, my GPUs, USB Controllers, Wireless adapter has their own dedicated group.

If your devices are in the same group, be sure to double check for ACS (Access Control Service) in your BIOS. Should you not find it anywhere, you can apply an override patch by including pcie_acs_override=downstream,multifunction in your grub or systemd file, if so go back to the step where we edit our said system file.

Blacklisting Modules

Blacklisting driver modules (drivers) and devices is recommended to give the VM(s) proper access to your devices, Proxmox might not always want to let go of one, causing them unavailable for the VM(s).

Let’s start off with the modules, edit the following file

nano /etc/modprobe.d/pve-blacklist.conf 

Add the following lines

blacklist nouveau
blacklist nvidia
blacklist nvidiafb
blacklist snd_hda_codec_hdmi
blacklist snd_hda_intel
blacklist snd_hda_codec
blacklist snd_hda_core
blacklist radeon
blacklist amdgpu

Hit Ctrl + X -> Y -> Enter to save changes.

Blacklisting Devices

Finding the corresponding IDs for your PCI devices, run the following command

lspci -nn | grep -i {device}

{device} = vga, nvidia, usb, audio, wireless, and so on. Don’t include the { } brackets.

You should then see a list similar to below;
“0x.00.x USB controller … [1234:5678]”
“0x:00.x VGA compatible controller … [1234:5678]”
“0x.00.x Audio Device … [1234:5678]”
Not necessary to take note of the GPU Audio ID.
Only do one device reference at the time, and take note of the IDs you need.

As you can see here, the ids for my GPUs are 10de:1287 and 1002:67d.
You might want to look for your USB Controller ids as well, if you encounter any issues with USB Passthrough down the line.
Keep yours noted for the next step!

In general it’s recommended to add the IDs for all devices you plan on using for a VM. What the blacklisting essentially does, is to block the devices for Proxmox holding them hostage, and not letting a VM have full access to it, at least in most cases.

Blacklisting the PCI device IDs for the host, edit the following file

nano /etc/modprobe.d/vfio-pci.conf

Add your Device IDs in this file like this;

options vfio-pci ids=1234:5678,1234:5678 disable_vga=1

Hit Ctrl + X -> Y -> Enter to save changes.

Again, do not add the GPU Audio ID.
Note that adding disable_vga=1 here will probably prevent guests from booting in SeaBIOS mode, if you use that for anything.

You can also add disable_idle_d3=1 to the end of the line, essentially disables the D3 Device Power State; which will prevent certain hardware to enter a low-power mode, that might cause issues when some are passed through, read more here. Does no harm with disabling it, might only consume more power down the line. Has shown better stability for Thunderbolt cards for example.

Now here you want it looking like this.

Now when you got all that sorted, you want to reboot your Host machine once again.

reboot

Adding your Devices to your VM(s)

This is the setting you want to have set for your GPU to fully work. For some, only All Functions and ROM-Bar can be ticked. Do your own experimenting with these options!

Some cards also requires a dumped .rom file for it to work. To acquire that you must dump the file yourself from your specific card, look for the guide further down in this post.

We recently discovered that if PCI-Express is not ticked, things like HDMI-Audio might not work. But do note that your install might not boot with it ticked, then have it un-ticked.

When your Host is up and running again, we can finally start adding in our accessories to our VMs! In this case, I’m using a macOS VM, it’s the same for Windows/Linux VM(s).

This is totally normal with the Console window whenever a GPU is added or Display is set to none.

So, make sure you got your monitor hooked up or at least a dummy plug so that you are able to remote into the VM, with your preferred Remote App.