GPU Passthrough On FreeBSD 14.3-RELEASE - Gaming in a Virtual Machine - AMD CPU/GPU + Windows 10 Pro
Author: Jonathan Vasquez <jon@xyinn.org>
Last Updated: 2025-09-24-0800
Running: FreeBSD 14.3-RELEASE-p2 GENERIC amd64
pub fn main() !void
The below documentation shows how to get a working GPU Passthrough setup for the following configuration primarily for gaming, but can also be used for other tasks requiring 3D acceleration:
- CPU:
AMD Ryzen 9 7950X 16-Core Processor
- GPU:
AMD Radeon RX 6900 XT
- Host OS:
FreeBSD 14.3-RELEASE
- Guest OS:
Windows 10 Pro
- Hypervisor:
bhyve
- Bhyve Firmware:
1.0_2
This will not work for Intel/NVIDIA as is. You will need to tweak some of my instructions for your specific config.
I want to give a shout out to Corvin Köhne. He has done a lot of work over the years to make all of this possible on FreeBSD. Thank you for everything and thanks for answering my questions :).
Videos
Remarks
-
I'm tempted to say that I finally have a stable setup that no longer has random crashes, but I'll need to test it for longer before I say that. However, after my recent PCI passthrough adjustments, I haven't received any crashes for about 20~ hours. I've also rebooted the VM/Host a few times to make sure that it is a deterministically stable setup, and so far so good. In the situations where the VM would previously crash, the host would still be alive, so that's good. I was previously debugging the errors I was receiving and my random crashes may have been related to the USB controller I originally passed in (the one connected to
18:0:X
, which is a multi-functional port containing the integrated graphics card). There may have been some signals being mixed which caused the instability. I was able to switch to using a completely different USB controller on13:0:0
, which just has USB ports, nothing else. So far this seems to have stabilized the VM. I'll update after I'm able to test this for longer. So far I've made this my primary gaming machine. -
If you want to use this for gaming purposes (which is my main purpose), you'll want to:
- Connect your monitor's cable directly to your GPU. Do not use RDP/VNC for your actual day-to-day experience.
- You'll want to pass in some of your mobo's USB ports over to the VM so you can use your keyboard/mouse.
- You'll want to pass in whatever device you will be using for sound.
For my case, I have a KVM that has my keyboard/wireless receivers, sound via its headphone jack, and ethernet all connected to it. The KVM's USB cable is connected to one of the passthrough USB ports on the mobo. This means I can easily switch between the Windows VM and my laptop with one click on the KVM. Due note I only use Windows in an offline capacity exclusively, so I disabled the Ethernet adapter in Windows and I never let Windows connect to the internet even during installation. You can read more about my Dark Island stategy at the previously mentioned link.
More info at the Devices To Passthrough section below.
-
You must disable
Resize BAR Support
if you have that in the BIOS. This was the main setting that prevented my card from working for my setup. FreeBSD doesn't currently support this so if this is enabled and the Host or VM decide to resize, you will get unexpected results (most likely Code 43). Disabling this means you won't be able to take full advantage of the GPU's memory. From what Corvin mentioned to me, this should only give you a penalty if you are doing memory intensive operations, not necessarily computation intensive operations. So we might be fine for most games.. Although I think FreeBSD will eventually have Resize BAR support. -
Since AMD cards suffer from the old AMD Hardware Reset Bug, the card will only work the first time you start up the VM from the host. As long as you don't reboot the VM, you should be fine. If you need to reboot the VM, you'll need to restart the host to get the card working again.
-
After you have installed the GFX drivers, the Windows VM should boot up with the card already working. So if you have your monitor cable directly connected to the card, you will get video output immediately. If you don't, then something is wrong.
-
If you aren't gaming and want good performance, you should use Remote Desktop (RDP) instead of VNC. You can use VNC to install the VM, and then enable RDP after. If you are gaming, follow the other instructions I mentioned.
-
So far I've gotten 100% game compatibility. I can't guarantee that all of your games will work, but all of mine currently are. Although I am not playing any multiplayer games or games that have DRM (I'm not including old school 2000s DRM like SecuROM since this type of DRM doesn't negatively affect me at this point lol). Most of my games so far seem to run relatively smoothly (including Cyperpunk 2077 and Witcher III). Games like Baldur's Gate 3 make take a bit of time to start up when you open it, but once you are in game it seems to run smoothly.
-
I'll be showing how to do this using a raw bhyve setup, and not use
vm-bhyve
for now. This allows us to directly interact and debug the core technologies without abstractions. Since we don't need to pass in the vBIOS file through the VM when using Windows, it should be relatively easy to do this same setup in vm-bhyve. -
When you are installing Windows, you'll want to only use 1 CPU and 1 Core. You can change it to whatever you want afterwards.
-
You can ignore the errors regarding the 4 COM devices in the Device Manager, or the failed QEMU FwCfg Device. They don't seem to cause any issues.
-
If you are using GPU-Z to check your card info, you should be able to see some checkboxes at the bottom of the first tab under the Computing and Technologies sections. If you don't see stuff enabled there, then your passthrough is probably not working. A proper passthrough should also show your card in the Windows' Task Manager. There should be an entry there for
GPU 0
. -
Given that this card is installed on my server, and I'm not messing with installing Windows on it (not even on an external hard drive, I don't want to deal with its bootloader hijacking shenanigans), I'm not sure how much performance is being lost by using this passthrough setup. But I guess good enough for me lol?
Screenshots
Heaven Benchmark
Furmark
Grim Dawn
Terraria
Cyberpunk 2077: Phantom Liberty
Baldur's Gate 3
The Witcher III
Hollow Knight
Hollow Knight - Silksong
It Lurks Below
Devices To Passthrough
There are a few things we'll want to have connected directly into the VM (passthrough) to be able to game on it comfortably. This would be:
- A dedicated graphics card (used for video output and optimal performance)
- A keyboard and mouse (used for input and to play the games and other OS tasks)
- Sound (used to hear the games)
My KVM has multiple USB ports, a headphone jack, and an ethernet port. The KVM
itself is connected to the host machine via a single USB cable. This means that
we can get everything we need for the VM if we can pass in a single USB port
into the VM. Luckily after many trial and error, I was able to look at all of
the subclass = USB
devices in pciconf -vl
, passed them into the
VM, and test each port on my box to see which controller is controlling which
port. For my previous instructions I was using the USB controller that's connected
to the same bus that my integrated card is running on. I believe I may have been
experiencing VM crashes because of this at undetermined points in time. I have
now been able to switch over to a separate USB controller that bypasses the
integrated card bus completely. I'm hoping that there being less things being
passed in, and signals crossed, will increase VM stability. Only time will tell.
My motherboard seems to have 4 USB controllers, although I wasn't able to find the ports connected to the "USB 2.0" controller. But for the other three remaining, I was able to choose the following one:
ppt2@pci0:13:0:0: class=0x0c0330 rev=0x01 hdr=0x00 vendor=0x1022 device=0x43f7 subvendor=0x1b21 subdevice=0x1142
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = '600 Series Chipset USB 3.2 Controller'
class = serial bus
subclass = USB
This one was chosen after my testing each port and seeing which ports would be comfortable and stable for my setup. I want to leave the other USB controllers on the host since I also have an external hdd connected to the host which has automatic ZFS incremental backups being sent over to it. I was able to connect my KVM and also the host's external backup hdd to two separate SS10 ports, so I'm still able to get pretty good performance transferring files from the LAN into the VM, and also doing external backups.
As for the graphic card, this was actually more straightforward. We have the AMD
card sitting on Bus 3
, and it has two functions: 0
, and 1
. We
will need to pass both of these into the VM later. It looks as follows:
ppt0@pci0:3:0:0: class=0x030000 rev=0xc0 hdr=0x00 vendor=0x1002 device=0x73af subvendor=0x1043 subdevice=0x04fe
vendor = 'Advanced Micro Devices, Inc. [AMD/ATI]'
device = 'Navi 21 [Radeon RX 6900 XT]'
class = display
subclass = VGA
ppt1@pci0:3:0:1: class=0x040300 rev=0x00 hdr=0x00 vendor=0x1002 device=0xab28 subvendor=0x1002 subdevice=0xab28
vendor = 'Advanced Micro Devices, Inc. [AMD/ATI]'
device = 'Navi 21/23 HDMI/DP Audio Controller'
class = multimedia
subclass = HDA
Finding the right USB ports is a bit time consuming but it's well worth
finding the right combination for you. There were very weird behaviors I saw
where I would enable a USB controller to pass into the VM, and all of a sudden
a bluetooth device would become available (in a failed state) inside of the VM.
Thus it seems that the way that vendors are wiring their peripherals is not
directly assumable. Meaning a USB controller may not just be a USB controller.
There may be some other stuff connected there. When I attempted to install the
Bluetooth drivers to this "bluetooth device", I got a BSOD with a
Kernel Security Check Failure
error code. There is also documentation at
this link mentioning some weirdness
with multi-functional buses.
Required UEFI/BIOS Changes
The following options have either been enabled or disabled in UEFI:
- Enable:
SVM
- Enable:
IOMMU
- Enable:
Above 4G Decoding
- Disable:
Resize BAR Support
- FreeBSD doesn't support this and this was the main culprit that was preventing my card from working in the Windows VM.
- Disable:
CSM
. The system should be running inUEFI
mode.
If IOMMU
is properly enabled, you should see the following in your FreeBSD
host:
# dmesg | grep IOMMU
ivhd0: <AMD-Vi/IOMMU ivhd with EFR> on acpi0
Install Some Packages
We can start off by installing the bhyve firmware.
pkg install bhyve-firmware
You'll need to install a VNC and RDP client on the machine that will be
connecting to the VM. On my FreeBSD laptop I have installed: tigervnc
and
xfreerdp
.
Configure the system for bhyve, passthrough, etc
The following settings allow the VM system to be used, allows for IOMMU to work on AMD, and also specifies the PCI devices for our dedicated card so that they aren't attached to the host and allows us to pass it into bhyve later.
NOTE: The PCI values below depend on your machine. Use pciconf -vl
to
find them.
Load the bhyve kernel driver
kldload vmm
Add some configuration options to your bootloader config
/boot/loader.conf
:
# Load Bhyve VM Driver
vmm_load="YES"
# Allow AMD-Vi (IOMMU) Passthrough Support
hw.vmm.amdvi.enable="1"
# Prevent FreeBSD from reserving these slots. We will use them for: GPU / USB / Audio.
pptdevs="3/0/0 3/0/1 13/0/0"
Create your storage backend
I'm just going to create a simple flat file for now. Adjust the size to your needs.
truncate -s 100G disk0.img
Create the network devices
TAP Devices
This tunnable will allow your TAP devices to switch to the UP
state as
soon as they are created. This TAP device will also be what we will give to
the VM so it can communicate back up to the host via the FreeBSD Kernel.
Let's first run the following commands:
sysctl net.link.tap.up_on_open=1
ifconfig tap0 create
and now add the following to your configuration so it persists across reboots:
/etc/sysctl.conf
:
# Allow TAP devices to switch to UP state when they are created.
net.link.tap.up_on_open=1
Bridge
This will create a network device for the VM to use, and a bridge so that our host network adapter and the VM network adapter can communicate on the same network.
ifconfig bridge0 create addm ix0 addm tap0
ifconfig bridge0 up
NOTE: Replace ix0
with your host's network interface.
If you want to make these permanent, you can add something like this to your
/etc/rc.conf
:
cloned_interfaces="bridge0 tap0"
ifconfig_bridge0="addm ix0 addm tap0 up"
Now the bridge0
and tap0
interfaces will be created at start up,
and the bridge will be configured so both of these interfaces can communicate
with each other.
Bhyve Scripts
start script
This is the main script that will start your vm. Save this as start.sh
in a folder called gaming
somewhere on your system:
#!/bin/sh
VM_DIR="$(dirname $(realpath $0))"
VM_NAME="$(basename $VM_DIR)"
cd "$VM_DIR"
bhyve -AHPSw -c sockets=1,cores=16,threads=1 -m 32G \
-s 0,hostbridge \
-s 1,nvme,disk0.img \
-s 3:0,passthru,3/0/0 \
-s 3:1,passthru,3/0/1 \
-s 13:0,passthru,13/0/0 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu \
-o console=stdio \
$VM_NAME
# Exit the script here and leave some options for debugging more
# ergonomically after the exit line.
exit
# Only use this when installing the VM for the first time:
# VNC. Once you install Windows and enable RDP, you can turn this off.
-s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,wait \
# The Windows ISO and the VirtIO Drivers ISO.
# You can download the latest stable virtio drivers at:
# https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso
-s 4,ahci-cd,../files/Win10_22H2_English_x64v1_2023.iso \
-s 5,ahci-cd,../files/virtio-win-0.1.271.iso \
# If you want a network device without using virtio, you can use e1000.
# You should install the virtio drivers and use virtio-net instead for
# better performance.
-s 2,e1000,tap0 \
-s 2,virtio-net,tap0 \
stop script
This is your stop script. Save this as stop.sh
somewhere.
NOTE:
This is currently set up to kill any process with bhyve
in
the name. On this system I'm only running one bhyve VM, which is the gaming one.
Adjust accordingly to restrict its scope.
#!/bin/sh
# Send SIGTERM twice to make sure Windows listens to
# the ACPI shutdown signal.
pkill bhyve
pkill bhyve
# Wait a bit for the guest to shutdown properly before
# we continue shutting down the host.
sleep 10
bhyvectl --vm=gaming --destroy
rc.d script
This script will allow you to start the VM automatically at boot time. Save this
file in your /usr/local/etc/rc.d/
directory with the name vm_gaming
.
Make sure to adjust any paths to where you saved your start/stop scripts:
NOTE:
At the moment I noticed that if I do a shutdown -r now
, I don't
actually see Windows gracefully shutting down (like when I send the ACPI shutdown
signal). Currently looking into this.
#!/bin/sh
# PROVIDE: vm_gaming
# REQUIRE: LOGIN
# KEYWORD: nojail shutdown
. /etc/rc.subr
name=vm_gaming
rcvar=vm_gaming_enable
start_cmd="${name}_start"
stop_cmd="${name}_stop"
restart_cmd="${name}_restart"
path_to_start_script="/atlantis/vms/gaming/start.sh"
path_to_stop_script="/atlantis/vms/gaming/stop.sh"
vm_gaming_start()
{
daemon \
-o /var/log/${name}.log \
"${path_to_start_script}"
}
vm_gaming_stop()
{
"${path_to_stop_script}"
}
vm_gaming_restart()
{
# NOTE: AMD users will most likely experience the famous
# AMD Hardware Reset Bug. This means that after you reboot
# the guest, you most likely won't have video out. If this
# happens, you'll need to restart the host. Sometimes this
# command will work for AMD users though. So you can try a
# restart and see if it works.
vm_gaming_stop
vm_gaming_start
}
load_rc_config $name
: ${vm_gaming_enable:="NO"}
run_rc_command "$1"
Now you can add vm_gaming_enable="YES"
to /etc/rc.conf
so that
your gaming VM starts up automatically at boot time.
pciconf -vl
hostb0@pci0:0:0:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x14d8 subvendor=0x1043 subdevice=0x8877
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge Root Complex'
class = bridge
subclass = HOST-PCI
amdviiommu0@pci0:0:0:2: class=0x080600 rev=0x00 hdr=0x00 vendor=0x1022 device=0x14d9 subvendor=0x1043 subdevice=0x8877
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge IOMMU'
class = base peripheral
subclass = IOMMU
hostb1@pci0:0:1:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x14da subvendor=0x0000 subdevice=0x0000
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge Dummy Host Bridge'
class = bridge
subclass = HOST-PCI
pcib1@pci0:0:1:1: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x14db subvendor=0x1043 subdevice=0x8877
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge GPP Bridge'
class = bridge
subclass = PCI-PCI
pcib4@pci0:0:1:2: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x14db subvendor=0x1043 subdevice=0x8877
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge GPP Bridge'
class = bridge
subclass = PCI-PCI
hostb2@pci0:0:2:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x14da subvendor=0x0000 subdevice=0x0000
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge Dummy Host Bridge'
class = bridge
subclass = HOST-PCI
pcib5@pci0:0:2:1: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x14db subvendor=0x1043 subdevice=0x8877
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge GPP Bridge'
class = bridge
subclass = PCI-PCI
pcib17@pci0:0:2:2: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x14db subvendor=0x1043 subdevice=0x8877
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge GPP Bridge'
class = bridge
subclass = PCI-PCI
hostb3@pci0:0:3:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x14da subvendor=0x0000 subdevice=0x0000
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge Dummy Host Bridge'
class = bridge
subclass = HOST-PCI
hostb4@pci0:0:4:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x14da subvendor=0x0000 subdevice=0x0000
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge Dummy Host Bridge'
class = bridge
subclass = HOST-PCI
hostb5@pci0:0:8:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x14da subvendor=0x0000 subdevice=0x0000
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge Dummy Host Bridge'
class = bridge
subclass = HOST-PCI
pcib18@pci0:0:8:1: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x14dd subvendor=0x1043 subdevice=0x8877
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A]'
class = bridge
subclass = PCI-PCI
pcib19@pci0:0:8:3: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x14dd subvendor=0x1043 subdevice=0x8877
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A]'
class = bridge
subclass = PCI-PCI
none0@pci0:0:20:0: class=0x0c0500 rev=0x71 hdr=0x00 vendor=0x1022 device=0x790b subvendor=0x1043 subdevice=0x8877
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'FCH SMBus Controller'
class = serial bus
subclass = SMBus
isab0@pci0:0:20:3: class=0x060100 rev=0x51 hdr=0x00 vendor=0x1022 device=0x790e subvendor=0x1043 subdevice=0x8877
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'FCH LPC Bridge'
class = bridge
subclass = PCI-ISA
hostb6@pci0:0:24:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x14e0 subvendor=0x0000 subdevice=0x0000
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge Data Fabric; Function 0'
class = bridge
subclass = HOST-PCI
hostb7@pci0:0:24:1: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x14e1 subvendor=0x0000 subdevice=0x0000
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge Data Fabric; Function 1'
class = bridge
subclass = HOST-PCI
hostb8@pci0:0:24:2: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x14e2 subvendor=0x0000 subdevice=0x0000
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge Data Fabric; Function 2'
class = bridge
subclass = HOST-PCI
hostb9@pci0:0:24:3: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x14e3 subvendor=0x0000 subdevice=0x0000
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge Data Fabric; Function 3'
class = bridge
subclass = HOST-PCI
hostb10@pci0:0:24:4: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x14e4 subvendor=0x0000 subdevice=0x0000
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge Data Fabric; Function 4'
class = bridge
subclass = HOST-PCI
hostb11@pci0:0:24:5: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x14e5 subvendor=0x0000 subdevice=0x0000
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge Data Fabric; Function 5'
class = bridge
subclass = HOST-PCI
hostb12@pci0:0:24:6: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x14e6 subvendor=0x0000 subdevice=0x0000
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge Data Fabric; Function 6'
class = bridge
subclass = HOST-PCI
hostb13@pci0:0:24:7: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x14e7 subvendor=0x0000 subdevice=0x0000
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge Data Fabric; Function 7'
class = bridge
subclass = HOST-PCI
pcib2@pci0:1:0:0: class=0x060400 rev=0xc0 hdr=0x01 vendor=0x1002 device=0x1478 subvendor=0x0000 subdevice=0x0000
vendor = 'Advanced Micro Devices, Inc. [AMD/ATI]'
device = 'Navi 10 XL Upstream Port of PCI Express Switch'
class = bridge
subclass = PCI-PCI
pcib3@pci0:2:0:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1002 device=0x1479 subvendor=0x1002 subdevice=0x1479
vendor = 'Advanced Micro Devices, Inc. [AMD/ATI]'
device = 'Navi 10 XL Downstream Port of PCI Express Switch'
class = bridge
subclass = PCI-PCI
ppt0@pci0:3:0:0: class=0x030000 rev=0xc0 hdr=0x00 vendor=0x1002 device=0x73af subvendor=0x1043 subdevice=0x04fe
vendor = 'Advanced Micro Devices, Inc. [AMD/ATI]'
device = 'Navi 21 [Radeon RX 6900 XT]'
class = display
subclass = VGA
ppt1@pci0:3:0:1: class=0x040300 rev=0x00 hdr=0x00 vendor=0x1002 device=0xab28 subvendor=0x1002 subdevice=0xab28
vendor = 'Advanced Micro Devices, Inc. [AMD/ATI]'
device = 'Navi 21/23 HDMI/DP Audio Controller'
class = multimedia
subclass = HDA
nvme0@pci0:4:0:0: class=0x010802 rev=0x00 hdr=0x00 vendor=0x144d device=0xa80a subvendor=0x144d subdevice=0xa801
vendor = 'Samsung Electronics Co Ltd'
device = 'NVMe SSD Controller PM9A1/PM9A3/980PRO'
class = mass storage
subclass = NVM
pcib6@pci0:5:0:0: class=0x060400 rev=0x01 hdr=0x01 vendor=0x1022 device=0x43f4 subvendor=0x1b21 subdevice=0x3328
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = '600 Series Chipset PCIe Switch Upstream Port'
class = bridge
subclass = PCI-PCI
pcib7@pci0:6:0:0: class=0x060400 rev=0x01 hdr=0x01 vendor=0x1022 device=0x43f5 subvendor=0x1b21 subdevice=0x3328
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = '600 Series Chipset PCIe Switch Downstream Port'
class = bridge
subclass = PCI-PCI
pcib8@pci0:6:4:0: class=0x060400 rev=0x01 hdr=0x01 vendor=0x1022 device=0x43f5 subvendor=0x1b21 subdevice=0x3328
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = '600 Series Chipset PCIe Switch Downstream Port'
class = bridge
subclass = PCI-PCI
pcib9@pci0:6:8:0: class=0x060400 rev=0x01 hdr=0x01 vendor=0x1022 device=0x43f5 subvendor=0x1b21 subdevice=0x3328
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = '600 Series Chipset PCIe Switch Downstream Port'
class = bridge
subclass = PCI-PCI
pcib15@pci0:6:12:0: class=0x060400 rev=0x01 hdr=0x01 vendor=0x1022 device=0x43f5 subvendor=0x1b21 subdevice=0x3328
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = '600 Series Chipset PCIe Switch Downstream Port'
class = bridge
subclass = PCI-PCI
pcib16@pci0:6:13:0: class=0x060400 rev=0x01 hdr=0x01 vendor=0x1022 device=0x43f5 subvendor=0x1b21 subdevice=0x3328
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = '600 Series Chipset PCIe Switch Downstream Port'
class = bridge
subclass = PCI-PCI
pcib10@pci0:9:0:0: class=0x060400 rev=0x01 hdr=0x01 vendor=0x1022 device=0x43f4 subvendor=0x1b21 subdevice=0x3328
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = '600 Series Chipset PCIe Switch Upstream Port'
class = bridge
subclass = PCI-PCI
pcib11@pci0:10:0:0: class=0x060400 rev=0x01 hdr=0x01 vendor=0x1022 device=0x43f5 subvendor=0x1b21 subdevice=0x3328
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = '600 Series Chipset PCIe Switch Downstream Port'
class = bridge
subclass = PCI-PCI
pcib12@pci0:10:8:0: class=0x060400 rev=0x01 hdr=0x01 vendor=0x1022 device=0x43f5 subvendor=0x1b21 subdevice=0x3328
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = '600 Series Chipset PCIe Switch Downstream Port'
class = bridge
subclass = PCI-PCI
pcib13@pci0:10:12:0: class=0x060400 rev=0x01 hdr=0x01 vendor=0x1022 device=0x43f5 subvendor=0x1b21 subdevice=0x3328
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = '600 Series Chipset PCIe Switch Downstream Port'
class = bridge
subclass = PCI-PCI
pcib14@pci0:10:13:0: class=0x060400 rev=0x01 hdr=0x01 vendor=0x1022 device=0x43f5 subvendor=0x1b21 subdevice=0x3328
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = '600 Series Chipset PCIe Switch Downstream Port'
class = bridge
subclass = PCI-PCI
ix0@pci0:11:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1557 subvendor=0x1dcf subdevice=0x0317
vendor = 'Intel Corporation'
device = '82599 10 Gigabit Network Connection'
class = network
subclass = ethernet
ppt2@pci0:13:0:0: class=0x0c0330 rev=0x01 hdr=0x00 vendor=0x1022 device=0x43f7 subvendor=0x1b21 subdevice=0x1142
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = '600 Series Chipset USB 3.2 Controller'
class = serial bus
subclass = USB
ahci0@pci0:14:0:0: class=0x010601 rev=0x01 hdr=0x00 vendor=0x1022 device=0x43f6 subvendor=0x1b21 subdevice=0x1062
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = '600 Series Chipset SATA Controller'
class = mass storage
subclass = SATA
xhci0@pci0:15:0:0: class=0x0c0330 rev=0x01 hdr=0x00 vendor=0x1022 device=0x43f7 subvendor=0x1b21 subdevice=0x1142
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = '600 Series Chipset USB 3.2 Controller'
class = serial bus
subclass = USB
ahci1@pci0:16:0:0: class=0x010601 rev=0x01 hdr=0x00 vendor=0x1022 device=0x43f6 subvendor=0x1b21 subdevice=0x1062
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = '600 Series Chipset SATA Controller'
class = mass storage
subclass = SATA
nvme1@pci0:17:0:0: class=0x010802 rev=0x00 hdr=0x00 vendor=0x144d device=0xa80a subvendor=0x144d subdevice=0xa801
vendor = 'Samsung Electronics Co Ltd'
device = 'NVMe SSD Controller PM9A1/PM9A3/980PRO'
class = mass storage
subclass = NVM
vgapci0@pci0:18:0:0: class=0x030000 rev=0xc1 hdr=0x00 vendor=0x1002 device=0x164e subvendor=0x1043 subdevice=0x8877
vendor = 'Advanced Micro Devices, Inc. [AMD/ATI]'
device = 'Raphael'
class = display
subclass = VGA
hdac0@pci0:18:0:1: class=0x040300 rev=0x00 hdr=0x00 vendor=0x1002 device=0x1640 subvendor=0x1043 subdevice=0x8877
vendor = 'Advanced Micro Devices, Inc. [AMD/ATI]'
device = 'Rembrandt Radeon High Definition Audio Controller'
class = multimedia
subclass = HDA
none1@pci0:18:0:2: class=0x108000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1649 subvendor=0x1043 subdevice=0x8877
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Family 19h PSP/CCP'
class = encrypt/decrypt
xhci1@pci0:18:0:3: class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1022 device=0x15b6 subvendor=0x1043 subdevice=0x8877
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge USB 3.1 xHCI'
class = serial bus
subclass = USB
xhci2@pci0:18:0:4: class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1022 device=0x15b7 subvendor=0x1043 subdevice=0x8877
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge USB 3.1 xHCI'
class = serial bus
subclass = USB
xhci3@pci0:19:0:0: class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1022 device=0x15b8 subvendor=0x1043 subdevice=0x8877
vendor = 'Advanced Micro Devices, Inc. [AMD]'
device = 'Raphael/Granite Ridge USB 2.0 xHCI'
class = serial bus
subclass = USB
Other
Resize BAR Support (Before/After Disabling)
This was how the BAR information looked like before and after disabling
Resize BAR Support
in the BIOS. Pay attention to the size of Bar 10
and 18:
Resize BAR Support - Enabled
# pciconf -lbevV pci0:3:0:0
ppt0@pci0:3:0:0: class=0x030000 rev=0xc0 hdr=0x00 vendor=0x1002 device=0x73af subvendor=0x1043 subdevice=0x04fe
vendor = 'Advanced Micro Devices, Inc. [AMD/ATI]'
device = 'Navi 21 [Radeon RX 6900 XT]'
class = display
subclass = VGA
bar [10] = type Prefetchable Memory, range 64, base 0xf800000000, size 17179869184, disabled
bar [18] = type Prefetchable Memory, range 64, base 0xfc00000000, size 268435456, disabled
bar [20] = type I/O Port, range 32, base 0xf000, size 256, disabled
bar [24] = type Memory, range 32, base 0xf6a00000, size 1048576, disabled
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
Resize BAR Support - Disabled
# pciconf -lbevV pci0:3:0:0
ppt0@pci0:3:0:0: class=0x030000 rev=0xc0 hdr=0x00 vendor=0x1002 device=0x73af subvendor=0x1043 subdevice=0x04fe
vendor = 'Advanced Micro Devices, Inc. [AMD/ATI]'
device = 'Navi 21 [Radeon RX 6900 XT]'
class = display
subclass = VGA
bar [10] = type Prefetchable Memory, range 64, base 0xfce0000000, size 268435456, disabled
bar [18] = type Prefetchable Memory, range 64, base 0xfcf0000000, size 2097152, disabled
bar [20] = type I/O Port, range 32, base 0xf000, size 256, disabled
bar [24] = type Memory, range 32, base 0xf6a00000, size 1048576, disabled
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
Extract vBIOS (Not Required for Windows)
The vBIOS is not required
to be extracted nor used in bhyve when using Windows.
However, below I'll put the instructions I used to successfully extract the
ROM from my card. FreeBSD does not support this so the easiest way to do this
is to boot a Linux Live CD and extract it from there. You should try and use a
Linux Live CD with a modern kernel in order to extract this file. Originally I
tried using my Linux Mint 22.1 ISO to extract, but no rom file was shown at
/sys/bus/pci/devices/0000\:03\:00.0/rom
. However, when I booted into a
Fedora 42 Live CD running kernel 6.14.0-63.fc42.x86_64
, the file was
available, and I was able to successfully extract it with the following steps:
root@localhost-live:~# echo 1 > /sys/bus/pci/devices/0000\:03\:00.0/rom
root@localhost-live:~# cat /sys/bus/pci/devices/0000\:03\:00.0/rom > vbios_6900.rom
root@localhost-live:~# echo 0 > /sys/bus/pci/devices/0000\:03\:00.0/rom
This is how the directories looked like for devices: 3/0/0
and 3/0/1
:
root@localhost-live:~# ls /sys/bus/pci/devices/0000\:03\:00.0/
aer_dev_correctable current_link_width local_cpulist reset revision
aer_dev_fatal d3cold_allowed local_cpus reset_method rom
aer_dev_nonfatal device max_link_speed resource subsystem
ari_enabled dma_mask_bits max_link_width resource0 subsystem_device
boot_vga driver_override modalias resource0_resize subsystem_vendor
broken_parity_status enable msi_bus resource0_wc uevent
class firmware_node numa_node resource2 vendor
config iommu power resource2_resize waiting_for_supplier
consistent_dma_mask_bits iommu_group power_state resource2_wc
consumer:pci:0000:03:00.1 irq remove resource4
current_link_speed link rescan resource5
root@localhost-live:~# ls /sys/bus/pci/devices/0000\:03\:00.1/
aer_dev_correctable current_link_width iommu msi_bus revision
aer_dev_fatal d3cold_allowed iommu_group msi_irqs sound
aer_dev_nonfatal device irq numa_node subsystem
ari_enabled dma_mask_bits link power subsystem_device
broken_parity_status driver local_cpulist power_state subsystem_vendor
class driver_override local_cpus remove supplier:pci:0000:03:00.0
config enable max_link_speed rescan uevent
consistent_dma_mask_bits firmware_node max_link_width resource vendor
current_link_speed hdaudioC0D0 modalias resource0
I was also able to extract the ROM from within the Windows VM inside of bhyve.
You probably want to do it outside but it worked for me and gave me a ROM file
with the same hash and file size as the amdvbflash
dump. You can also use
the amdvbflash utility that apparently
has been taking down from some other github repos. I also did notice that the file
sizes and contents of the vbios file between the direct Linux extract and the
amdvbflash/GPU-Z versions were different. I just did a basic analysis of the file sizes
and some of the hex contents between both files, and the files start off with the
same hex information, but they eventually diverge in terms of both memory addresses
mentioned, and contents as the file progresses. For example:
SHA1: amdvbflash Extracted ROM: 517e09d7ac6be5da0b0fac7d09f782a3c9494343
File Size: 1.0M
SHA1: GPU-Z Extracted ROM: 517e09d7ac6be5da0b0fac7d09f782a3c9494343
File Size: 1.0M
SHA1: Direct Linux Extracted ROM: abbdfd11c246f570839fa7ad78186dcb12408595
File Size: 119K
It also seems TechPowerUp has a public BIOS collection, but you probably should still dump it directly from your current card for best results since there may be incompatibilities or weirdness if you mix them up. I also would not recommend flashing the vBIOS since you may end up bricking your card.
References
- https://wiki.freebsd.org/bhyve
- https://wiki.freebsd.org/bhyve/pci_passthru
- https://docs.freebsd.org/en/books/handbook/virtualization/#virtualization-host-bhyve
- https://forums.freebsd.org/threads/a-solid-howto-with-bhyve-on-freebsd-14-and-windows-11.92236/
- GPU passthrough with bhyve - Corvin Köhne - EuroBSDcon 2023
- November 2021: FreeBSD Vendor Summit: GPU and TPM2 Passthrough with bhyve, Beckoff
- https://forums.freebsd.org/threads/experience-from-bhyve-freebsd-14-1-gpu-passthrough-with-windows-10-guest.94118/
- https://wiki.freebsd.org/bhyve/Windows
- https://www.ateamsystems.com/tech-blog/howto-windows-10-bhyve-w-freebsd-11/
- https://github.com/isc30/ryzen-gpu-passthrough-proxmox
- https://freebsdfoundation.org/wp-content/uploads/2023/05/a66452da2b483e5e92bde6533b7236c4d11382eb-1-1.pdf
- https://curtisshoward.com/post/fixing-amd-gpu-passthrough-reset-issues-in-windows/
- https://www.reddit.com/r/bhyve/comments/10cwu47/bhyve_gpu_passthrough_windows_11_gaming/