Information Technology Grimoire

Version .0.0.1

IT Notes from various projects because I forget, and hopefully they help you too.

GPU Passthrough

Setup Prox Sources

apt update
apt install vim net-tools

Apt Sources

vim /etc/apt/sources.list.d/pve-enterprise.list 

# remove enterprise if you aren't paying subscription
#deb https://enterprise.proxmox.com/debian/pve bullseye pve-enterprise

# add in community
deb http://deb.debian.org/debian bullseye main contrib non-free
deb http://deb.debian.org/debian bullseye-updates main contrib non-free
deb http://deb.debian.org/debian bullseye-backports main contrib non-free

# add in proxmox community
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

# and debian security
deb http://security.debian.org/debian-security/ bullseye-security main contrib non-free

do updates

apt update && apt upgrade -y

disable ipv6 apt

echo 'Acquire::ForceIPv4 "true";' | tee /etc/apt/apt.conf.d/99force-ipv4

BIOS

Verify Vt-x

apt install cpu-checker

kvm-ok

root@pve128:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

modprobe/rdmsr

3 or 5 is good

root@pve128:~# modprobe msr
root@pve128:~# rdmsr 0x3a
5

iommu

vim /etc/default/grub

For Intel CPUs:

GRUB_CMDLINE_LINUX_DEFAULT="quiet initcall_blacklist=sysfb_init intel_iommu=on"

update grub after edits

update-grub

/etc/modules

vim /etc/modules 

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

IOMMU interrupt remapping

proxmox warns this could cause instability

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

other modprobe edits

echo "options snd-hda-intel enable_msi=1"> /etc/modprobe.d/snd-hda-intel.conf

Blacklisting Drivers

You can see existing drivers attached to device, that you will need to blacklist so proxmox doesn’t use:

lspic -nnk 

echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidiafb" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
echo "blacklist snd_hda_intel" >> /etc/modprobe.d/blacklist.conf

what it looks like before these edits take effect:

06:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2216] (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. Device [19da:1612]
        Kernel driver in use: nouveau
        Kernel modules: nvidiafb, nouveau
06:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. Device [19da:1612]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

this is the desired result after loading vfio and disabling other modules, though it won’t look like this yet

06:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2216] (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. Device [19da:1612]
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau
06:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. Device [19da:1612]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

Add GPU to VFIO

Find your video card ID:

lspci -v | grep NVIDIA 

06:00.0 VGA compatible controller: NVIDIA Corporation Device 2216 (rev a1) (prog-if 00 [VGA controller])
06:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1)

Get the vendor ID codes:

lspci -n -s 06:00 

06:00.0 0300: 10de:2216 (rev a1)
06:00.1 0403: 10de:1aef (rev a1)

Modify the vfio to disable vga

echo "options vfio-pci ids=10de:2216,10de:1aef,19da:1612 disable_vga=1"> /etc/modprobe.d/vfio.conf

update and restart

update-grub
update-initramfs -u
reset
reboot

Verify Remapping

root@pve128:~# dmesg | grep remapping 

[    0.362971] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.364368] DMAR-IR: Enabled IRQ remapping in x2apic mode

Verify vfi modules

lsmod | grep vfi 

vfio_pci               16384  0
vfio_pci_core          73728  1 vfio_pci
vfio_virqfd            16384  1 vfio_pci_core
irqbypass              16384  2 vfio_pci_core,kvm
vfio_iommu_type1       40960  0
vfio                   45056  2 vfio_pci_core,vfio_iommu_type1

Verify iommu Groups

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nnks "${d##*/}"; done; 

<snip>
IOMMU group 17 06:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2216] (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. Device [19da:1612]
        Kernel modules: nvidiafb, nouveau
IOMMU group 17 06:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. Device [19da:1612]
        Kernel modules: snd_hda_intel
<snip>

or

find /sys/kernel/iommu_groups/ -type l 

/sys/kernel/iommu_groups/17/devices/0000:06:00.0
/sys/kernel/iommu_groups/17/devices/0000:06:00.1
/sys/kernel/iommu_groups/7/devices/0000:00:19.0
/sys/kernel/iommu_groups/7/devices/0000:00:19.1
/sys/kernel/iommu_groups/15/devices/0000:04:00.0
/sys/kernel/iommu_groups/5/devices/0000:00:16.0
/sys/kernel/iommu_groups/13/devices/0000:01:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.2
/sys/kernel/iommu_groups/3/devices/0000:00:14.0
/sys/kernel/iommu_groups/11/devices/0000:00:1c.4
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.0
/sys/kernel/iommu_groups/16/devices/0000:05:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:17.0
/sys/kernel/iommu_groups/14/devices/0000:02:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:15.1
/sys/kernel/iommu_groups/4/devices/0000:00:15.2
/sys/kernel/iommu_groups/4/devices/0000:00:15.0
/sys/kernel/iommu_groups/4/devices/0000:00:15.3
/sys/kernel/iommu_groups/12/devices/0000:00:1f.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.5
/sys/kernel/iommu_groups/12/devices/0000:00:1f.3
/sys/kernel/iommu_groups/12/devices/0000:00:1f.4
/sys/kernel/iommu_groups/2/devices/0000:00:06.0
/sys/kernel/iommu_groups/10/devices/0000:00:1c.3
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.2

If you do not have iommu groups, then you will get an error when you try to select:

0dc0d3b69d8bd312a1bdec8475698cd2.png

Verify IOMMU + DMAR

dmesg | grep -e DMAR -e IOMMU 

[    0.019691] ACPI: DMAR 0x00000000390FF000 000050 (v02 INTEL  EDK2     00000002      01000013)
[    0.019741] ACPI: Reserving DMAR table memory at [mem 0x390ff000-0x390ff04f]
[    0.364458] DMAR: Host address width 39
[    0.364458] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.364464] DMAR: dmar0: reg_base_addr fed91000 ver 5:0 cap d2008c40660462 ecap f050da
[    0.364467] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 0
[    0.364468] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.364469] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.365868] DMAR-IR: Enabled IRQ remapping in x2apic mode

Verify /proc/cmdline

cat /proc/cmdline

before

BOOT_IMAGE=/boot/vmlinuz-5.15.83-1-pve root=/dev/mapper/pve-root ro quiet

after

BOOT_IMAGE=/boot/vmlinuz-5.15.83-1-pve root=/dev/mapper/pve-root ro quiet initcall_blacklist=sysfb_init intel_iommu=on

Verify ecap

dmesg | grep ecap 

[    0.364464] DMAR: dmar0: reg_base_addr fed91000 ver 5:0 cap d2008c40660462 ecap f050da

On the IOMMU lines, the hexadecimal value after “ecap” indicates whether interrupt remapping is supported. If the last character of this value is an 8, 9, a, b, c, d, e, or an f, interrupt remapping is supported. For example, “ecap 1000” indicates there is no interrupt remapping support. “ecap 10207f” indicates interrupt remapping support, as the last character is an “f”.

VM Setup

4b700ffcc4f76318880bfcc7b73a6324.png 7f065f3ad260931dcf4541543fbca866.png c10abea52ebe08450b64d058dbfbf57f.png 0aceed3a611ded56adfdf687f56c57bd.png eb6b448e971ffe01b6bfc867df64b1ef.png 945c2124839963f58b27defa8ec8fdd7.png a0033f6d92180c74a6796e0b6eeed62c.png

Final version (note, differs than the screenshots above)

757f55d78952750f530800f677f16bb9.png

Misc Notes

https://192.168.7.33:8006/pve-docs/chapter-qm.html#qm_pci_passthrough_vm_config

HDMI Kernel 5.15

since kernel 5.15 windows VM boot only works after boot host with HDMI disconnected. Monitor power supply disconnect is not enough, need to unplug hdmi cable.

They make a loopback plug for this too.

Direct to Monitor

If you want it directly to monitor you will need to pass through Keyboard and mouse too, but hook monitor into GPU card, and set the video display to “none” to force it. It will no longer allow vnc when you do this.

Windows Install

Don’t forget to enable RDP before getting too far in. You will need tested RDP before you start monkeying with other stuff.

Drivers

https://pve.proxmox.com/wiki/Windows_VirtIO_Drivers https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.225-2/

BIOS Resolution

In case you can’t get the resolution you want, force it in F2 Bios of VM

86cc5e22dea41707f040b9851df7810f.png

RDP

placeholder

Hide Virtual CPU

You need to hide the virtual cpu from itself because some programs don’t like it

vim /etc/pve/nodes/pve128/qemu-server/742.conf

before:

bios: ovmf
boot: order=scsi0;net0
cores: 2
cpuunits: 1024
efidisk0: local-lvm:vm-742-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:06:00,pcie=1
machine: pc-q35-7.1
memory: 16384
meta: creation-qemu=7.1.0,ctime=1675639736
name: win10gpupt
net0: virtio=E6:C4:47:DC:7C:6D,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsi0: local-lvm:vm-742-disk-1,cache=writeback,size=350G
scsihw: virtio-scsi-pci
smbios1: uuid=5467c817-e172-4a79-944d-8d03e44903eb
sockets: 2
vmgenid: 2632d05d-3bcb-455d-b4a8-a72e5f4e269a

add this line after cores:

cpu: host,hidden=1,flags=+pcid

Error Message/Fixes

pci device listed 2x (like video + sound)

GPU passthrough issue: Device is already attached

need to hit “any key”

Failed to start Boot0001 "UEFI QEMU DVD-ROM" Time out

Faster RDP

virtio-win-guest-tools wizard to install the QEMU Guest Agent and the SPICE agent for an improved remote-viewer experience.

see also: https://pve.proxmox.com/wiki/SPICE

https://www.amazon.com/gp/product/B074FT1P8M/ref=as_li_ss_tl?ie=UTF8&psc=1&linkCode=sl1&tag=technotim-20&linkId=3b9392ed95611c91ec566a04f8c45719&language=en_US

Proxmox VM Settings

???

Set Display to VirtIO-GPU (virtio)
Set PCI Device as Primary GPU

Moonlight

https://github.com/moonlight-stream/
https://github.com/moonlight-stream/moonlight-qt/releases