AMD平台虚拟机透传虚拟加密卡

​问题背景​

加密卡在Intel平台下开启IOMMU，在KVM虚拟机上可以正常透传虚拟出来的PCIE加密卡。换成AMD平台后开启IOMMU，在KVM虚拟机上透传虚拟PCIE加密卡就报如下错误：

Error starting domain: internal error: qemu unexpectedly closed the monitor: 2022-03-29T06:11:19.162847Z qemu-kvm: -device vfio-pci,host=01:01.0,id=hostdev0,bus=pci.0,addr=0x9: vfio: error, group 30 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
2022-03-29T06:11:19.162894Z qemu-kvm: -device vfio-pci,host=01:01.0,id=hostdev0,bus=pci.0,addr=0x9: vfio: failed to get group 30
2022-03-29T06:11:19.162913Z qemu-kvm: -device vfio-pci,host=01:01.0,id=hostdev0,bus=pci.0,addr=0x9: Device initialization failed.
2022-03-29T06:11:19.162933Z qemu-kvm: -device vfio-pci,host=01:01.0,id=hostdev0,bus=pci.0,addr=0x9: Device 'vfio-pci' could not be initialized

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 82, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1508, in startup
    self._backend.create()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1080, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-03-29T06:11:19.162847Z qemu-kvm: -device vfio-pci,host=01:01.0,id=hostdev0,bus=pci.0,addr=0x9: vfio: error, group 30 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
2022-03-29T06:11:19.162894Z qemu-kvm: -device vfio-pci,host=01:01.0,id=hostdev0,bus=pci.0,addr=0x9: vfio: failed to get group 30
2022-03-29T06:11:19.162913Z qemu-kvm: -device vfio-pci,host=01:01.0,id=hostdev0,bus=pci.0,addr=0x9: Device initialization failed.
2022-03-29T06:11:19.162933Z qemu-kvm: -device vfio-pci,host=01:01.0,id=hostdev0,bus=pci.0,addr=0x9: Device 'vfio-pci' could not be initialized

​解决方案​

1、在BIOS中开启IOMUU、SR-IOV、PCIE ARI、SVM、ACS功能支持

2、​开机自动加载内核vfio模块​

vim /etc/sysconfig/modules/vfio.modules

#!/bin/bash

/sbin/modinfo -F filename vfio > /dev/null 2>&1
if [ $? -eq 0 ]; then
    /sbin/modprobe vfio
    /sbin/modprobe vfio-pci
fi

3、配置vfio_iommu非安全设备访问

vim /etc/modprobe.d/vfio_iommu_type1.conf

options vfio_iommu_type1 allow_unsafe_interrupts=1

4、​开启Linux内核IOMMU及PCIE ACS功能

vim /etc/default/grub

GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet amd_iommu=on iommu=pt pcie_acs_override=downstream"

5、更新grub配置文件

#操作系统非UEFI安装
grub2-mkconfig -o /boot/grub2/grub.cfg

#操作系统UEFI安装
grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

6、重启服务器

reboot

​最终结果​