KVM : GPU Passthrough2018/05/30

	Configure GPU Passthrough for Virtual Machines. By this configuration, it's possible to use GPU on Virtual Machines and run GPU Computing by CUDA, Machine learning/Deep Learning by TensorFlow. Before configuration, Enable VT-d (Intel) or AMD IOMMU (AMD) on BIOS Setting first.
[1]	Enable IOMMU on KVM Host, and also Enable vfio-pci kernel module.

root@dlp:~#

vi /etc/default/grub

# line 12: add (if AMD CPU, add [amd_iommu=on])

GRUB_CMDLINE_LINUX="

intel_iommu=on

root@dlp:~#

grub-mkconfig -o /boot/grub/grub.cfg

# show PCI identification number and [vendor-ID:device-ID] of Graphic card

# PCI number ⇒ it matchs [03:00.*] below, vendor-ID:device-ID ⇒ it matchs [10de:***] below

root@dlp:~#

lspci -nn | grep -i nvidia

03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1)
03:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)

root@dlp:~#

vi /etc/modprobe.d/vfio.conf

# create new: for [ids=***], specify [vendor-ID:device-ID]

options vfio-pci ids=10de:1c03,10de:10f1

root@dlp:~#

echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf

root@dlp:~#

reboot

# after rebooting, verify IOMMU is enabled like follows

root@dlp:~#

dmesg | grep -E "DMAR|IOMMU"

[    0.000000] ACPI: DMAR 0x00000000BF79E0D0 000118 (v01 AMI    OEMDMAR  00000001 MSFT 00000097)
[    0.000000] DMAR: IOMMU enabled
[    0.000000] DMAR: Host address width 40
[    0.000000] DMAR: DRHD base: 0x000000fbffe000 flags: 0x1
[    0.000000] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap c90780106f0462 ecap f020fe
[    0.000000] DMAR: RMRR base: 0x000000000ec000 end: 0x000000000effff
[    0.000000] DMAR: RMRR base: 0x000000bf7ec000 end: 0x000000bf7fffff
[    0.000000] DMAR: ATSR flags: 0x0
[    0.000000] DMAR-IR: IOAPIC id 6 under DRHD base  0xfbffe000 IOMMU 0
[    0.000000] DMAR-IR: Enabled IRQ remapping in xapic mode
[    1.228883] DMAR: dmar0: Using Queued invalidation
[    1.228904] DMAR: Setting RMRR:
[    1.229153] DMAR: Setting identity map for device 0000:00:1a.0 [0xbf7ec000 - 0xbf7fffff]
[    1.229426] DMAR: Setting identity map for device 0000:00:1a.1 [0xbf7ec000 - 0xbf7fffff]
[    1.229676] DMAR: Setting identity map for device 0000:00:1a.2 [0xbf7ec000 - 0xbf7fffff]
[    1.229969] DMAR: Setting identity map for device 0000:00:1a.7 [0xbf7ec000 - 0xbf7fffff]
[    1.230270] DMAR: Setting identity map for device 0000:00:1d.0 [0xbf7ec000 - 0xbf7fffff]
[    1.230572] DMAR: Setting identity map for device 0000:00:1d.1 [0xbf7ec000 - 0xbf7fffff]
[    1.230830] DMAR: Setting identity map for device 0000:00:1d.2 [0xbf7ec000 - 0xbf7fffff]
[    1.231095] DMAR: Setting identity map for device 0000:00:1d.7 [0xbf7ec000 - 0xbf7fffff]
[    1.231119] DMAR: Setting identity map for device 0000:00:1a.0 [0xec000 - 0xeffff]
[    1.231136] DMAR: Setting identity map for device 0000:00:1a.1 [0xec000 - 0xeffff]
[    1.231152] DMAR: Setting identity map for device 0000:00:1a.2 [0xec000 - 0xeffff]
[    1.231168] DMAR: Setting identity map for device 0000:00:1a.7 [0xec000 - 0xeffff]
[    1.231185] DMAR: Setting identity map for device 0000:00:1d.0 [0xec000 - 0xeffff]
[    1.231201] DMAR: Setting identity map for device 0000:00:1d.1 [0xec000 - 0xeffff]
[    1.231217] DMAR: Setting identity map for device 0000:00:1d.2 [0xec000 - 0xeffff]
[    1.231235] DMAR: Setting identity map for device 0000:00:1d.7 [0xec000 - 0xeffff]
[    1.231253] DMAR: Prepare 0-16MiB unity mapping for LPC
[    1.231507] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    1.231645] DMAR: Intel(R) Virtualization Technology for Directed I/O

# OK if vfio_pci is enabled like follows

root@dlp:~#

dmesg | grep -i vfio

[    4.338817] VFIO - User Level meta-driver version: 0.3
[    4.348806] vfio_pci: add [10de:1c03[ffff:ffff]] class 0x000000/00000000
[    4.348958] vfio_pci: add [10de:10f1[ffff:ffff]] class 0x000000/00000000

[2]	It's OK all. Create a new VM with GPU. For [--host-device], specify GPU, and for [--machine], specify [q35], for [--features], specify [kvm_hidden=on].

root@dlp:~#

virt-install \
--name ubuntu1804 \
--ram 8192 \
--disk path=/var/kvm/images/ubuntu1804.img,size=30 \
--vcpus 4 \
--os-type linux \
--os-variant ubuntu17.10 \
--network bridge=br0 \
--graphics none \
--console pty,target_type=serial \
--location 'http://jp.archive.ubuntu.com/ubuntu/dists/bionic/main/installer-amd64/' \
--extra-args 'console=ttyS0,115200n8 serial' \
--host-device 03:00.0 \
--features kvm_hidden=on \
--machine q35

[3]	After finishing create VM, verify Graphic card is shown on VM. If no problem, try to Install Graphic Driver, GPU Computing by CUDA, Machine learning/Deep Learning by TensorFlow.

root@localhost:~#

lspci | grep -i nvidia

03:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1)

Matched Content