KVM : GPU Passthrough2018/01/31 |
Configure GPU Passthrough for Virtual Machines.
By this configuration, it's possible to use GPU on Virtual Machines and run
GPU Computing by CUDA,
Machine learning/Deep Learning by TensorFlow.
Before configuration, Enable VT-d (Intel) or AMD IOMMU (AMD) on BIOS Setting first.
|
|
[1] | Enable IOMMU on KVM Host. |
[root@dlp ~]#
vi /etc/default/grub # line 6: add (if AMD CPU, add [amd_iommu=on]) GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet intel_iommu=on "GRUB_DISABLE_RECOVERY="true" grub2-mkconfig -o /boot/grub2/grub.cfg
[root@dlp ~]#
reboot
# after rebooting, verify IOMMU is enabled like follows [root@dlp ~]# dmesg | grep -E "DMAR|IOMMU" [ 0.000000] ACPI: DMAR 00000000bf79e0d0 00118 (v01 AMI OEMDMAR 00000001 MSFT 00000097) [ 0.000000] DMAR: IOMMU enabled [ 0.069846] DMAR: Host address width 40 [ 0.069848] DMAR: DRHD base: 0x000000fbffe000 flags: 0x1 [ 0.069856] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap c90780106f0462 ecap f020fe [ 0.069858] DMAR: RMRR base: 0x000000000ec000 end: 0x000000000effff [ 0.069859] DMAR: RMRR base: 0x000000bf7ec000 end: 0x000000bf7fffff [ 0.069861] DMAR: ATSR flags: 0x0 [ 0.069863] DMAR-IR: IOAPIC id 6 under DRHD base 0xfbffe000 IOMMU 0 [ 0.070128] DMAR-IR: Enabled IRQ remapping in xapic mode [ 0.960840] DMAR: dmar0: Using Queued invalidation [ 0.960860] DMAR: Setting RMRR: [ 0.960890] DMAR: Setting identity map for device 0000:00:1a.0 [0xbf7ec000 - 0xbf7fffff] [ 0.960925] DMAR: Setting identity map for device 0000:00:1a.1 [0xbf7ec000 - 0xbf7fffff] [ 0.960958] DMAR: Setting identity map for device 0000:00:1a.2 [0xbf7ec000 - 0xbf7fffff] [ 0.960994] DMAR: Setting identity map for device 0000:00:1a.7 [0xbf7ec000 - 0xbf7fffff] [ 0.961031] DMAR: Setting identity map for device 0000:00:1d.0 [0xbf7ec000 - 0xbf7fffff] [ 0.961064] DMAR: Setting identity map for device 0000:00:1d.1 [0xbf7ec000 - 0xbf7fffff] [ 0.961100] DMAR: Setting identity map for device 0000:00:1d.2 [0xbf7ec000 - 0xbf7fffff] [ 0.961132] DMAR: Setting identity map for device 0000:00:1d.7 [0xbf7ec000 - 0xbf7fffff] [ 0.961153] DMAR: Setting identity map for device 0000:00:1a.0 [0xec000 - 0xeffff] [ 0.961167] DMAR: Setting identity map for device 0000:00:1a.1 [0xec000 - 0xeffff] [ 0.961182] DMAR: Setting identity map for device 0000:00:1a.2 [0xec000 - 0xeffff] [ 0.961196] DMAR: Setting identity map for device 0000:00:1a.7 [0xec000 - 0xeffff] [ 0.961210] DMAR: Setting identity map for device 0000:00:1d.0 [0xec000 - 0xeffff] [ 0.961224] DMAR: Setting identity map for device 0000:00:1d.1 [0xec000 - 0xeffff] [ 0.961238] DMAR: Setting identity map for device 0000:00:1d.2 [0xec000 - 0xeffff] [ 0.961255] DMAR: Setting identity map for device 0000:00:1d.7 [0xec000 - 0xeffff] [ 0.961270] DMAR: Prepare 0-16MiB unity mapping for LPC [ 0.961287] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.961424] DMAR: Intel(R) Virtualization Technology for Directed I/O |
[2] | Enable vfio-pci kernel module. It's OK to keep common kernel provided by RHEL/CentOS official. |
# OK with common kernel [root@dlp ~]# uname -a Linux dlp.srv.world 3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux # show PCI identification number and [vendor-ID:device-ID] of Graphic card # PCI number ⇒ it matchs [03:00.*] below, vendor-ID:device-ID ⇒ it matchs [10de:***] below [root@dlp ~]# lspci -nn | grep -i nvidia 03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1) 03:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
[root@dlp ~]#
vi /etc/modprobe.d/vfio.conf # create new: for [ids=***], specify [vendor-ID:device-ID] options vfio-pci ids=10de:1c03,10de:10f1 echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf
[root@dlp ~]#
reboot
# it's OK if it's enabled like follows [root@dlp ~]# dmesg | grep -i vfio [ 4.338817] VFIO - User Level meta-driver version: 0.3 [ 4.348806] vfio_pci: add [10de:1c03[ffff:ffff]] class 0x000000/00000000 [ 4.348958] vfio_pci: add [10de:10f1[ffff:ffff]] class 0x000000/00000000 |
[3] | Update QEMU. |
[root@dlp ~]#
/usr/libexec/qemu-kvm -version QEMU emulator version 1.5.3 (qemu-kvm-1.5.3-141.el7_4.6), Copyright (c) 2003-2008 Fabrice Bellard
[root@dlp ~]#
yum -y install centos-release-qemu-ev # disable usually [root@dlp ~]# sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-QEMU-EV.repo
# for this installing, [qemu-kvm] package is replaced to [qemu-kvm-ev] package [root@dlp ~]# yum --enablerepo=centos-qemu-ev -y install qemu-kvm-ev
[root@dlp ~]#
systemctl restart libvirtd
[root@dlp ~]#
/usr/libexec/qemu-kvm -version QEMU emulator version 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1) Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers # verify q35 is included like follows [root@dlp ~]# /usr/libexec/qemu-kvm -machine help Supported machines are: pc RHEL 7.4.0 PC (i440FX + PIIX, 1996) (alias of pc-i440fx-rhel7.4.0) pc-i440fx-rhel7.4.0 RHEL 7.4.0 PC (i440FX + PIIX, 1996) (default) pc-i440fx-rhel7.3.0 RHEL 7.3.0 PC (i440FX + PIIX, 1996) pc-i440fx-rhel7.2.0 RHEL 7.2.0 PC (i440FX + PIIX, 1996) pc-i440fx-rhel7.1.0 RHEL 7.1.0 PC (i440FX + PIIX, 1996) pc-i440fx-rhel7.0.0 RHEL 7.0.0 PC (i440FX + PIIX, 1996) rhel6.6.0 RHEL 6.6.0 PC rhel6.5.0 RHEL 6.5.0 PC rhel6.4.0 RHEL 6.4.0 PC rhel6.3.0 RHEL 6.3.0 PC rhel6.2.0 RHEL 6.2.0 PC rhel6.1.0 RHEL 6.1.0 PC rhel6.0.0 RHEL 6.0.0 PC q35 RHEL-7.4.0 PC (Q35 + ICH9, 2009) (alias of pc-q35-rhel7.4.0) pc-q35-rhel7.4.0 RHEL-7.4.0 PC (Q35 + ICH9, 2009) pc-q35-rhel7.3.0 RHEL-7.3.0 PC (Q35 + ICH9, 2009) none empty machine |
[4] | It's OK all. Create a new VM with GPU. For [--host-device], specify GPU, and for [--machine], specify [q35], for [--features], specify [kvm_hidden=on]. |
[root@dlp ~]# virt-install \
--name centos7 \ --ram 8192 \ --disk path=/var/kvm/images/centos7.img,size=30 \ --vcpus 4 \ --os-type linux \ --os-variant rhel7 \ --network bridge=br0 \ --graphics none \ --console pty,target_type=serial \ --location 'http://ftp.iij.ad.jp/pub/linux/centos/7/os/x86_64/' \ --extra-args 'console=ttyS0,115200n8 serial' \ --host-device 03:00.0 \ --features kvm_hidden=on \ --machine q35 |
[5] | After finishing create VM, verify Graphic card is shown on VM. If no problem, try to Install Graphic Driver, GPU Computing by CUDA, Machine learning/Deep Learning by TensorFlow. |
[root@localhost ~]# lspci | grep -i nvidia 04:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1) |
Sponsored Link |
|