Nested Virtualization

2y ago
86 Views
2 Downloads
785.35 KB
29 Pages
Last View : 1m ago
Last Download : 5m ago
Upload by : Troy Oden
Transcription

Nested VirtualizationState of the art and future directionsBandan DasYang Z ZhangJan Kiszka

2Outline Introduction Changes and Missing Features for AMD Changes and Missing Features for Intel Working hypervisors and performance evaluation Discussion on other ongoing work –Migration Support–VT-d emulationWrap-up and Questions

3Introduction Nested Virtualization –WindowsLinuxWindowsXenESXLinux/KVMHardware

4Introduction Uses–Operating system hypervisors (Linux/KVM, WinXP modein newer versions of Windows)–Cloud Computing – Give users the ability to run their ownhypervisors!–Security – Mcafee DeepSafe–Testing/debugging hypervisors–InteroperabilityUse Me!

5Introduction How it works (on Intel)–L0 runs L1 with VMCS01–L1 wants to run L2 and executes vmlaunchwith VMCS12–vmlaunch traps to L0–L0 merges VMCS01 with VMCS12 tocreate VMCS02 and run L2–If L2 traps, we are back in L0–L0 decides whether to handle trap itself orforward to L1–Eventually L0 resumes dwareL0

6Nested Virtualization - AMD Stable codebase– “nested” is enabled by defaultAMD-v–Advanced virtual Interrupt Controller (AVIC)–Hardware yet to arrive!More Testing–Hard to find bugs always exist!–Newer releases of common and new hypervisors–Nesting introduces I/O bottlenecksAre we spec compliant ?

7Nested Virtualization - Intel Recent Changes–Specification conformance –Intel Memory Protection Extensions –Additional error checks on emulated vmx functionsCorresponding tests in kvm-unit-testsBounds checking on memory referencesVMX support: “clear BNDCFGS” and “BNDCFGS” VMCS exit controls and“BNDCFGS” VMCS fieldNested Support: Let L1 hypervisor read and write the MPX controls(vmcs12 guest bndcfgs)Tracing improvements

8Nested Virtualization - Intel Recent Changes––L2 runningInterrupt AcknowledgementEmulationExternal InterruptExit to L0Interrupt Injection Rework Handle InterruptInspired by JailhousehypervisorAlso speeds up Windowsexecution (Complementedby TPR Shadow support)Interruptfor L1YesInject VirtualInterruptNoAcknowledge ?YesResume L1Write Vectorto VMCS12

9Nested Virtualization - Intel Improve Stability–More testing–Nested vmx is still disabled by default!–The test matrix is quite complicated with so manyconfigurations and hypervisorsAre we specification compliant ?–Also helps in identifying buggy hypervisors

10Nested Virtualization - Intel Nested VPID– –Tag address space and avoid a TLBflushWe don't advertise vpid to the L1hypervisor–L0 uses the same vpid to run L1and all its guests–KVM flushes vpid when switchingbetween L1 and L2–L0Virtual Processor IdentifierAdvertise vpid and maintain amapping for L1's vpidsTLB FlushVPID1Add Translation 1RunL1VPID1Add Translation 2RunL2

11Nested Virtualization - Intel MSR load/store–Hypervisor loads/saves a MSR list duringVMENTER/VMEXIT–Mandatory according to specificationNested APIC-v–Reduce VMEXITS–Motivation: performance gains

12AMD – Status Test Environment–Host (L0) – AMD Opteron(tm) Processor 6386 SE (16 cores), 32 GB RAM,Fedora 20–Qemu options to run L1 : -cpu host -m 20G -smp 10–Qemu options L1 uses to run L2 : -cpu qemu64 -m 8G -smp 8Guest Status (L1 hypervisor)–Linux (Fedora 20 64 bit)–Xen 4.4.3 running in Ubuntu 12.04–JailHouse–ESX

13AMD Performance Evaluation Test Environment–Host: AMD Opteron(tm) Processor 6386 SE / 32 GB RAM–L0, L1 and L2: Fedora 20–Kernel 3.17.0-rc1 (L0)–SPECJBB (2013) –Backend only, Controller/Transaction Injectors on a different hostQemu cmdline: -smp n (1, 2, 4 and 8) -m 16G -cpu qemu64Compare L1 and L2 performance numbersKernel Compilation Use “time” to measure compilation times under the same setup

14AMD Performance EvaluationKernel Compilation100959080Approximate Time 787060L1L252504240293022201210020124Number of vCPUS8

15AMD Performance Evaluation Kernel Compilation (Evaluation)–Comparable times across the vCPU range–“make” is CPU intensive

16AMD Performance EvaluationSPECJBB (Distributed with Backend in L2)max-jOPS (%) 100908070605040302010071615960L1L2124Number of vCPUS8

17AMD Performance Evaluation SPECJBB (Evaluation)–L2 nearly at 50% of L1's performance –TODO: Investigating bottlenecks in the nested setupBottlenecks I/O Bottlenecks ? The test setup creates a qcow2 image inside L1– File systems are nestedCan APIC-v help ?

18Intel - Status Test Environment–Host (L0) – IvyTown EP 16 Cores 128GB RAM–Qemu options to run L1 : -cpu host -m 20G -smp 10–Qemu options L1 uses to run L2 : -cpu qemu64 -m 8G -smp8Guest Status . not so good news

19Intel - Status Some not yet impressive matrixL2 GuestL1 GuestRHEL 6.564-bitRHEL 6.532-bitWindows 764-bitWindows 732-bitXen KVM VMware ESX VMware Player HAXM Win7 XP ModeN/AN/A Hyper-V VirtualBox

20Intel Performance EvaluationKernel Compilation60Approximate Time 485040413932313024201914100124Number of vCPUS8L1L2

21Intel Performance Evaluation Kernel Compilation (Evaluation)–CPU intensive workloads fare quite well–But . do they always ?

22Intel Performance EvaluationSPECJBBmax-jOPS (%) 1009080706050403020100L1L29%16%26%5%4Number of vCPUS8

23Intel Performance Evaluation SPECJBB (Evaluation)–What went wrong ?–Incorrect Test Setup ?–Newer machines newer processor features how isNested Virtualization affected ?–Maturity: still needs “right setup” to workI wish I was better :(

24Nested Virtualization and Migration Nested VMs implies no migration ! ;-)But in all seriousness: Challenge: Live migrate L1 with all its L2 guestsSave all nested state: vmcs12, struct nested vmx, etcbut how ?

25Nested Virtualization and Migration One option:–Force an exit from L2 to L1 (if running in L2) – feasible with all L1setups?–Save all current vmcs02 state to vmcs12–L2 specific dirtied pages need to be copied–Nested state metadata gets transferred to destination with L1's memory–If running in L2 on source, need to do the same on destinationAnother option:–Save/restore additional CPU states, just like additional registers

26Nested IOMMU Use cases–Testing–Device assignment to L2History–AMD IOMMU emulation for QEMU(Eduard-Gabriel Munteanu, 2011)–Lacking memory layer abstractions–Required many device model hooksSPARC QEMU model with own IOMMU layer

27Nested IOMMU - Today IOMMU support in QEMU memory layer, used for–POWER–Alpha–.and Intel!VT-d emulation developed as GSoC project by Le Tan–DMAR emulation, supports all PCI device models–Error reporting–Cache emulationVT-d interrupt remapping emulation–Working prototype–Lacks error reporting

28Nested IOMMU – Open Topics Support for physical devices–Full in-kernel IOMMU model? ARM SMMU model by Will Deacon,see Linux Plumber IOMMU track– IR emulation with in-kernel irqchips– Use of VFIO from userspace model?Requires extension to translate IOAPIC IRQsAMD IOMMU, reloaded?

29Wrap-Up AMD Nested Virtualization support in good shape– Regular testing required nevertheless (autotest?)Intel Nested Virtualization–Add missing mandatory features–More testing (Intel integration tests , autotest?) Once stable, address migration IOMMU emulation & nesting approaching Non-x86.?

Nested VPID – Virtual Processor Identifier Tag address space and avoid a TLB flush – We don't advertise vpid to the L 1 hypervisor – L0 uses the same vpid to run L1 and all its guests – KVM flushes vpid when switching between L1 and L2 – Advertise vpid and maintain a mapping for L1's vpids Nested Virtualization

Related Documents:

The basic structure of a SQL query 1s a query block, which consists pnnclpally of a SELECT clause, a FROM clause, and zero or more WHERE clauses The first query block m a nested . Kim's Algorithms for Processing Nested Queries Km observed that for type-N and type-J nested queries, the nested iteraaon method for processmg nested quenes is .

In this section, we give an overview of virtualization and describe virtio, the virtualization standard for I/O devices. In addition, we discuss the state-of-the-art for network I/O virtualization. 2.1 Overview of Virtualization and virtio The virtualization technology is generally classi ed into full-virtualization and paravirtualization.

Approximation for New Products, Estimated Elasticities (Median of 6.5) Nested CES, Elasticity 11.5 from Broda and Weinstein (2010) Nested CES, Elasticity 7 from Montgomery and Rossi (1999) Nested CES, Elasticity 4 from Dube et al (2005) Nested CES, Elasticity 2.09 from Handbury (2013) Nested

This guide also explains the advantages of virtualization and dispels some common myths that exist regarding virtualization. 1.1. Who should read this guide? This guide is designed for anyone wishing to understand the basics of virtualization, but may be of particular interest to: Those who are new to virtualization.

TU Dresden, 2009-12-01 MOS - Virtualization Slide 6 von 58 Virtualization – a hype A lot of interest in the research community within the last years, e.g.: SOSP 03: Xen and the Art of Virtualization EuroSys 07: a whole session on virtualization Many virtualization products: VMware, QEmu, VirtualBox, KVM

Lots of features (Contd.) Domain Isolation: VCPU and Host Interrupt Affinity Spatial and Temporal Memory Isolation Device Virtualization: Pass-through device support Block device virtualization Network device virtualization Input device virtualization Display device virtualization VirtIO v0.9.5 for Para-virtualization

physical entities, and categorizes virtualization on two levels: resource (or infrastructure) virtualization and service (or application) virtualization. In resource virtualization, physical resources such as network, compute, and storage resources are segmented or pooled as logical resources. An example of resource virtualization: Sharing a load

Final 2012 policy CMS will implement the third year of the 4-year transition to new practice expense RVUs developed using the PPIS data. Available in the appendix of this summary, Table 84 of the .