Nested VirtualizationState of the art and future directionsBandan DasYang Z ZhangJan Kiszka
2Outline Introduction Changes and Missing Features for AMD Changes and Missing Features for Intel Working hypervisors and performance evaluation Discussion on other ongoing work –Migration Support–VT-d emulationWrap-up and Questions
3Introduction Nested Virtualization –WindowsLinuxWindowsXenESXLinux/KVMHardware
4Introduction Uses–Operating system hypervisors (Linux/KVM, WinXP modein newer versions of Windows)–Cloud Computing – Give users the ability to run their ownhypervisors!–Security – Mcafee DeepSafe–Testing/debugging hypervisors–InteroperabilityUse Me!
5Introduction How it works (on Intel)–L0 runs L1 with VMCS01–L1 wants to run L2 and executes vmlaunchwith VMCS12–vmlaunch traps to L0–L0 merges VMCS01 with VMCS12 tocreate VMCS02 and run L2–If L2 traps, we are back in L0–L0 decides whether to handle trap itself orforward to L1–Eventually L0 resumes dwareL0
6Nested Virtualization - AMD Stable codebase– “nested” is enabled by defaultAMD-v–Advanced virtual Interrupt Controller (AVIC)–Hardware yet to arrive!More Testing–Hard to find bugs always exist!–Newer releases of common and new hypervisors–Nesting introduces I/O bottlenecksAre we spec compliant ?
7Nested Virtualization - Intel Recent Changes–Specification conformance –Intel Memory Protection Extensions –Additional error checks on emulated vmx functionsCorresponding tests in kvm-unit-testsBounds checking on memory referencesVMX support: “clear BNDCFGS” and “BNDCFGS” VMCS exit controls and“BNDCFGS” VMCS fieldNested Support: Let L1 hypervisor read and write the MPX controls(vmcs12 guest bndcfgs)Tracing improvements
8Nested Virtualization - Intel Recent Changes––L2 runningInterrupt AcknowledgementEmulationExternal InterruptExit to L0Interrupt Injection Rework Handle InterruptInspired by JailhousehypervisorAlso speeds up Windowsexecution (Complementedby TPR Shadow support)Interruptfor L1YesInject VirtualInterruptNoAcknowledge ?YesResume L1Write Vectorto VMCS12
9Nested Virtualization - Intel Improve Stability–More testing–Nested vmx is still disabled by default!–The test matrix is quite complicated with so manyconfigurations and hypervisorsAre we specification compliant ?–Also helps in identifying buggy hypervisors
10Nested Virtualization - Intel Nested VPID– –Tag address space and avoid a TLBflushWe don't advertise vpid to the L1hypervisor–L0 uses the same vpid to run L1and all its guests–KVM flushes vpid when switchingbetween L1 and L2–L0Virtual Processor IdentifierAdvertise vpid and maintain amapping for L1's vpidsTLB FlushVPID1Add Translation 1RunL1VPID1Add Translation 2RunL2
11Nested Virtualization - Intel MSR load/store–Hypervisor loads/saves a MSR list duringVMENTER/VMEXIT–Mandatory according to specificationNested APIC-v–Reduce VMEXITS–Motivation: performance gains
12AMD – Status Test Environment–Host (L0) – AMD Opteron(tm) Processor 6386 SE (16 cores), 32 GB RAM,Fedora 20–Qemu options to run L1 : -cpu host -m 20G -smp 10–Qemu options L1 uses to run L2 : -cpu qemu64 -m 8G -smp 8Guest Status (L1 hypervisor)–Linux (Fedora 20 64 bit)–Xen 4.4.3 running in Ubuntu 12.04–JailHouse–ESX
13AMD Performance Evaluation Test Environment–Host: AMD Opteron(tm) Processor 6386 SE / 32 GB RAM–L0, L1 and L2: Fedora 20–Kernel 3.17.0-rc1 (L0)–SPECJBB (2013) –Backend only, Controller/Transaction Injectors on a different hostQemu cmdline: -smp n (1, 2, 4 and 8) -m 16G -cpu qemu64Compare L1 and L2 performance numbersKernel Compilation Use “time” to measure compilation times under the same setup
14AMD Performance EvaluationKernel Compilation100959080Approximate Time 787060L1L252504240293022201210020124Number of vCPUS8
15AMD Performance Evaluation Kernel Compilation (Evaluation)–Comparable times across the vCPU range–“make” is CPU intensive
16AMD Performance EvaluationSPECJBB (Distributed with Backend in L2)max-jOPS (%) 100908070605040302010071615960L1L2124Number of vCPUS8
17AMD Performance Evaluation SPECJBB (Evaluation)–L2 nearly at 50% of L1's performance –TODO: Investigating bottlenecks in the nested setupBottlenecks I/O Bottlenecks ? The test setup creates a qcow2 image inside L1– File systems are nestedCan APIC-v help ?
18Intel - Status Test Environment–Host (L0) – IvyTown EP 16 Cores 128GB RAM–Qemu options to run L1 : -cpu host -m 20G -smp 10–Qemu options L1 uses to run L2 : -cpu qemu64 -m 8G -smp8Guest Status . not so good news
19Intel - Status Some not yet impressive matrixL2 GuestL1 GuestRHEL 6.564-bitRHEL 6.532-bitWindows 764-bitWindows 732-bitXen KVM VMware ESX VMware Player HAXM Win7 XP ModeN/AN/A Hyper-V VirtualBox
20Intel Performance EvaluationKernel Compilation60Approximate Time 485040413932313024201914100124Number of vCPUS8L1L2
21Intel Performance Evaluation Kernel Compilation (Evaluation)–CPU intensive workloads fare quite well–But . do they always ?
22Intel Performance EvaluationSPECJBBmax-jOPS (%) 1009080706050403020100L1L29%16%26%5%4Number of vCPUS8
23Intel Performance Evaluation SPECJBB (Evaluation)–What went wrong ?–Incorrect Test Setup ?–Newer machines newer processor features how isNested Virtualization affected ?–Maturity: still needs “right setup” to workI wish I was better :(
24Nested Virtualization and Migration Nested VMs implies no migration ! ;-)But in all seriousness: Challenge: Live migrate L1 with all its L2 guestsSave all nested state: vmcs12, struct nested vmx, etcbut how ?
25Nested Virtualization and Migration One option:–Force an exit from L2 to L1 (if running in L2) – feasible with all L1setups?–Save all current vmcs02 state to vmcs12–L2 specific dirtied pages need to be copied–Nested state metadata gets transferred to destination with L1's memory–If running in L2 on source, need to do the same on destinationAnother option:–Save/restore additional CPU states, just like additional registers
26Nested IOMMU Use cases–Testing–Device assignment to L2History–AMD IOMMU emulation for QEMU(Eduard-Gabriel Munteanu, 2011)–Lacking memory layer abstractions–Required many device model hooksSPARC QEMU model with own IOMMU layer
27Nested IOMMU - Today IOMMU support in QEMU memory layer, used for–POWER–Alpha–.and Intel!VT-d emulation developed as GSoC project by Le Tan–DMAR emulation, supports all PCI device models–Error reporting–Cache emulationVT-d interrupt remapping emulation–Working prototype–Lacks error reporting
28Nested IOMMU – Open Topics Support for physical devices–Full in-kernel IOMMU model? ARM SMMU model by Will Deacon,see Linux Plumber IOMMU track– IR emulation with in-kernel irqchips– Use of VFIO from userspace model?Requires extension to translate IOAPIC IRQsAMD IOMMU, reloaded?
29Wrap-Up AMD Nested Virtualization support in good shape– Regular testing required nevertheless (autotest?)Intel Nested Virtualization–Add missing mandatory features–More testing (Intel integration tests , autotest?) Once stable, address migration IOMMU emulation & nesting approaching Non-x86.?
Nested VPID – Virtual Processor Identifier Tag address space and avoid a TLB flush – We don't advertise vpid to the L 1 hypervisor – L0 uses the same vpid to run L1 and all its guests – KVM flushes vpid when switching between L1 and L2 – Advertise vpid and maintain a mapping for L1's vpids Nested Virtualization
Approximation for New Products, Estimated Elasticities (Median of 6.5) Nested CES, Elasticity 11.5 from Broda and Weinstein (2010) Nested CES, Elasticity 7 from Montgomery and Rossi (1999) Nested CES, Elasticity 4 from Dube et al (2005) Nested CES, Elasticity 2.09 from Handbury (2013) Nested
In this section, we give an overview of virtualization and describe virtio, the virtualization standard for I/O devices. In addition, we discuss the state-of-the-art for network I/O virtualization. 2.1 Overview of Virtualization and virtio The virtualization technology is generally classi ed into full-virtualization and paravirtualization.
This guide also explains the advantages of virtualization and dispels some common myths that exist regarding virtualization. 1.1. Who should read this guide? This guide is designed for anyone wishing to understand the basics of virtualization, but may be of particular interest to: Those who are new to virtualization.
TU Dresden, 2009-12-01 MOS - Virtualization Slide 6 von 58 Virtualization – a hype A lot of interest in the research community within the last years, e.g.: SOSP 03: Xen and the Art of Virtualization EuroSys 07: a whole session on virtualization Many virtualization products: VMware, QEmu, VirtualBox, KVM
Lots of features (Contd.) Domain Isolation: VCPU and Host Interrupt Affinity Spatial and Temporal Memory Isolation Device Virtualization: Pass-through device support Block device virtualization Network device virtualization Input device virtualization Display device virtualization VirtIO v0.9.5 for Para-virtualization
physical entities, and categorizes virtualization on two levels: resource (or infrastructure) virtualization and service (or application) virtualization. In resource virtualization, physical resources such as network, compute, and storage resources are segmented or pooled as logical resources. An example of resource virtualization: Sharing a load
11 Nested Blocks and Variable Scope Statements can be nested wherever an executable statement is allowed. Nested block becomes a statement. Exception section can contain nested blocks. Scope of an object is the region of the program that can refer to the object. Identifier is visible in the regions in which you can reference the unqualified identifier.
4 Virtualization For Dummies, Red Hat Special Edition Virtualization is a technology that lets one big physical server pretend to be a whole bunch of little PC machines, each with its own processor, memory, disk, and network devices. Explaining Server Virtualization When most organizations get started with virtualization, they
Desktop virtualization is driven by a combination of cost savings, increased ability to comply with myriad regulations, and an improvement in data and application security. The two fundamental forms of desktop virtualization are: Server-side application/desktop virtualization Client-side application/desktop virtualization !
The Red Hat Enterprise Linux Virtualization Guide contains information on installation, configuring, administering, and troubleshooting virtualization technologies included with Red Hat Enterprise Linux. iii . I. Requirements and Limitations for Virtualization with Red Hat Enterprise Linux 1 1. System requirements 3
Virtualization A s part of the growing trend toward commoditizing computing resources, . layer called the Xen-Blanket, the supercloud maintains the control necessary . expose state-of-the-art nested virtualization
13.Which of the following statements regarding if constructs is FALSE? A. Proper indentation of else will determine to which if it belongs in a nested construct. B. Nested if constructs can be used to test different variables. C. While there is no limit to the number of levels of nesting in a nested if construct, a larger number of levels may make the code difficult to read.
1 Nested Hierarchical Dirichlet Processes John Paisley 1, Chong Wang3, David M. Blei4 and Michael I. Jordan;2 1Department of EECS, 2Department of Statistics, UC Berkeley, Berkeley, CA 3Department of Machine Learning, Carnegie Mellon University, Pittsburgh, PA 4Department of Computer Science, Princeton University, Princeton, NJ Abstract We develop a nested
The ANOVA table of two factor nested design showing their respective sum of square, degree of freedom, . 17.25 3 5.75 1.60 F 0.05 (3, 6) 4.76 0.100 . of factor A to another in such a way that the degree of freedom for MSA and MS(B/A) are almost equal. .
locking concurrency control unsuitable for nested transaction. 3. Optimistic concurrency control can be dangerous . If a nested transaction is aborted, the number of dependent transactions that need be aborted will be larger, which in turn will cause a even large r number of transactions to abort. That is, the
A stretched grid model has an important advantage over the more common nested grid approach in that the lateral boundary conditions required to define the interface be-tween the window region and the external medium in the nested grid model are unnecessary with the stretched grid formulation. On the other hand, stretched grid models D22307 WANG .
GUIDE TO SECURITY FOR FULL VIRTUALIZATION TECHNOLOGIES ES-1 Executive Summary Virtualization is the simulation of the software and/or hardware upon which other software runs. This simulated environment is called a virtual machine (VM).There are many forms of virtualization,
Title: Virtualization - IBM Insights in Sizing Servers for Virtualization Abstract: Come hear how IBM can help you build a better virtualized infrastructure. Learn how to use the new IBM Virtualization Sizing Guide, to help you understand the method and tools that IBM has used to
Virtualization For Dummies, 2nd Sun and AMD Special Edition explains how virtualization works and how it can benefit your organization. The book covers the kinds of issues virtualization can ad
Literary Studies. London: Longman, 1993. INTRODUCTION While most of you have already had experience of essay writing, it is important to realise that essay writing at University level may be different from the practices you have so far encountered. The aim of this tutorial is to discuss what is required of an English Literature essay at University level, including: 1. information on the .