Virtualization
Faculty of Computer Science Institute for System Architecture, Operating Systems GroupVirtualizationHenning SchildDresden, 2009-12-01
So Far . Basics IntroductionThreads & synchronizationMemory Real-time Resource Management Device DriversTU Dresden, 2009-12-01MOS - VirtualizationSlide 2 von 58
Today: Virtualization Introduction Motivation & classification, flavors L4Linux: Para-virtualization on top of L4 ArchitectureAddress space layoutScenarios NOVA – a μ-hypervisor KVM on FiascoOCTU Dresden, 2009-12-01MOS - VirtualizationSlide 3 von 58
One possible definition . Introduction of layers of abstraction betweenphysical ressources and users/applications. partitioning of ressourcesaggregation of ressourcescombinationsTU Dresden, 2009-12-01MOS - VirtualizationSlide 4 von 58
Virtualization flavours Multitasking OS as layer of abstractionmachine partitioning, virtual memory and timeslicesapplication level Unix chroot FreeBSD Jails, Solaris Zones, Linux Vserver Wine multiple OSs on one machine VMWare, QEMU, VirtualBOX UML, Xen, L4LinuxTU Dresden, 2009-12-01MOS - VirtualizationSlide 5 von 58
Virtualization – a hype A lot of interest in the research communitywithin the last years, e.g.: SOSP 03: Xen and the Art of VirtualizationEuroSys 07: a whole session on virtualizationMany virtualization products: VMware, QEmu, VirtualBox, KVM, Hyper-V x86 Hardware support further increasing demand: VMware: from 240 to 6300 employees withinthe last few yearsTU Dresden, 2009-12-01MOS - VirtualizationSlide 6 von 58
Virtualization - a new idea? Originates in IBM's CP/CMS series used onSystem/3xx mainframes (starting 1964) Control Program - VMM Cambridge Monitor System Guest OS Memory protection SIE instruction (VM mode) CP encodes much of the guest privileged statein a hardware-defined formatIBM's first virtual memory systemTU Dresden, 2009-12-01MOS - VirtualizationSlide 7 von 58
MotivationTU Dresden, 2009-12-01MOS - VirtualizationSlide 8 von 58
Virtualization - Motivation optimize utilization Isolation security reasonsincompatibilityreusing legacy software server consolidationi.e. Windows on Linuxdevelopment virtual test machinesTU Dresden, 2009-12-01MOS - VirtualizationSlide 9 von 58
Virtualization - tyManageabilityVirtual ApplianceTU Dresden, 2009-12-01MOS - VirtualizationConsolidationSlide 10 von 58
Formal Requirements Equivalence Isolation guest behaviour should match real machinehost controls ressource accessguests are isolated from host and from eachotherEfficiency guest code should be executed nativelysee paper reading 2010-01-12: “Formalrequirements for virtualizable third generationarchitectures”TU Dresden, 2009-12-01MOS - VirtualizationSlide 11 von 58
Classification help Virtualization - an overloaded term Some classification criteria: Objective target: hardware, OS API or ABI ?Emulation vs. virtualization: do we have tointerpret some or all instructions ? Binary vs. byte code interpretation (e.g.: JVM)Can we modify the target software ?(e.g. using para-virtualization techniques)TU Dresden, 2009-12-01MOS - VirtualizationSlide 12 von 58
Reimplementation of the OS interface used to integrate a bunch of existing softwareto other respectively newly created OSeswhen copying the API of an OS, targetsoftware needs to be re-linkedin contrast to that, ABI emulation can rununmodified binaries e.g.: WineDisadvantages of both approaches: huge effortshooting at a moving targetTU Dresden, 2009-12-01MOS - VirtualizationSlide 13 von 58
Virtualize the hardware instead of emulating the OS API or ABI, takethe underlying platform Emulation interprete/translate guest codeVirtualization common to many OSsnative execution of guest codewith or without HW-SupportParavirtualization modification of the guestTU Dresden, 2009-12-01MOS - VirtualizationSlide 14 von 58
Emulation binary translation/interpretation of guest code no native execution contradicts with efficiency requirement applicable to a lot of architectures often used for peripheral devices Example: QEMU, Bochs QEMU emulates x86, ARM, SPARC, PowerPC .TU Dresden, 2009-12-01MOS - VirtualizationSlide 15 von 58
Platform virtualization in software guest OS runs natively in less privileged modeprivileged instructions fail and are handled bythe VMM (trap-and-emulate)VMM derives and manages shadow structuresfrom guest's primary structures, e.g.: shadowpage tables JIT binary translation Examples: VMware, KQEMU, VirtualBoxTU Dresden, 2009-12-01MOS - VirtualizationSlide 16 von 58
X86 VirtualizationTU Dresden, 2009-12-01MOS - VirtualizationSlide 17 von 58
Problems with x86 virtualization Ring-alias problem Address space compression part of the guest OS's address space used bythe VMM (e.g. IDT, GDT)some instructions do not trap, e.g.: guest OS runs in privilege level 0popf: pop stack into EFLAGS register,causes interrupt handling problems(IF not updated in user-mode)faulting implies performance loss kernel entry/exit - doubled context switchTU Dresden, 2009-12-01MOS - VirtualizationSlide 18 von 58
Hardware enabled virtualization Example Intel-VT root and non-root mode, VM entry and exit Virtual Machine Control Structure in physicalmemory holds information of guest and hoststate and some additional control informationVMCS is used to investigate VM exitconditions, e.g.: whether a guest traps whenmasking or unmasking interruptsAMD SVM is similarTU Dresden, 2009-12-01MOS - VirtualizationSlide 19 von 58
Hardware enabled virtualization problematic instructions trap reduced software complexity Examples: KVM, VirtualBox, Xen, Hyper-V,Windows 7 XP Mode, Parallels .TU Dresden, 2009-12-01MOS - VirtualizationSlide 20 von 58
MMU VirtualizationTU Dresden, 2009-12-01MOS - VirtualizationSlide 21 von 58
Shadow page tables Memory tracing of the page tables decode and emulate guest's pagefaultsguest virtual memoryguest page tableguest physical memoryhost virtual memoryshadow page tablehost page tablehost physical memoryTU Dresden, 2009-12-01MOS - VirtualizationSlide 22 von 58
Shadow page tables1) pagefault in guest (GVA)2) caught by hypervisor/VMM3) parse guest page tables (GVA GPA)GVAGPAHVAHPAguest virtual addressguest physical addresshost virtual addresshost physical address4) maybe inject pagefault into guest and parse again5) translate guest pt entry to shadow pt entry (GPA HVA HPA)6) create mapping in shadow pt and resume costly, recent x86 processors come with hardware supportguest virtual memoryguest page tableguest physical memoryhost virtual memoryshadow page tablehost page tablehost physical memoryTU Dresden, 2009-12-01MOS - VirtualizationSlide 23 von 58
MMU Virtualization with HW support hardware can parse two page table levelsVM page table constructed by VMM maps HPAto GPA guest manages its own GPA to GVA tables no shadow paging in software required pagefaults can be resolved without modeswitchingAMD: nested paging, Intel: EPT significant performance increase for VMsTU Dresden, 2009-12-01MOS - VirtualizationSlide 24 von 58
ParavirtualizationTU Dresden, 2009-12-01MOS - VirtualizationSlide 25 von 58
Paravirtualization modify guest OS to integrate it in the runtimeenvironment of another OSadvantages: no hardware support requiredcooperation from guests possibledisadvantages: source code requiredhigh development cost L4Linux, Xen, User Mode Linux, coLinux Afterburner (Karlsruhe): modify binary code paravirtualized drivers: VMware, KVM (virtio)TU Dresden, 2009-12-01MOS - VirtualizationSlide 26 von 58
XENTU Dresden, 2009-12-01MOS - VirtualizationSlide 27 von 58
Examples fromTUDOS groupTU Dresden, 2009-12-01MOS - VirtualizationSlide 28 von 58
L4LinuxTU Dresden, 2009-12-01MOS - VirtualizationSlide 29 von 58
L4Linux: history presented at SOSP '97 (L4)Linux has evolved over the years based on x86 Linux 2.0 on top of first L4 kernel2.2 supported MIPS and x862.4 first version to run on L4Env2.6 uses 'paravirtualization' L4 kernel featuresrecently latest L4Linux release 2.6.31x86 and ARM supportSMPTU Dresden, 2009-12-01MOS - VirtualizationSlide 30 von 58
Linux pplicationkernelArchDepend.System-Call InterfaceLinuxKernelFile SystemsVFSFile System hedulingIPCDevice DriversArchDepend.MemoryManagementPage allocationAddress spacesSwappingHardware AccessHardwareCPU, Memory, PCI, DevicesTU Dresden, 2009-12-01MOS - VirtualizationSlide 31 von 58
Linux Architecture Architecture dependent part Small, for x86 about 2% of the kerneluser kernel ApplicationApplicationSystem call interface:ApplicationKernel entryArch Signal deliveryDepend. System-Call Interface Copy from/to user space File nelProcessesSchedulingIPCFile System Impl.ProtocolsCPU stateand featuresArch Device DriversInd.MMU InterruptArch HardwareMemory mapped I/O, I/OportsAccessDepend. ApplicationMemoryManagementPage allocationAddress spacesSwappingArchitecture dependent part implementsHardwaregeneric interfaceused by independent partCPU, Memory, PCI, Devices, TU Dresden, 2009-12-01MOS - VirtualizationSlide 32 von 58
Linux pplicationkernelArchDepend.System-Call InterfaceLinuxKernelFile SystemsVFSFile System hedulingIPCDevice DriversArchDepend.MemoryManagementPage allocationAddress spacesSwappingHardware AccessHardwareCPU, Memory, PCI, DevicesTU Dresden, 2009-12-01MOS - VirtualizationSlide 33 von 58
L4Linux ArchitectureL4 TaskApplicationL4 TaskL4 TaskApplicationApplicationL4 TaskApplicationL4 TaskArchDepend.System-Call InterfaceLinuxKernelFile SystemsVFSFile System Impl.ArchInd.NetworkingSocketsProtocolsDevice MemoryManagementPage allocationAddress spacesSwappingHardware Accesssigma0L4IOConsolemoeFiascoOCHardwareTU Dresden, 2009-12-01MOS - VirtualizationSlide 34 von 58
L4Linux Architecture Linux kernel and Linux user processes runeach within a single L4 taskL4/L4RE specific part is implemented asseparate architecture: arch/l4include/asm-l4L4/L4RE architecture dependent part itselfdivides into x86 and ARM specific partmost code is reused from x86 resp. ARMspecific partTU Dresden, 2009-12-01MOS - VirtualizationSlide 35 von 58
Linux address space layout 0xFFFFFFFF0x0 – TASK SIZE vmalloc, kmap, KernelAddressSpacePAGE OFFSET0xC0000000TASK SIZEUserAddressSpace Phys. MemoryKernel Image TASK SIZE – 0xF. Application,Libraries, 0x00000000TU Dresden, 2009-12-01 user partchanges on everycontext switchkernel partconstant in alladdress spacesPhysical memorymapped beginning atPAGE OFFSETMOS - VirtualizationSlide 36 von 58
L4Linux address space layout0xFFFFFFFF0xFFFFFFFFvmalloc, kmap, KernelAddressSpacePAGE OFFSET0xC0000000TASK SIZEUserAddressSpace0xC0000000Phys. MemoryL4Linux UserProcessApplication,Libraries, Kernel Image0x000000000xFFFFFFFFApplication,Libraries, Linux Servervmalloc, kmap, 0x00000000Guest-phys. MemoryPAGE OFFSE0x00000000TTU Dresden, 2009-12-01MOS - VirtualizationKernel ImageSlide 37 von 58
L4Linux: problems to be solved L4Linux server has to: L4Linux user processes have to: have some basic resources (memory, I/O)manage page tables of its user processeshandle exceptions from user processesschedule its tasks'enter' the L4Linux kernel (now in a differentaddress space)Kernel needs information from user processesformerly accessible in the same addressspace, e.g.: syscall argumentsTU Dresden, 2009-12-01MOS - VirtualizationSlide 38 von 58
Linux address space management Architecture-independent part: general page table managementimplements allocator strategiespage replacement strategiesassumes 4-level page table byarchitecture-dependent partArchitecture-dependent part set, remove and test entriesTLB handlingLinux for x86 uses 2 level pagetablesTU Dresden, 2009-12-01MOS - VirtualizationApplicationLinux KernelMemoryManagement– Page allocation– Address spaces– Swappingthread infoArchitectureDependentPart (i386)HardwareSlide 39 von 58
L4Linux address space management L4Linux user processes areactually L4 tasksL4Linux server is the pagerHardware page tables aremanaged by L4 kernelL4Linux page tables are mirrored ApplicationLinux KernelMemoryManagement– Page allocation– Address spaces– Swappingthread infoArchitectureDependentPart (i386)L4Linux uses map/unmapoperationsFiascoadding page table entries is done Kernellazy (pagefault occurs)HardwareTU Dresden, 2009-12-01MOS - VirtualizationSlide 40 von 58
General exception handling if a L4 task raises an exception kernel sendsexception IPC to handler (feature in FiascoOCand L4.X2)Exception IPC contains CPU state of the clientException handler can reply with a new state,for instance another instruction pointerException IPC can be used to recognize Linuxsystem calls: INT 0x80 will trigger an exceptionL4Linux server acts as exception handler for itsuser processesTU Dresden, 2009-12-01MOS - VirtualizationSlide 41 von 58
L4Linux kernel entry System call costs: 2x kernel entry/exit (exception and reply)2x address space switchL4Linux UserProcessINT 0x80L4Linux Server3arch. dependent214TU Dresden, 2009-12-01arch. independentFiasco microkernelMOS - VirtualizationSlide 42 von 58
Interrupt handling Interrupt messages are receivedin separate threadsInterrupt threads run on ahigher priority than other Linuxthreads (Linux semantic)Interrupt thread wake up idlethread or force the running userprocess to enter the linux serverPlain Linux disables interruptsfor syncronization Use a lock instead of CLI/STITU Dresden, 2009-12-01MOS - VirtualizationL4Linux ServerDevice Driverr equest i r q( i r q no,handl er , )MainThreadInterruptThreadsL4IOFiasco KernelHardwareSlide 43 von 58
not covered in detail here . Linux kernel needs to access address space ofuser processes (e.g. syscall arguments) Security problems with DMA walk page tables of user processmove device drivers out of L4LinuxI/O MMUL4Linux scheduling only one L4Linux process is active at a timeother processes are waiting in IPC (exceptionor pagefault)TU Dresden, 2009-12-01MOS - VirtualizationSlide 44 von 58
Hybrid applications Linux applications that are 'L4 aware' Needs to be detected by Linux server Linux server puts them in UNINTERRUPTIBLEstate in its own data structuresWill not disturb ongoing IPC in hybrid taskL4Linux user processes run as Aliens Special alien flag used when creating a taskAliens trap when calling L4 systemException handler monitors system callFiasco-only featureTU Dresden, 2009-12-01MOS - VirtualizationSlide 45 von 58
L4Linux Use - casesTU Dresden, 2009-12-01MOS - VirtualizationSlide 46 von 58
Real-time video player L4Linux user processes might use L4 rLoaderRoottaskmoeDOpEFiascoOC kernelTU Dresden, 2009-12-01MOS - VirtualizationSlide 47 von 58
Multiple L4Linux instances Using multipleinstances concurrently,e.g. for each securitydomainApp.L4Linux serverDevices need to bemultiplexed (seeresource managementlesson: ORe, nitpicker,windhoek, )Communicationthrough network,special IPC monitors .TU Dresden, 2009-12-01App.App.App.L4Linux serverVirtualization infrastructureLoaderMOS - VirtualizationRoottaskconsolemoeFiascoOC kernelSlide 48 von 58
Use L4Linux as a toolbox L4Linux instances can provide access tovarious complex software stacks, e.g.: Network nuxL4 AppLoaderRoottaskmoeFiasco kernelTU Dresden, 2009-12-01MOS - VirtualizationSlide 49 von 58
FaithfulVirtualizationTU Dresden, 2009-12-01MOS - VirtualizationSlide 50 von 58
NOVA – μ hypervisor approach NOVA OS Virtualization Architecture Separate hypervisor and VMM(s)Guest OSGuest OSGuest OSnon-rootrootuserServerkernelTU Dresden, 2009-12-01VMMVMMVMMhypervisorMOS - VirtualizationSlide 51 von 58
NOVA Hypervisor manages protection domains: address spaces and virtual machinesVirtual machine has associated virtualizationhandler - the VMM (codename: Vancouver)VMMs handle virtualization faults andimplement virtual devicessplit functionality of hypervisor and VMMreduced complexity of hypervisor whichruns security-sensitive applications beside theVMsTU Dresden, 2009-12-01MOS - VirtualizationSlide 52 von 58
FiascoOC and KVM-L4 FiascoOC provides AMD SVM support KVM can be reused with little modificationqemu-kvmqemu-kvmL4Linux serverGuest OSGuest OSKVM-L4guesthostuserLoaderkernelTU Dresden, 2009-12-01RoottaskDMPhysNamesFiasco kernelMOS - VirtualizationSlide 53 von 58
FiascoOC and KVM-L4 FiascoOC supports AMD SVMmemory is mapped to VMs using map/unmapmechanism invoke VM capability to enter guest mode existing VMM can be reused KVM with little modificationlow development costVirtual Machines next to secure applicationsTU Dresden, 2009-12-01MOS - VirtualizationSlide 54 von 58
Summary Virtualization flavours L4Linux – paravirtualization in detail API or ABI emulationEmulationFull virtualizationHardware (especially x86) or OSParavirtualizitionAddress space layout & managementTaming Linux (interrupts, I/O memory)Faithful Virtualization Nova – minimal hypervisor VMM from scratchKVM-L4 reusing a VMMTU Dresden, 2009-12-01MOS - VirtualizationSlide 55 von 58
References Tom Van Vleck: 'The IBM 360/67 and CP/CMS'http://www.multicians.org/thvv/360-67.html Keith Adams and Ole Agesen: 'A Comparision ofSoftware and Hardware Techniques for x86Virtualization' ASPLOS 2006http://www.vmware.com/pdf/asplos235 adams.pdf Intel Virtualization /v10i3/1-hardware/1-abstract.htm H. Härtig, M. Roitzsch, A. Lackorzynski, B. Döbeland A. Böttcher: 'L4 – Virtualization andBeyond'TU Dresden, 2009-12-01MOS - VirtualizationSlide 56 von 58
References Udo Steinberg: 'NOVA Hypervisor ArchitectureWhitepaper' Internal Report 2007L4Linux Webpagehttp://os.inf.tu-dresden.de/L4/LinuxOnL4 Adam Lackorzynski: 'L4Linux PortingOptimizations' Diploma Thesis 2004http://os.inf.tu-dresden.de/papers ps/adam-diplom.pdfTU Dresden, 2009-12-01MOS - VirtualizationSlide 57 von 58
Outlook now, paper reading: Singularity - Rethinking the Software Stacknext weeks: legacy containersOS PersonalitiesTU Dresden, 2009-12-01MOS - VirtualizationSlide 58 von 58
TU Dresden, 2009-12-01 MOS - Virtualization Slide 6 von 58 Virtualization – a hype A lot of interest in the research community within the last years, e.g.: SOSP 03: Xen and the Art of Virtualization EuroSys 07: a whole session on virtualization Many virtualization products: VMware, QEmu, VirtualBox, KVM
In this section, we give an overview of virtualization and describe virtio, the virtualization standard for I/O devices. In addition, we discuss the state-of-the-art for network I/O virtualization. 2.1 Overview of Virtualization and virtio The virtualization technology is generally classi ed into full-virtualization and paravirtualization.
This guide also explains the advantages of virtualization and dispels some common myths that exist regarding virtualization. 1.1. Who should read this guide? This guide is designed for anyone wishing to understand the basics of virtualization, but may be of particular interest to: Those who are new to virtualization.
Lots of features (Contd.) Domain Isolation: VCPU and Host Interrupt Affinity Spatial and Temporal Memory Isolation Device Virtualization: Pass-through device support Block device virtualization Network device virtualization Input device virtualization Display device virtualization VirtIO v0.9.5 for Para-virtualization
physical entities, and categorizes virtualization on two levels: resource (or infrastructure) virtualization and service (or application) virtualization. In resource virtualization, physical resources such as network, compute, and storage resources are segmented or pooled as logical resources. An example of resource virtualization: Sharing a load
4 Virtualization For Dummies, Red Hat Special Edition Virtualization is a technology that lets one big physical server pretend to be a whole bunch of little PC machines, each with its own processor, memory, disk, and network devices. Explaining Server Virtualization When most organizations get started with virtualization, they
Desktop virtualization is driven by a combination of cost savings, increased ability to comply with myriad regulations, and an improvement in data and application security. The two fundamental forms of desktop virtualization are: Server-side application/desktop virtualization Client-side application/desktop virtualization !
The Red Hat Enterprise Linux Virtualization Guide contains information on installation, configuring, administering, and troubleshooting virtualization technologies included with Red Hat Enterprise Linux. iii . I. Requirements and Limitations for Virtualization with Red Hat Enterprise Linux 1 1. System requirements 3
PK-2 Next Generation ELA Standards at a Glance . PK-2 Reading Standards (Literary and Informational Text) Review the . PK, K, 1. st, and 2 nd grade ELA introductions for information regarding: guidance and support, range of student reading experiences, text complexity, English language learners/multilingual learners, and students with disabilities. Key Ideas and Details PK K 1 2 PKR1 .
