My VM Is Lighter (and Safer) Than Your Container

2y ago
12 Views
2 Downloads
1.85 MB
16 Pages
Last View : 4m ago
Last Download : 2m ago
Upload by : Ophelia Arruda
Transcription

My VM is Lighter (and Safer) than your ContainerFilipe MancoNEC Laboratories Europe lipe.manco@gmail.comJose MendesNEC Laboratories Europejose.mendes@neclab.euKenichi YasukataNEC Laboratories Europekenichi.yasukata@neclab.euCostin LupuUniv. Politehnica of Bucharestcostin.lupu@cs.pub.roSimon KuenzerNEC Laboratories Europesimon.kuenzer@neclab.euCostin RaiciuUniv. Politehnica of Bucharestcostin.raiciu@cs.pub.roABSTRACTContainers are in great demand because they are lightweightwhen compared to virtual machines. On the downside, containers o er weaker isolation than VMs, to the point wherepeople run containers in virtual machines to achieve properisolation. In this paper, we examine whether there is indeeda strict tradeo between isolation (VMs) and e ciency (containers). We nd that VMs can be as nimble as containers, aslong as they are small and the toolstack is fast enough.We achieve lightweight VMs by using unikernels for specialized applications and with Tinyx, a tool that enablescreating tailor-made, trimmed-down Linux virtual machines.By themselves, lightweight virtual machines are not enoughto ensure good performance since the virtualization controlplane (the toolstack) becomes the performance bottleneck.We present LightVM, a new virtualization solution basedon Xen that is optimized to o er fast boot-times regardlessof the number of active VMs. LightVM features a completeredesign of Xen’s control plane, transforming its centralizedoperation to a distributed one where interactions with thehypervisor are reduced to a minimum. LightVM can boot aVM in 2.3ms, comparable to fork/exec on Linux (1ms), andtwo orders of magnitude faster than Docker. LightVM canpack thousands of LightVM guests on modest hardware withmemory and CPU usage comparable to that of processes.Permission to make digital or hard copies of part or all of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for pro t or commercial advantage and that copiesbear this notice and the full citation on the rst page. Copyrights for thirdparty components of this work must be honored. For all other uses, contactthe owner/author(s).SOSP ’17, October 28, 2017, Shanghai, China 2017 Copyright held by the owner/author(s).ACM ISBN 32747.3132763Florian SchmidtNEC Laboratories Europe orian.schmidt@neclab.euSumit SatiNEC Laboratories Europesati.vicky@gmail.comFelipe HuiciNEC Laboratories Europefelipe.huici@neclab.euCCS CONCEPTS Software and its engineering Virtual machines;Operating Systems;KEYWORDSVirtualization, unikernels, specialization, operating systems,Xen, containers, hypervisor, virtual machine.ACM Reference Format:Filipe Manco, Costin Lupu, Florian Schmidt, Jose Mendes, SimonKuenzer, Sumit Sati, Kenichi Yasukata, Costin Raiciu, and FelipeHuici. 2017. My VM is Lighter (and Safer) than your Container. InProceedings of SOSP ’17: ACM SIGOPS 26th Symposium on OperatingSystems Principles, Shanghai, China, October 28, 2017 (SOSP ’17),16 ODUCTIONLightweight virtualization technologies such as Docker [6]and LXC [25] are gaining enormous traction. Google, forinstance, is reported to run all of its services in containers [4],and Container as a Service (CaaS) products are availablefrom a number of major players including Azure’s ContainerService [32], Amazon’s EC2 Container Service and Lambdao erings [1, 2], and Google’s Container Engine service [10].Beyond these services, lightweight virtualization is crucial to a wide range of use cases, including just-in-time instantiation of services [23, 26] (e.g., lters against DDoSattacks, TCP acceleration proxies, content caches, etc.) andNFV [41, 51], all while providing signi cant cost reductionthrough consolidation and power minimization [46].The reasons for containers to have taken the virtualization market by storm are clear. In contrast to heavyweight,hypervisor-based technologies such as VMWare, KVM orXen, they provide extremely fast instantiation times, smallper-instance memory footprints, and high density on a singlehost, among other features.

Manco et al.No. of syscallsSOSP ’17, October 28, 2017, Shanghai, China An overhaul of Xen’s architecture, completely removing its back-end registry (a.k.a. the XenStore), whichconstitutes a performance bottleneck. We call this noxs(no XenStore), and its implementation results in signi cant improvements for boot and migration times,among other metrics. A revamp of Xen’s toolstack, including a number ofoptimizations and the introduction of a split toolstackthat separates functionality that can be run periodically, o ine, from that which must be carried out whena command (e.g., VM creation) is issued. The development of Tinyx, an automated system forbuilding minimalistic Linux-based VMs, as well as thedevelopment of a number of unikernels. These lightweight VMs are fundamental to achieving high performance numbers, but also for discovering performancebottlenecks in the underlying virtualization platform. A prototypical implementation along with an extensive performance evaluation showing that LightVM isable to boot a (unikernel) VM in as little as 2.3ms, reachsame-host VM densities of up to 8000 VMs, migrationand suspend/resume times of 60ms and 30ms/25ms respectively, and per-VM memory footprints of as littleas 480KB (on disk) and 3.6MB (running).4003503002502002002 2004 2006 2008 2010 2012 2014 2016 2018Linux Release YearFigure 1: The unrelenting growth of the Linux syscallAPI over the years (x86 32) underlines the di cultyof securing containers.However, no technology is perfect, and containers are noexception: security is a continuous thorn in their side. Themain culprit is the hugely powerful kernel syscall API thatcontainers use to interact with the host OS. This API is verybroad as it o ers kernel support for process and thread management, memory, network, lesystems, IPC, and so forth:Linux, for instance, has 400 di erent system calls [37], mostwith multiple parameters and many with overlapping functionality; moreover, the number of syscalls is constantly increasing (see gure 1). The syscall API is fundamentally moredi cult to secure than the relatively simple x86 ABI o eredby virtual machines where memory isolation (with hardwaresupport) and CPU protection rings are su cient. Despite themany isolation mechanisms introduced in the past few years,such as process and network namespaces, root jails, seccomp,etc, containers are the target of an ever increasing number ofexploits [11, 22]. To complicate matters, any container thatcan monopolize or exhaust system resources (e.g., memory, le descriptors, user IDs, forkbombs) will cause a DoS attackon all other containers on that host [14, 35].At least for multi-tenant deployments, this leaves us witha di cult choice between (1) containers and the securityissues surrounding them and (2) the burden coming fromheavyweight, VM-based platforms. Securing containers inthe context of an ever-expanding and powerful syscall API iselusive at best. Could we make virtualization faster and morenimble, much like containers? The explicit goal would beto achieve performance in the same ball-park as containers:instantiation in milliseconds, instance memory footprints ofa few MBs or less, and the ability to concurrently run onethousand or more instances on a single host.In this paper we introduce LightVM, a lightweight virtualization system based on a type-1 hypervisor. LightVM retainsthe strong isolation virtual machines are well-known forwhile providing the performance characteristics that makecontainers such an attractive proposition. In particular, wemake the following contributions: An analysis of the performance bottlenecks preventing traditional virtualization systems from achievingcontainer-like dynamics (we focus our work on Xen).To show its applicability, we use LightVM to implementfour use cases: personalized rewalls, just-in-time service instantiation, high density TLS termination and a lightweightcompute service akin to Amazon Lambda or Google’s CloudFunctions but based on a Python unikernel. LightVM is available as open source at NTSThe goal is to be able to provide lightweight virtualizationon top of hypervisor technology. More speci cally, as requirements, we are interested in a number of characteristicstypical of containers: Fast Instantiation: Containers are well-known fortheir small startup times, frequently in the range ofhundreds of milliseconds or less. In contrast, virtualmachines are infamous for boot times in the range ofseconds or longer. High Instance Density: It is common to speak ofrunning hundreds or even up to a thousand containerson a single host, with people even pushing this boundary up to 10,000 containers [17]. This is much higherthan what VMs can typically achieve, which lies morein the range of tens or hundreds at most, and normallyrequires fairly powerful and expensive servers. Pause/unpause: Along with short instantiation times,containers can be paused and unpaused quickly. This

Boot time (ms)My VM is Lighter (and Safer) than your ContainerSOSP ’17, October 28, 2017, Shanghai, Chinain images that are a few tens of MBs in size and needaround 30MBs of RAM to boot.100080060040020003.1UnikernelsThe single biggest factor limiting both the scalability andperformance of virtualization is the size of the guest virtual machines: for instance, both the on-disk image size aswell as the running memory footprint are on the order ofhundreds of megabytes to several gigabytes for most Linuxdistributions. VM memory consumption imposes a hard upper bound on the number of instances that can be run on thesame server. Containers typically require much less memorythan virtual machines (a few MBs or tens of MBs) becausethey share the kernel and have smaller root lesystems.Large VMs also slow down instantiation times: the timeneeded to read the image from storage, parse it and lay itout in memory grows linearly with image size. This e ect isclearly shown in Figure 2 where we boot the same unikernelVM from images of di erent sizes, all stored in a ramdisk.We increase the size by injecting binary objects into theuncompressed image le. This ensures that the results aredue to the e ects that image size has on VM initialization.There is a lot of prior work showing that unikernels havevery low memory footprint, and for speci c applicationsthere already exist images that one can re-use: ClickOS isone such example that can run custom Click modular routercon gurations composed of known elements. Mirage [27] isanother example that takes applications written in OCamland creates a minimalistic app OS combo that is packed asa guest VM.If one needs to create a new unikernel, the simplest isto rely on Mini-OS [34], a toy guest operating system distributed with Xen: its functionality is very limited, there is nouser/kernel separation and no processes/fork. For instance,only 50 LoC are needed to implement a TCP server overMini-OS that returns the current time whenever it receives aconnection (we also linked the lwip networking stack). Theresulting VM image, which we will refer to as the daytimeunikernel, is only 480KB (uncompressed), and can run in aslittle as 3.6MB of RAM.1 We use the daytime unikernel as alower bound of memory consumption for possible VMs.We have also created unikernels for more interesting applications, including a TLS termination proxy and Minipython, a Micropython-based unikernel to be used by Amazonlambda-like services; both have images of around 1MB andcan run with just 8MB of memory.In general, though, linking existing applications that relyon the Linux syscall API to Mini-OS is fairly cumbersome andrequires a lot of expert time. That is why we also exploredanother approach to creating lightweight VMs based on theLinux kernel, described next.33.2Tinyx02004006008001000VM image size (MB)Figure 2: Boot times grow linearly with VM image size.can be used to achieve even higher density by pausingidle instances, and more generally to make better use ofCPU resources. Amazon Lambda, for instance, “freezes”and “thaws” containers.LIGHTWEIGHT VMSThe rst step towards achieving our goals is to reduce boththe image size and the memory footprint of virtual machines.We observe, as others [27], that most containers and virtualmachines run a single application; if we reduce the functionality of the VM to include only what is necessary for thatapplication, we expect to reduce the memory usage dramatically. Concretely, we explore two avenues to optimize virtualmachine images: Unikernels: tiny virtual machines where a minimalistic operating system (such as MiniOS [34]) is linked directly with the target application. The resulting VM istypically only a few megabytes in size and can only runthe target application; examples include ClickOS [29]and Mirage [27]. Tinyx: a tool that we have built to create a tiny Linuxdistribution around a speci ed application. This resultsTinyx is an automated build system that creates minimalisticLinux VM images targeted at running a single application (although the system supports having multiple ones). The toolbuilds, in essence, a VM consisting of a minimalistic, Linuxbased distribution along with an optimized Linux kernel. Itprovides a middle point between a highly specialized unikernel, which has the best performance but requires porting ofapplications to a minimalistic OS, and a full- edged generalpurpose OS VM that supports a large number of applicationsout of the box but incurs performance overheads.The Tinyx build system takes two inputs: an application tobuild the image for (e.g., nginx) and the platform the imagewill be running on (e.g., a Xen VM). The system separatelybuilds a lesystem/distribution and the kernel itself. For the1 The3.6MB requires a small patch to Xen’s toolstack to get around the factthat it imposes a 4MB minimum by default.

Manco et al.SOSP ’17, October 28, 2017, Shanghai, China4VIRTUALIZATION TODAYArmed with our tiny VM images, we are now ready to explore the performance of existing virtualization technologies.We base our analysis on Xen [3], which is a type-1 hypervisor widely used in production (e.g., in Amazon EC2). Xenhas a small trusted computing base and its code is fairlymature, resulting in strong isolation (the ARM version ofDom0 (Linux/NetBSD)xllibxltoolstacklibxcDomU 1appslibxsOS ackblockSW switchdriversNICdistribution, Tinyx includes the application, its dependencies,and BusyBox (to support basic functionality).To derive dependencies, Tinyx uses (1) objdump to generate a list of libraries and (2) the Debian package manager.To optimize the latter, Tinyx includes a blacklist of packages that are marked as required (mostly for installation, e.g.,dpkg) but not strictly needed for running the application. Inaddition, we include a whitelist of packages that the usermight want to include irrespective of dependency analysis.Tinyx does not directly create its images from the packages since the packages include installation scripts whichexpect utilities that might not be available in the minimalistic Tinyx distribution. Instead, Tinyx rst mounts an emptyOverlayFS directory over a Debian minimal debootstrap system. In this mounted directory we install the minimal set ofpackages discovered earlier as would be normally done inDebian. Since this is done on an overlay mounted system,unmounting this overlay gives us all the les which are properly con gured as they would be on a Debian system. Beforeunmounting, we remove all cache les, any dpkg/apt related les, and other unnecessary directories. Once this is done,we overlay this directory on top of a BusyBox image as anunderlay and take the contents of the merged directory; thisensures a minimalistic, application-speci c Tinyx “distribution”. As a nal step, the system adds a small glue to run theapplication from BusyBox’s init.To build the kernel, Tinyx begins with the “tinycon g”Linux kernel build target as a baseline, and adds a set ofbuilt-in options depending on the target system (e.g., Xenor KVM support); this generates a working kernel image.By default, Tinyx disables module support as well as kerneloptions that are not necessary for virtualized systems (e.g.,baremetal drivers). Optionally, the build system can takea set of user-provided kernel options, disable each one inturn, rebuild the kernel with the olddefconfig target, bootthe Tinyx image, and run a user-provided test to see if thesystem still works (e.g., in the case of an nginx Tinyx image,the test includes attempting to wget a le from the server);if the test fails, the option is re-enabled, otherwise it is leftout of the con guration. Combined, all these heuristics helpTinyx create kernel images that are half the size of typicalDebian kernels and signi cantly smaller runtime memoryusage (1.6MB for Tinyx vs. 8MB for the Debian we tested).Xen HypervisorHardware (CPU, Memory, MMU, NICs, )Figure 3: The Xen architecture including toolstack, theXenStore, software switch and split drivers betweenthe driver domain (Dom0) and the guests (DomUs).the hypervisor, for instance, consists of just 11.4K LoC [43],and dissagregation [5] can be used to keep the size of criticalDom0 code low). The competing hypervisor, KVM, is basedon the Linux kernel and has a much larger trusted computingbase. To better understand the following investigation, westart with a short introduction on Xen.4.1Short Xen PrimerThe Xen hypervisor only manages basic resources such asCPUs and memory (see Figure 3). When it nishes booting,it automatically creates a special virtual machine called Dom0.Dom0 typically runs Linux and hosts the toolstack, which includes the xl command and the libxl and libxc librariesneeded to carry out commands such as VM creation, migration and shutdown.Dom0 also hosts the XenStore, a proc-like central registrythat keeps track of management information such as whichVMs are running and information about their devices, alongwith the libxs library containing code to interact with it.The XenStore provides watches that can be associated withparticular directories of the store and that will trigger callbacks whenever those directories are modi ed.Typically, Dom0 also hosts a software switch (Open vSwitchis the default) to mux/demux packets between NICs and theVMs, as well as the (Linux) drivers for the physical devices.2For communication between Dom0 and the other guests, Xenimplements a split-driver model: a virtual back-end driverrunning in Dom0 (e.g., the netback driver for networking)communicates over shared memory with a front-end driverrunning in the guests (the netfront driver). So-called event2 Strictlyspeaking, this functionality can be put in a separate VM called adriver domain, but in most deployments Dom0 acts as a driver domain.

My VM is Lighter (and Safer) than your ContainerDebian BootDebian CreateTinyx BootTinyx CreateMiniOS BootMiniOS CreateSOSP ’17, October 28, 2017, Shanghai, ChinaDocker BootDocker RunProcess CreateTime [ms]105Time 1002004006008001000Number of running guests02004006008001000Number of running guestsFigure 4: Comparison of domain instantiation andboot times for several guest types. With small guests,instantiation accounts for most of the delay whenbringing up a new VM.channels, essentially software interrupts, are used to notifydrivers about the availability of ckOverhead InvestigationAs a testing environment, we use a machine with an IntelXeon E5-1630 v3 CPU at 3.7 GHz and 128 GiB of DDR4 memory, and Xen and Linux versions 4.8. We then sequentiallystart 1000 virtual machines and measure the time it takes tocreate each VM (the time needed for the toolstack to preparethe

main culprit is the hugely powerful kernel syscall API that containers use to interact with the host OS. This API is very broad as it oers kernel support for process and thread man-agement, memory, network,lesystems, IPC, and so forth: Linux, for instance, has 400 dierent system calls [37], most

Related Documents:

Charlie’s Angels - Digital Artist What Planet Are You From? - Lighter/Compositor Stuart Little - Previz Lead and Lighter/Compositor Godzilla - Previz Artist, Effects Animator and Lighter/Compositor Starship Troopers

lighters are shipped unfilled. zippo blu accessories genuine zippo lighter bottom has three important elements: fuel port, date code and mark. fuel indicator window lighter has convenient fuel indicator and large fueltank. new zippo blu lighter, butane gas fuel 3801 1.9 oz (54 gra

Contact your Zippo Sales Representative for additional Zippo outdoor-themed pocket lighters. WHEN THE OUTDOORS COMES CALLING, ANSWER WITH THE FAMOUS ZIPPO CLICK. 24817 Street Chrome 27.95 3341 Lighter Fluid 4 Oz. 3.00 All Zippo Pocket Lighters are shipped empty without lighter fluid. So be sure to stock Zippo Lighter Fluid for your customers .

Zippo’s MultiCut lighter designs are leaders in the luxury lighter market and as consumer “must-haves.” Available exclusively on the Zippo’s Armor lighters, Zippo’s 360 MultiCut process seamlessly deep-carves the lighter case with incredible precision and detail, envelopin

Zippo lighter! George G. laisdell founded the Zippo Manufacturing ompany in 1932, and made the first Zippo lighter in early 1933. He was inspired by lighters produced by IMO, an Austrian cigarette lighter manufacturer. laisdell liked the sound of the word zipper and "zippo" sounded more m

The World Famous Zippo Guarantee Any Zippo pocket lighter, when returned for repair, will be put in rst-class mechanical condition free of charge regardless of age or condition. The nish, however, is not guaranteed. When returning a Zippo lighter for repair, please enclose all parts: top and bottom of the lighter

Point the tip of the torch (4) in a safe direction. To light with a spark lighter, (5) open fuel valve (1) slowly just before sparking. Hold spark lighter at a 45 degree angle to the tip of the torch (4). Squeeze spark lighter to create sparks and remove once torch is lit. NOTE: It

RONSON VARAFLAME PREMIER POCKET LIGHTER Chrom ium plate. Bright Fini sh. Engraved design. 17103 14.95 ME BERS' PRICE POCKET LIGHTER Satin and Bright fin ish. En graved design . 67002 9.95 MEMBERS ' PRICE 6.47 I& / ZIPPO BRUSH FINISH LIGHTER 200 . 3.50 MEMB ERS' PRICE 2.10 Adjustab le ladies' shaver . Exclusive roller combs. 1