Voyager: Complete Container State Migration

2017 IEEE 37th International Conference on Distributed Computing SystemsVoyager: Complete Container State MigrationShripad Nadgowda, Sahil Suneja, Nilton Bila and Canturk IsciIBM T.J. Watson Research CenterAbstract—Due to the small memory footprint andfast startup times offerred by container virtualization,made ever more popular by the Docker platform, containers are seeing rapid adoption as a foundationalcapability to build PaaS and SaaS clouds. For suchcontainer clouds, which are fundamentally differentfrom VM clouds, various cloud management servicesneed to be revisited. In this paper, we present ourVoyager - just-in-time live container migration service,designed in accordance with the Open Container Initiative (OCI) principles. Voyager is a novel filesystemagnostic and vendor-agnostic migration service thatprovides consistent full-system migration.Voyager combines CRIU-based memory migration together withthe data federation capabilities of union mounts tominimize migration downtime. With a union view ofdata between the source and target hosts, Voyagercontainers can resume operation instantly on the targethost, while performing disk state transfer lazily in thebackground.I. IntroductionContainer virtualization has existed since two decadesin the form of FreeBSD Jails[18], Solaris Zones[26],IBM AIX Workload Partitional WPAR[5], and LXC forLinux, amongst others. But these have recently startedgaining acceptance as a lightweight alternative to virtual machines (VMs), owing to technology maturity andpopularization by platforms like Docker[14], CoreOS’srocket[12], Cloud Foundry Warden[9]. Containers are being adopted as a foundational vitualization capability inbuilding Platform-as-a-Service (PaaS) and Software-as-aService (SaaS) cloud solutions, e.g. Amazon Containerservice[6], Google Container Engine [15] and IBM’s Container Cloud[17]. For such container clouds, which arefundamentally different from VM clouds, various cloudmanagement services need to be revisited. In this paper,we target one such service- container migration.An efficient migration solution becomes essential ascontainers start running production workloads. Borrowingthe scenarios from their VM counterparts, migrations arerequired during host maintenance, load balancing, serverconsolidation and movement between availability zones(e.g., a Silver zone with HDD storage and 100 IOPS vs.a Gold zone with SSD storage with 1000 IOPS).In this paper, we present our Voyager container migration service, tailored specifically for containers in accordance with OCI principles. Voyager is a novel filesystemagnostic and vendor-agnostic migration service that provides consistent full-system migration, unlike existing al1063-6927/17 31.00 2017 IEEEDOI 10.1109/ICDCS.2017.91ternatives that either provide memory-only migration, orrely on specific filesystems to migrate persistent storage(Section 2). Voyager performs just-in-time live containermigration with minimal downtime, by combining the datafederation capabilities of union mounts together withCRIU-based memory migration. With a union view of databetween the source and target hosts, Voyager containerscan resume operation instantly on the target host, whileperforming disk state transfer either on-demand (Copyon-Write) or through lazy replication. Our experimentsshow Voyager’s federation framework imposes no overheadduring data updates/writes, and 1% overheads for readsand upto 10% for scans.We have open-sourced an initial version of our data migration framework[8], although it works specifically withdocker containers and does not support live migration. Weintend to open-source the enhancements described in thispaper as well, which include in-memory state migration,OCI compliance, and support for multiple data storagetypes (rootfs, local and network attached data volumes).II. Background and Related WorkThe objective in this paper is not to compare andestablish performance advantages between VM and container migration techniques. Containers are fundamentallydifferent than VMs in terms of their system resourcerequirements, high density and agility. We believe a moreoptimal migration service can be designed for containerclouds than the prevalent ones in VM clouds.A. VM MigrationMigration has extensively been studied primarily forVM Clouds[7][25][24][16][22]. Different vendors use different virtual disk (vDisk) formats to encapsulate a VM’spersistent state (e.g. vmdk, VHD, qcow2), which aremigrated via proprietary hypervisor-assisted services likevMotion[29] and Hyper-V Live Migration[27], third partytools [7], [25], [13], or via explicit vDisk conversions[28][19]across hypervisors. Containers, on the other hand, are(being) standardised by the Open Container Initiative(OCI)[3] which specifies an industry standard for containerimage format- a f ilesystem bundle or rootf s, and multipledata volumes (Section III). These being essentially directories on the host filesystem, a generic file-based migrationsolution can be designed for containers. There also existvendor-agnostic file-based migration solutions for VMs like2137986

I2Map[22] and racemi[4], but there require extra agents tobe installed inside the systems.Some container clouds[6][15] provision VMs to host containers, primarily to mitigate security and isolation concerns for containers, while others run containers directlyon cloud hosts [17]. This paper shows how to migrate thesmall execution state that is part of a container withoutresorting to installing and migrating VMs that imposedifferent constraints on their hosts. Installing VMs forthe purpose of migration defeats the advantages of usingcontainers for application deployment.B. Container MigrationContainers have been popular with the microservicearchitecture. Although container migration may seem redundant for stateless containerized applications, but itis still pertinent to several stateful microservice applications like databases (e.g. Mysql, cassandra), message brokers (kafka), and state-coordination service (zookeeper),amongst others. This is being acknowledged and supportedin standard frameworks like Kubernetes’ ‘StatefulSet’.Portability of stateful containers is also explored in existing solutions like ClusterHQ’s Flocker[10], Virtuozzo[20]and Picocenter[30].Flocker[10] primarily is a data management solutionspecifically for docker containers. It supports migrationfor network-attached storage backends like Amazon EBS,Openstack Cinder, VMware vSphere etc., by re-attachingthese network storage for containers. Local attached volume migration is supported only for ZFS filesystem. Onthe other hand, Voyager is a generic, filesystem-agnosticand vendor-agnostic migration solution.Virtuozzo is a bare-metal virtualization solution thatincludes container virtualization. It facilitates Zerodowntime live migration for containers[23][20]. During thismigration it first transfer container’s filesystem and virtualmemory to target host. Once transfer is finished, it freezeall container processes and disable networking. It thendump this memory state to file and copies these dump fileto target host. Any changed memory and disk blocks sincethe last transfer are then migrated to target host and thencontainer is resumed. It has an underlying assumptionthat amount of memory pages and disk blocks changed(deltas) is small, thus outage time imperceptible. For anydata intensive application these deltas specially persistentdata changes could be large. Voyager differs from thistechnique primarily on two aspects. First, Voyager is aJust-in-Time (jit) Zero-copy migration solution, whereincontainer is migrated immediately before whole data iscopied to the target host. Secondly, Voyager does notrequire the task of second data transfer performed byVirtuozzo, thus application downtime for Voyager is stillsmaller than Virtuozzo. Further Voyager provided featureslike data federation access, dual-band data replication,OCI compliance which are not provided by any existingcontainer migr seconds. We have veryrecently also incorporated CRIU’s incremental memorycheckpointing capabilities in Voyager, which should lowerthe application downtime significantly. As future work, weplan to evaluate this optimization, and also instrumentCRIU to measure both the checkpoint and restore time ata per resource granularity, so that we can prioritize andoptimize for individual resources.C. Performance overheadIn Voyager, once a container is resumed at the target,it has immediate access to its persistent data storagethrough Voyager’s data federation layer. This layer incursperformance overhead, which we measure using YCSBfor different types of workload profiles including reads,inserts, updates, and scans. For each profile, in YCSB’sload stage we inserted 1M records to a database table,and in the run stage we performed 1M record operationsof respective types. The records were accessed using zipfiandistribution for popularity-based long tail access pattern.For each workload profile, we measured average application throughput (operations/sec) every 10 seconds. Eachexperiment was performed at two application states- (i)baseline: application state at the source host before it ismigrated, and (ii) f ederation: application state after it ismigrated at the target host, having access to data throughthe federation layer. For common application read/writeworkload patterns we observe zero to 3% overhead insteady state. Performance impact on the individual workload profiles is discussed below:D. Application container: startup vs. restoreWhen contrasted against VMs containers have faststartup times. But, applications inside container also typically have their own startup or initialization time, whichranges from milliseconds to few seconds. We measuredaverage initialization time to for two application namelyMySQL and elasticsearch. The average initialization timefor these application was recorded as: 10secs for MySQLand 7sec for elasticsearch. Then we checkpointed theseapplication containers right after they were initialized andrestored them on the same host. Restore time for both ofthem was less than 500ms, much faster than their startuptime. We are exploring use of Voyager on such platformswhere applications are instantiated by restoring their stateand rootf s is provisioned through data federation and lazycopy.2141990