UC#BERKELEY# Mesos: A Platform For Fine- Grained Resource .

UCBERKELEYMesos: A Platform for FineGrained Resource Sharingin Data Centers (II)Anthony D. JosephLASER Summer SchoolSeptember 2013

My Talks at LASER 20131. AMP Lab introduction2. The Datacenter Needs an Operating System3. Mesos, part one4. Dominant Resource Fairness5. Mesos, part two6. Spark2

Collaborators Matei Zaharia Benjamin Hindman Andy Konwinski Ali Ghodsi Randy Katz Scott Shenker Ion Stoica3

Apache MesosSpark CassandraMPIHiveHadoopHypertbaleA common resource sharing layer for diverse frameworksPIQL SCADSDatacenter “OS” (e.g., Apache Mesos)Node OS(e.g. Linux)Node OS(e.g. Windows) Node OS(e.g. Linux)Run multiple instances of the same framework» Isolate production and experimental jobs» Run multiple versions of the framework concurrentlySupport specialized frameworks for problem domains4

Implementation20,000 lines of C APIs in C, C , Java, and PythonMaster failover using ZooKeeperFrameworks ported: Hadoop, MPI, TorqueNew specialized frameworks: Spark, Apache/HaProxyOpen source Apache project http://mesos.apache.org/5

FrameworksPorted frameworks:» Hadoop (900 line patch)» MPI (160 line wrapper scripts)New frameworks:» Spark, Scala framework for iterative jobs (1300 lines)» Apache haproxy, elastic web server farm (200 lines)6

IsolationMesos has pluggable isolation modules to isolatetasks sharing a nodeCurrently supports Linux Containers and Solarisprojects» Can isolate memory, CPU, IO, network bandwidthCould be a great place to use VMs7

Apache e servers require coordination» Leader Election, Group Membership, Work Queues, DataSharding, Event Notifications, Configuration, and ClusterManagementHighly available, scalable, distributed coordination kernel» Ordered updates and strong persistence guarantees» Conditional updates (version), Watches for data changes

Connect tocurrently activemasterMaster FailureSchedulerHadoopJobTrackerMesos Master2Mesos Master3Mesos Master1SchedulerMPISchedulerZooKeeper used toelect one activeMesos masterZoo1Zoo5ZooKeeperZoo4Zoo2Zoo3Connect tocurrently activemasterSlave 1HadoopExecutorMPI ExecutorSlave 2HadoopExecutorHadoop ExecutorSlave 3JVM Executor9

Resource RevocationKilling tasks to make room for other usersKilling typically not needed for short tasks» If avg task length is 2 min, a new framework gets 10% ofall machines within 12 seconds on avgHadoop job and task durations at Facebook10

Resource Revocation (2)Not the normal case because fine-grained tasks enablequick reallocation of resourcesSometimes necessary:» Long running tasks never relinquishing resources» Buggy job running forever» Greedy user who decides to makes his task longSafe allocation lets frameworks have long running tasksdefined by allocation policy» Users will get at least safe share within specified time» If stay below safe allocation, task won’t be killed11

Resource Revocation (3)Dealing with long tasks monopolizing nodes» Let slaves have long slots and short slots» Short slots killed if used too long by a taskRevoke only if a user is below its safe share and isinterested in offers» Revoke tasks from users farthest above their safe share» Framework given a grace period before killing its tasks12

Example: Running MPI on MesosUsers always told their safe share» Avoid revocation by staying below itGiving each user a small safe share may not beenough if jobs need many machinesCan run a traditional HPC scheduler as a userwith a large safe share of the cluster, and have MPIjobs queue up on it» E.g. Torque gets 40% of cluster13

Example: Torque on MesosFacebook.com20%Job 1Job 2TorqueAdsSpamUser 140%40%User 2Job 1Safe share 40%MPIJobMPI MPI MPIJobJobJobJob 414

Some Mesos Deployments1,000’s of nodes running over a dozenproduction servicesGenomics researchers using Hadoop andSpark on MesosSpark in use by Yahoo! ResearchSpark for analyticsHadoop and Spark used by machine learningresearchers15

Results

Share of ClusterDynamic Resource park1316191121151181211241271301331Time (s)17

Elastic Web Server FarmLoadcalculationHTTP d genframeworkresourceofferMesos masterMesos slaveMesos slaveLoad genexecutorWebexecutorLoad che)statusupdateMesos slaveLoad gen executorWebexecutortasktasktask(Apache)18

Web Framework Results19

ScalabilityTask startup overhead with 200 frameworks20

Fault ToleranceMean time to recovery, 95% confidence21

Deep Dive ExperimentsMacrobenchmark experiment» Test the benefits of using Mesos to multiplex a clusterbetween multiple diverse frameworksHigh level goals of experiment» Demonstrate increased CPU/memory utilization due tomultiplexing available resources» Demonstrate job runtime speedups22

Macrobenchmark setup100 Extra Large EC2 instances (4 cores/15GB ramper machine)Experiment length: 25 minutesRealistic workload1. A Hadoop instance running a mix of small and largejobs based on the workload at Facebook2. A Hadoop instance running a set of large batch jobs3. Spark running a series of machine learning jobs4. Torque running a series of MPI jobs23

Goal of experimentRun the four frameworks and correspondingworkloads » 1st on a cluster that is shared via Mesos» 2nd on 4 partitioned clusters, each ¼ the size of theshared clusterCompare resource utilization and workloadperformance (i.e., job run times) on staticpartitioning vs. sharing with Mesos24

Macrobenchmark Details: Breakdown of the Facebook Hive(Hadoop) Workload mixBinJob TypeMap TasksReduce TasksJobs Run1Selection1NA382Text gation1001066Selection200NA67Text Search400NA48Join40030225

Results: CPU Allocation100 node clusterHadoop (Facebook Mix)Hadoop (Batch Jobs)26

Sharing With Mesos vs. No-Sharing (Dedicated Cluster)10.80.60.40.20(b) Large Hadoop MixShare of ClusterShare of Cluster(a) Facebook Hadoop MixDedicated 01600Dedicated ClusterMesos05001000Time (s)2004006008001000Time (s)Share of ClusterShare of Cluster25003000(d) Torque / MPIDedicated ClusterMesos02000Time (s)(c) .20Dedicated ClusterMesos0200400600800100012001400Time (s)271600

CPU Utilization (%)Cluster Utilization Mesos vs. Dedicated 0014001600Memory Utilization (%)Time 1600Time (s)28

Job Run Times (and Speedup) Grouped By FrameworkFrameworkSum of Exec times on Sum of Exec TimesDedicated Cluster (s)on Mesos (s)SpeedupFacebook Hadoop Mix723563191.14Large Hadoop Mix314314942.10Spark168413381.26Torque / MPI321033520.962x speedup forLarge Hadoop Mix29

Job Run Times (and Speedup) Grouped by Job TypeFrameworkFacebook HadoopMixJob Typeselection (1)text search (2)aggregation (3)selection (4)aggregation (5)selection (6)text search (7)join (8)Large Hadoop Mixtext searchSparkALSTorque / MPIsmall tachyonlarge tachyonTime on DedicatedCluster (s)24318265192136137662314337261822Avg. Speedupon .8830

Discussion: Facebook Hadoop Mix ResultsSmaller jobs perform worse on Mesos:» Side effect of interaction between fair sharing performedby Hadoop framework (among its jobs) and performedby Mesos (among frameworks)» When Hadoop has more than 1/4 of the cluster, Mesosallocates freed up resources to framework farthest belowits share» Significant effect on any small Hadoop job submittedduring this time (long delay relative to its length)» In contrast, Hadoop running alone can assign resources tothe new job as soon as any of its tasks finishes31

Discussion: Facebook HadoopMix ResultsSimilar problem with hierarchical fair sharingappears in networks» Mitigation #1: run small jobs on a separate framework, or» Mitigation #2: use lottery scheduling as the Mesosallocation policy32

Discussion: Torque ResultsTorque is the only framework that performedworse, on average, on Mesos» Large tachyon jobs took on average 2 minutes longer» Small ones took 20s longer33

Discussion: Torque ResultsCauses of delay» Partially due to Torque having to wait to launch 24 tasks onMesos before starting each job – average delay is 12s» Rest of the delay may be due to stragglers (slow nodes)» In standalone Torque run, two jobs each took 60s longerto run than others» Both jobs used a node that performed slower on singlenode benchmarks than the others (Linux reported a 40%lower bogomips value on the node)» Since tachyon hands out equal amounts of work toeach node, it runs as slowly as the slowest node34

Macrobenchmark SummaryEvaluated performance of diverse set of frameworksrepresenting realistic workloads running on Mesosversus a statically partitioned clusterShowed 10% increase in CPU utilization, 18%increase in memory utilizationSome frameworks show significant speed ups in jobrun timeSome frameworks show minor slowdowns in job runtime due to experimental/environmental artifacts35

SummaryMesos is a platform for sharing data centersamong diverse cluster computing frameworks» Enables efficient fine-grained sharing» Gives frameworks control over schedulingMesos is» Scalable (50,000 slaves)» Fault-tolerant (MTTR 6 sec)» Flexible enough to support a variety of frameworks(MPI, Hadoop, Spark, Apache, )36

My Talks at LASER 20131. AMP Lab introduction2. The Datacenter Needs an Operating System3. Mesos, part one4. Dominant Resource Fairness5. Mesos, part two6. Spark37

per machine)! Experiment length: 25 minutes! Realistic workload! 1. A Hadoop instance running a mix of small and large jobs based on the workload at Facebook! 2. A Hadoop instance running a set of large batch jobs! 3. Spark running a series of machine learning jobs! 4. Torque running a series of MPI jobs! 23!