RLScheduler: An Automated HPC Batch Job Scheduler Using Reinforcement .

Previously, a scheduler is designed to optimize a fixedmetric. For example, Shortest Job First (SJF), Smallest JobFirst, and F1—F4 in Carastan-Santos et al. [4] target lowering average waiting time, increasing resource utilization, andminimizing average bounded slowdown, respectively. In thelifetime of a scheduler, when the system varies its schedulingmetric, the system administrator will tune the schedulingpolicies manually. In this study, we leverage reinforcementlearning and let the learning algorithm adjust its schedulingpolicies automatically for varying metrics.4) Scheduling and Backfilling: HPC platforms may provision multiple job queues and schedule jobs in different queuesdifferently. Without loss of generality, batch jobs are usuallysubmitted to batch queues and scheduled by centralized jobschedulers with backfiling techniques enabled [6].The process is straightforward. In batch queues, when a jobis selected, the system will seek for provisioning its requestedresources. If success, the resources will be allocated and thejob will start to run. Otherwise, the job will wait until itsrequest is satisfied [10]. In the mean time, backfilling can beactivated to search for the jobs whose resource allocations canbe satisfied now without affecting the planned execution forthe waiting job, to improve the efficiency of the system.B. Reinforcement Learning1) RL Concept: Reinforcement learning (RL) is a groupof machine learning techniques that enable agents to autonomously learn in an interactive environment by trials anderrors [14, 15]. In this study, we leverage this autonomy tobuild adaptive job schedulers for varying workloads and goals.AgentstateStrewardRtactionAtRt 1St 1EnvironmentFig. 1: General framework of reinforcement learning.Fig. 1 shows a general RL framework. At each step t,the agent observes the corresponding state St , and takes anaction At . Consequently, the action will transfer the environment state from St to St 1 and the agent will receivethe reward Rt 1 . In most cases, the agent does not have aprior knowledge on the environment state or the reward, andattain them gradually in the training process. The target ofreinforcement learning is to maximize the expected cumulativediscounted reward collected from the environment. The agenttakes actions based on policies, each defined as a probabilityof taking certain action at a given state. When the state spaceis enormous, memorizing all states becomes infeasible. DeepNeural Network (DNN) can be used to estimate the probability.The reinforcement learning using DNN to model the policy iscalled Deep Reinforcement Learning (DRL).2) RL training methods: Reinforcement learning has a largenumber of training methods, classified in different ways [20].But, a key difference among them is the training strategies,i.e., what the RL agent learns. The policy-based RL directlylearns the policy, which will output an action given a state;and policy gradient method is a typical example of them [21].The value-based RL learns proper value of each state, whichcan indirectly output an action by guiding the agent to movetowards better state; and Q-learning method is a typicalexample [22]. Between these two methods, policy gradientis proven to have strong convergence guarantees [21] andbecome our first choice. This is mostly due to the high varianceof batch job scheduling, which may lead to oscillations inQ-learning. To alleviate the known performance issues ofpolicy gradient, we follow the Actor-Critic model [23] inRLScheduler to combine both policy-based and value-basedlearning for better training efficiency. We return to this in § IV.III. D ISCUSSION ON C HALLENGESAt first glance, scheduling batch jobs with a deep reinforcement learning agent seems intuitive by repeating three simplesteps: 1) take the waiting jobs and idle compute resources ofthe target HPC environment as the input for a deep neuralnetwork (DNN); 2) use the DDN as the current schedulingpolicy to select a ‘best‘ job as the action; 3) apply the actionback to the environment. The training process repeats thethree steps until the last job in the job sequence is scheduled,which creates one sampled trajectory, and then computes thereward based on a given metric. With sufficient trajectoriesand their rewards, the policy gradient algorithm can be usedto update the policy (DNN) to maximize expected rewards ofthese trajectories, indicating a better scheduling algorithm.Although the process is standard for all policy gradientRL, the techniques used in each step is specific to the targetproblem and can affect the training efficiency and the agentcorrectness significantly. In this study, we address two keychallenges in the RL-based batch job scheduling problem.1) RL Network Architecture: In Fig. 2, we show how anDNN-based RL agent makes scheduling decisions: it takesthe waiting jobs and their features (e.g., a1 m ) as inputvector and outputs a probability distribution of each job beingscheduled next. The job with the highest probability (job8in this example) should be the selected job. One key issuehere, however, is the job orders in the waiting queue couldchange easily. As ‘step 1’ shown in the figure, job8 may holda different position in the queue next time, for example, fromthe second to the third. But, the RL’s DNN should still selectjob8 as the best even its placement is different.In general, there are two ways to achieve this. One wayis to consider the job orders in queue as the translation ordeformation of the inputs and learn these deformation byfeeding the DNN with more training data. The other optionis to make the DNN insensitive to job orders, and assign ajob with the same probability regardless its orders in the jobqueue. The former approach looks intuitive. Nevertheless, itusually requests much more training data, takes longer time

Job Queue1RL’s DNNjob0job0a1a2 amjob8b1b2 bmjob8job8b1b2 bmjob8zmjobk , jobkz1z2 1 Fig. 2: The DNN-based RL agent reads waiting jobs andselects a job as the scheduling decision.to converge, and may deliver lower model accuracy. In thisstudy, we take the latter approach and design a new kernelbased DNN architecture to be insensitive to job orders. Weexperimented on both approaches and confirmed that oursobtains much better performance in model training efficiencyand accuracy (§V).2) High Variance in Samples: The policy gradient algorithm essentially is a Monte-Carlo method, which samples alarge number of trajectories and uses their results to adjust theDNN (representing current policy). The key for this to work isthat these samples can accurately reflect the quality of policy,which, however, might not be true in batch job scheduling.For example, if a sampled job sequence arrive sparsely, theneach job can be instantly scheduled to run, their job waitingtimes will be 0 no matter what scheduling policy is used. Onthe contrary, if the sampled jobs arrive at the same time, theirwaiting times will be relatively long no matter what schedulingpolicy is used. If samples a lot of the first cases, then RL agentmay misinterpret the scheduling policy as good, which leadsto unstable or even non-converged results.sequence, and the horizontal axis shows the timeline of thejob trace (we show the first 10K jobs as an example).From this figure, we can see that, in most of the time, thejob slowdown is close to 1, which indicates the jobs barelywait in the queue. But there are short period of time (e.g.,the red range) where the average job slowdown reaches 80K,which indicates a really long job waiting time. The varianceis so high that it has two negative impacts. First, one ‘bad‘trajectory will diminish what RL agent has learned as we havediscussed. Second, too many ‘good’ trajectories will barelyteach RL agent anything during training, because no matterwhat scheduling policy it currently holds, the slowdown isgonna be 1. In RLScheduler, we propose strategies to eliminatethe high variance and ensure the convergence of RL trainingeven facing a workload like PIK-IPLEX-2009. More detailswill be discussed in the next section.IV. D ESIGN AND I MPLEMENTATIONRLScheduler uses reinforcement learning to derive adaptivepolicies for scheduling HPC batch jobs towards the varyingworkloads and optimization goals. Our approach is fundamentally different to the previous job schedulers, which rely onthe expert knowledge about workloads, job attributes, and theoptimization goals (discussed in §II-A, §II-B). RLScheduler isindependent from the knowledge and effort from experts. Theonly inputs it takes are the job traces and optimization goals,then it learns the scheduling by itself.This section first overviews the RLScheduler’s design andimplementation, then discusses its two key techniques: kernelbased neural network and variance reduction methods.A. RLScheduler OverviewActionPolicyNetworkJ1J2SchedGymRLScheduler AgentR valueValueNetworkJkObservable JobsRewardBaselineJob QueueStateRunning JobsMetric: average job slowdownState GenerationJ1J2 , JkJ3J7J20 , , Fig. 4: The overall architecture of RLScheduler.Timeline of the job traceFig. 3: The average bounded slowdown of scheduling sequence of 256 jobs in PIK-IPLEX-2009 job trace.The key is how high the variance of the batch job schedulingcan be in real world. In Fig. 3, we show an example of usingthe SJF (Shortest Job First, i.e., always selecting the job withthe shortest requested runtime) scheduling algorithm to schedule a sequence of 256 jobs sampled from the PIK-IPLEX2009 job trace, which is a real-world job trace collected fromIBM iDataPlex Cluster [17]. Here, the vertical axis shows theaverage job slowdown calculated from scheduling the wholeFig. 4 shows the RLScheduler architecture and its threemajor components: the Agent, the job scheduling Environment,and the environment State. In each step of the training process,the RLScheduler agent observes a state and takes an action.The state is collected and built from the environment. Theaction will be applied to the environment and consequentlygenerate the next state and a reward. Across steps, the agentlearns from its actions and the associated rewards.In particular, a reward is the feedback of the environment onan agent’s action. It serves as the key to guide the RL agent towards the better policies. In RLScheduler, reward is a function

addressing a user-given optimization goal. For instance, if theoptimization goal is to minimize average bounded slowdown(bsld), the reward can simply be reward bsld, whichmeans the RL will maximize the reward by minimizingthe average bounded slowdown. If the optimization goal isto maximize resource utilization (util), the reward can bereward util, which will directly maximize the utilization.Note that, the reward is supposed to be collected in eachlearning step whenever the agent takes an action. However,for most of the scheduling metrics, such as average waitingtime or average bounded slowdown, the calculation can not bedone until the whole job sequence gets scheduled. Thus, in themiddle of scheduling a job sequence, we just return rewards 0to each action and calculate the accurate reward for the entiresequence at the last action. This does not affect RL trainingas only the accumulated rewards are used for training.B. RLScheduler Kernel-based Neural NetworkThe RL agent’s deep neural networks play the key role inlearning. In RLScheduler, we leverage two networks: policynetwork and value network following the actor-critic model,to conduct the learning. They take the roles of generatingscheduling actions and facilitating the training respectively.1) Policy network: The policy network takes the updatedenvironmental state as its input and directly outputs an actiondetermining which job to run next.The key here is to make policy network insensitive to theorder of jobs (discussed in §III). To this end, we design athe pooling and fully connected layers of CNN as they aresensitive to job orders. We later confirmed the advantage ofour proposal in the evaluation section by comparing it withCNN and other MLP networks.The probability distribution of each waiting job serves twopurposes: 1) during training, it is sampled to obtain the nextaction. Sampling enables us to keep exploring new actions andpolicies; 2) during testing, it is directly used to select job withthe highest probability to ensure the best decision. There is noexploration anymore.The kernel-based design makes the policy network relativelysimple. Together with the small input dimension (i.e., a singlejob’s attributes each time), the parameter size of policy network becomes extremely small. In RLScheduler, we are ableto control the parameter size of the policy network less than1,000, which consequently improve the training efficiency.2) Value network: RLScheduler also includes a value network to formulate an Actor-Critic model to improve trainingefficiency. It takes a entire job sequence as inputs and outputsa value to indicate the expected reward of that sequence.StateJ1J2Value NetworkvalueJkStateJ1Kernel-based NetworkJ1Fig. 6: The RLScheduler value network structure. Its core isa 3-layer multiple layer perceptron network (MLP).J2softmaxJ2JkJkFig. 5: The RLScheduler policy network structure. Its core isa kernel-based neural network.kernel-based network as the policy network. Fig. 5 shows thenetwork in detail. The kernel network itself is a 3-layer fullyconnected network, structured the same as 3-layer perceptron(MLP) [24]. The difference is the kernel-based network will beapplied onto each waiting job one by one like a window. Foreach waiting job, the network outputs a value, a calculated‘score’ of the job. The values of all waiting jobs form avector. We then run softmax on the vector to generate aprobability distribution for each waiting job. In this way, oncejobs are reordered, their probabilities will also be reorderedaccordingly. This design is inspired by the kernel function inconvolutional neural networks (CNN) [25]. But, we eliminateFig. 6 shows the value network internal, which is a 3-layerMLP network, but does not have the kernel mechanism. Towork with MLP, the vectors of all jobs will be concated andflatten before feeding into the network.The value network is trained along with the policy network.Specifically, for a sequence of jobs, after the policy networkmakes all the scheduling decisions, we collect the rewards,then use this to train the value network to predict the rewardof a given job sequence.The output of value network can be intuitively consideredas the expected reward (expr ) of a set of jobs based on theagent’s current policies. It indicates how best the agent cando on this set of jobs historically. When we train the policynetwork, instead of directly using the accumulated rewards (r)collected from the environment, we can use (r expr ) to trainthe policy. This difference can be intuitively considered as theimprovement of current policy over historical policies on thisset of jobs. This strategy helps reduce the variance of inputsand lead to better training efficiency.3) The Inputs of DNNs: The inputs of both policy and valuenetworks are the state, which includes both job attributes andavailable resources in the systems. RLScheduler uses a vector

vj to embed such state info into each job (j). Then, multiplejobs will form a matrix as shown in Fig. 5.Each vector first contains all the available attributes of a job,such as job arriving time and request processors. A full list isshown in Table I. In addition to job attributes, the vector alsocontains available resources to indicate the available resourcein the system. The priority of a job actually varies dependingon the currently available resources.One practical issue of using DNNs to read all waiting jobsis the number of waiting jobs changes, but our DNNs only takefixed-size vector as inputs. To solve this, we limit RLSchedulerto only observe a fixed number (MAX OBSV SIZE) of jobs. Ifthere are fewer jobs, we pad the vector with all 0s job vector; ifthere are more jobs, we cut-off them selectively. The numberof observable jobs is a configurable training parameter. Weset it to 128 in RLScheduler by default, as many HPC jobmanagement systems, such as Slurm, also limit the numberof pending jobs to the same order of magnitude [26]. Whencut-off extra jobs, we simply leverage FCFS (first come firstserve) scheduling algorithm to sort all the pending jobs andselect the top MAX OBSV SIZE jobs.learn in a more stable way and converge faster. The secondstep trains on all the job sequences. Although they still havehigh variances, since the RL agent has already converged, ourexperiences show that the agent is hard to be misled again.Mid:1Mean: 7302 * Mean: 1460Fig. 7: The distribution of the average bounded slowdown ofscheduling sequence of 256 jobs in PIK-IPLEX-2009 job trace.C. RLScheduler Variance ReductionAs discussed in §III, the high variance in HPC batch jobswill impose significant challenges onto reinforcement learning.When we randomly sample job sequences from real-world jobtrace for training, such as the PIK-IPLEX-2009, RL agent willexperience the ‘easy sequences’ which mean any schedulingalgorithm will lead to good results and the ‘hard sequences’which mean any scheduling algorithm will lead to bad results.From the view of RL training, both cases are destructive: the‘easy sequences’ do not provide any meaningful knowledgeto the agent; while the ‘hard sequences’ simply confuse it.Recent studies have seen similar issues for training RLin such ‘input-driven’ environments and proposed to reducethe variances by memorizing the scheduling results for thesame job-arrival sequence and let the RL learn from therelative improvement on its own rather than from the absolutereward values [27, 28]. This solution has two major drawbacksin batch job scheduling: 1) it does not solve the issue of‘easy samples’, which could take a large portion of a realworld job trace and do not help in training RL except takingcomputation time; 2) memorizing history only helps whenthere are repeated visits on the same job sequence. Given thesize of real-world job traces, for example the PIK-IPLEX-2009has over 700K jobs, the re-visits are expected to be so rarethat the memorized history is sparse and insufficient to use.Instead, we introduce trajectory filtering in RLScheduler.The key idea is to filter out some job sequences during training,so that the RL agent will see sequences with controlledvariances and learn in a more stable way. In particular, itfilters the ‘easy sequences’ out since they will not contributeinfo to improve the RL agent. For the ‘non-easy sequences’,it categorizes all sequences into two ranges and trains the RLagent in two steps. The first step contains job sequences whosevariances fall into a specific range (R). So that the agent canThen, finding a good range (R) to rule out high variantjob sequences becomes important. To do so, we use a knownheuristic scheduling algorithm, i.e., Shortest Job First (SJF),to schedule a number of randomly sampled job sequencesfrom the job trace and collect their metrics values first. Wethen calculate its key statistical values: median, mean, andskewness. For a high varian

submitted to a centralized job scheduler via job scripts and wait in a job queue until the scheduler allocates the requested resources for them to execute. Once start, the jobs will run til finish, or fail, or get killed, in a batch way [1]. A batch job scheduler is designed to schedule jobs to obtain