Slurm Job Scheduling - Hprc.tamu.edu

18d ago
15 Views
0 Downloads
4.64 MB
43 Pages
Last View : Today
Last Download : n/a
Upload by : Samir Mcswain
Transcription

Slurm Job SchedulingTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu

Job Scheduling SBATCH Parameters Single node jobs single-core multi-core Multi-node jobs MPI jobs TAMULauncher array jobs Monitoring job resource usage at runtime after job completion job debuggingTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu2

HPC Diagramcompute node 1compute node 2compute node 3login node 1login node 2Slurm JobSchedulerlogin node 3compute node 4compute node 5compute node 6hundreds of more compute nodesShared File StorageTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu3

Terra Service Unit Calculations For the Terra 64GB memory nodes (56GB available), you are charged ServiceUnits (SUs) based on one of the following values whichever is greater. 1 SU per CPU per hour or 1 SU per 2GB of requested memory per hourNumber ofCoresTotal Memoryper node (GB)HoursSUs charged121113121561282856128unused SUs expire at the end of each fiscal year (Aug 31) and must be renewedTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu4

Nodes and Cores A node is one computer unit of an HPC cluster each containing memory and one ormore CPUs. There are generally two classifications of nodes: login and compute. login node this is where users first login to stage their job scripts and do file anddirectory manipulations with text file editors and Unix commands. compute node A cluster can contain a few compute nodes or thousands of computenodes. These are often referred to as just nodes since jobs are onlyscheduled on the compute nodes. number of compute nodes to reserve for a job can be specified with the--nodes parameterCores Slurm refers to cores as cpus. there are 28 cores on the Terra 64GB memory compute nodes number of cores can be specified with the --cpus per task parameterTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu5

Single vs Multi-Core Jobs When to use single-core jobs The software being used only supports commands utilizing a single-coreWhen to use multi-core jobs If the software supports multiple-cores (--threads, --cpus, ) then configurethe job script and software command options to utilize all CPUs on a computenode to get the job done faster unless the software specifically recommends alimited number of cores 28 CPUs (cores) per compute node on Terra 64GB memory nodes 56GB of available memory per compute node on Terra 64GB memorynodes Can group multiple single-core commands into a "multi-core" job usingTAMULauncher on one or multiple nodesTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu6

Submitting Slurm Jobs Jobs can be submitted using a job script or directly on the command lineA job script is a text file of Unix commands with #SBATCH parameters#SBATCH parameters provide resource configuration request values time, memory, nodes, cpus, output files, .Submit the job using sbatch command with the job script name your job script provides a record of commands used for an analysis Submit command on the command line by specifying all necessary parameters must rely on your bash history to see #SBATCH parameters used which is not reliable slurm.schedmd.com/sbatch.htmlTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu7

Slurm Job Script Parameters Always include the first two lines exactly as they are. In rare cases, line two is not needed.Slurm job parameters begin withand can add comments afterwards as aboveName the job script whatever you like run program project.sh my job script.job my job script.sbatch my job script.txtTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu8

Slurm Parameters: nodes, tasks, cpus --nodes number of nodes to use where a node is one computer unit of many in an HPC cluster used for multi-node jobs if number of cpus per node is not specified then defaults to 1 cpu defaults to 1 node if --nodes not used & can use together with --ntasks-per-node and --cpus-per-task do not use --nodes with --array--ntasks a task can be considered a command such as blastn, bwa, script.py, etc. when using --ntasks without --nodes, the values for --ntasks-per-node and --cpus-per-task willdefault to 1 node, 1 task per node and 1 cpu per task--ntasks-per-nodeone task28 cpus use together with --cpus-per-task --cpus-per-taskmagicblast -num threads 28 number of CPUs (cores) for each task (command) Texas A&M UniversityHigh Performance Research Computinghprc.tamu.edu9

Additional Slurm Parameters --job-name set the job name, keep it short and concise without spaces --time max runtime for job; format: days-hours minutes seconds (days- is optional) --mem total memory for each node (not for each CPU core as with Ada) --partition specify a partition (queue) to use partition is automatically assigned to short, medium, long and gpu (when using --gres gpu) only need to specify --partition parameter to use vnc, xlong, special, knl --output save all stdout to a specified file --error save all stderr to a specified file use just --output to save stdout and stderr to the same output file:Texas A&M UniversityHigh Performance Research Computinghprc.tamu.edu10

Optional Slurm Parameters --gres used to request 1 or 2 GPUs; automatically assigns --partition gpu --account specify which HPRC account to use; see your accounts with the default account fromoutput is used if not specified--mail-user send email to user --mail-type send email per job event: BEGIN, END, FAIL, ALL --dependency schedule a job to start after a previous job successfully completes Texas A&M UniversityHigh Performance Research Computingcommandhprc.tamu.edu11

Single-Node JobsTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu12

Single-Node Single-Core Job Scriptsspecify number ofthreads to matchSBATCHparametersExample 11Total CPUs requested11CPUs per node12total requested GB memory41SUs to start job2Texas A&M UniversityHigh Performance Research ComputingExample 2hprc.tamu.edu13

Slurm Parameter: --ntasksWhen only --ntasks is used, the --ntasks-per-node value will be automatically set to match--ntasks and defaults to --cpus-per-task 1 and --nodes 1 --ntasks 1 NumNodes 1 NumCPUs 1 NumTasks 1 CPUs/Task 1--ntasks 28 NumNodes 1 NumCPUs 28 NumTasks 28 CPUs/Task 1Texas A&M UniversityHigh Performance Research Computinghprc.tamu.edu14

Requesting all CPUs and Available Memoryon Terra Compute Nodes28 cores, 64 GB nodes (256 nodes available)68 cores, 96 GB KNL nodes (8 nodes available)Texas A&M University28 cores, 128 GB memory GPU nodes (48 nodes avail)72 cores, 96 GB memory nodes (8 nodes available)High Performance Research Computinghprc.tamu.edu15

Single-Node Multi-Core Job Scripts14Total CPUs requested2814CPUs per node2856total requested GB memory56672SUs to start job672Texas A&M UniversityHigh Performance Research Computingspecifynumber ofthreads tomatchSBATCHparametersIt is best torequest allcores andall memoryif using theentire nodehprc.tamu.edu16

GPU Jobs gpu partition (--partition gpu) will be automatically applied when using --gres gpu:n28 SUs will be charged for using GPU nodes whether you specify --ntasks 1 or --ntasks 28Texas A&M UniversityHigh Performance Research Computinghprc.tamu.edu17

Multi-Node JobsTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu18

Slurm Parameters: --nodes --ntasks-per-nodeIt is easier to scale jobs by using --nodes with --ntasks-per-node instead of with --ntasks.If you use --nodes with --ntasks, you need to calculate total CPUs for all nodes as the --ntasks value --nodes 2 --ntasks-per-node 28 NumNodes 2-2 NumCPUs 56 NumTasks 56 CPUs/Task 1 mem 112G --nodes 1 --ntasks 28 NumNodes 1 NumCPUs 28 NumTasks 28 CPUs/Task 1 mem 56G --nodes 2 --ntasks 56 NumNodes 2 NumCPUs 56 NumTasks 56 CPUs/Task 1 mem 112G when --nodes is 1, avoid using --ntasks: --nodes 2 --ntasks 28 will allocate 1 core on one node and 27 cores on a second nodeTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu19

MPI Multi-Node Multi-Core Job Script: Example 1280Total CPUs requested28CPUs per node56total requested GB memory per node6720SUs to start job (nodes * cpus * hours)Texas A&M UniversityHigh Performance Research Computinghprc.tamu.edu20

MPI Multi-Node Multi-Core Job Script: Example 2280Total CPUs requested28CPUs per node56total requested GB memory per node6720SUs to start job (nodes * cpus * hours)Texas A&M UniversityHigh Performance Research Computinghprc.tamu.edu21

TAMULauncher hprc.tamu.edu/wiki/SW:tamulauncherUse when you have hundreds or thousands of commands to run each utilizing asingle-core or a few cores tamulauncher keeps track of which commands completed successfully to see the list of completed commands if time runs out, then tamulauncher can be restarted and it will knowwhich was the last successfully completed command submit tamulauncher as a batch job within your job script can run tamulauncher interactively on login node; limited to 8 cores you can check the --status on the command line from the working directoryrun a single command of your thousands to make sure the command is correctand to get an estimate of resource usage (CPUs, memory, time)request all cores and memory on the compute node(s) and configure yourcommands to use all available coresTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu22

TAMULauncher Multi-Node Single-Core Commandscommands.txt(300 lines for example)run spades tamulauncher.shrun 28 spades.pycommands pernode with eachcommand using1 core.Requesting all 28cores reservesentire node foryour job run 28 single-core commands per node; useful when each command requires 2GB memorycreate a commands file (named whatever you want) to go with the the job scriptload the software module in the job script not the commands fileTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu23

TAMULauncher Multi-Node Multi-Core Commandscommands.txt(300 lines for example)run spades tamulauncher.shrun 7 spades.pycommands pernode with eachcommand using4 cores.Requesting all 28cores reservesentire node foryour job useful when each command requires more than 2GB but less than all available memoryuse OMP NUM THREADS if needed when running fewer commands than requested cores add on the line before the tamulauncher command Texas A&M UniversityHigh Performance Research Computinghprc.tamu.edu24

Making a TAMULauncher Commands FilePart 1Input files are two files per sample and named:Run this command to create the example files:s1 R1.fastq.gzs1 R2.fastq.gzRun the following commands to get familiar with useful shell commands for creating and manipulating variables(Texas A&M University)High Performance Research Computinghprc.tamu.edu25

Making a TAMULauncher Commands FilePart 2Input files are two files per sample and named:s1 R1.fastq.gzs1 R2.fastq.gzRun the following commands to loop through all R1 files in the reads directory and create the commands.txtUse just the R1 files because we only need to capture the sample names once.Match as muchas possible toavoid matchingsample namesTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu26

Other Useful Unix Commands}# will drop beginning of variable value up to first occurrence of ''# will drop beginning of variable value up to last occurrence of ''# will drop part of variable value from last occurrence of '' to the end# will drop part of variable value from first occurrence of '' to the endThese are useful if the part of the filename for each sample that needs to be removed is not the same.s1 S1 R1.fastq.gzs2 S2 R1.fastq.gzs3 S3 R1.fastq.gzwant to remove this partfrom each file nameMake a new directory and create a new set of files for this exercise.Texas A&M UniversityHigh Performance Research Computinghprc.tamu.edu27

Slurm Job Array Parameters andRuntime Environment Variables Array jobs are good to use when you have multiple samples each of which can utilize an entire computenode running software that supports multiple threads but does not support MPI Use the index value to select which commands to run either from a text file of commands or as part ofthe input file name or parameter value is the array index valuestdout and stderr files can be saved per index value Limit the number of simultaneously running tasks can help prevent reaching file and disk quotas due to many intermediate and temporary files as one job completes another array index is run on the available node Texas A&M UniversityHigh Performance Research Computinghprc.tamu.edu28

Slurm Job Array Example 1command (sed -n {SLURM ARRAY TASK ID}p commands.txt) command The sed command will print a specified line number from commands.txt based on theThe number of lines in your commands.txt file should be the same as the number of array indexesCan use %5 to limit the array to a maximum of 5 nodes used simultaneously but you need enough SUs to cover entirenumber of array indexes in order to submit the job. May be useful to prevent creating too many temporary files.There are other ways to use SLURM ARRAY TASK ID but this example is useful because it has a file of allcommands used in each array indexDoesn't work well when commands have redirection operators: Texas A&M UniversityHigh Performance Research Computinghprc.tamu.edu29

Slurm Job Array Example 2 Can useif your commands only differ by a number in the file namesThevariable will be assigned the array index from 1 to 40 in this exampleCan use this approach when commands have redirection operators: Texas A&M UniversityHigh Performance Research Computinghprc.tamu.edu30

Useful Slurm Runtime Environment Variables TMPDIR this is a temporary local disk space ( 848GB) created at runtime and is deleted whenthe job completes the directory is mounted on the compute node and files created in TMPDIR do notcount against your file and disk quotas SLURM CPUS PER TASK returns how many CPU cores were allocated on this node can be used in your command to match requested #SBATCH cpus SLURM ARRAY TASK ID can be used to select or run one of many commands when using a job array SLURM JOB NAME populated by the --job-name parameter SLURM JOB NODELIST can be used to get the list of nodes assigned at runtime SLURM JOBID can be used to capture JOBID at runtimeTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu31

Useful Unix Environment Variables Typeto see all Unix environment variables for your login session USER This will be automatically populated with your NetID SCRATCH You can use this to change to your /scratch/user/netid directory OMP NUM THREADS useful when software uses OpenMP for multithreading; default is 1 PWD contains the full path of the current working directoryTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu32

Monitoring JobResource UsageTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu33

Submit a Slurm Job Submit a job See status and JOBID of all your submitted jobs Cancel (kill) a queued or running job using JOBID Get an estimate of when your pending job will start. It is just an estimate based on all currently scheduled jobsrunning to the maximum specified runtime. Can be useful for queues: gpu, vnc, xlong, special, knl It will usually start before the estimated time. Texas A&M UniversityHigh Performance Research Computinghprc.tamu.edu34

Monitor a Running Job See status and JOBID of all your submitted jobs See summary of a running job (list node utilization) For the CPUS values: Active (in use by running jobs) Idle (available for jobs) Offline (unavailable for jobs) TotalSee CPU and memory usage of all your running jobs stats for pestat are updated every 3 minutes can use with watch command to run pestat every 2 seconds Low CPU load utilization highlighted in RedGood CPU load utilization highlighted in PurpleIdeal CPU load utilization displayed in White(Freemem should also be noted)Texas A&M UniversityHigh Performance Research Computinghprc.tamu.edu35

Monitor a Running Job Array List node utilization of a running job CPU LOAD of 28.00 means 100% node utilization all 28 of 28 total cores are at 100% utilization For the CPUS values: Active (in use by running jobs) Idle (available for jobs) Offline (unavailable for jobs) Total this appears to be two jobs because it is a job array Texas A&M UniversityHigh Performance Research Computinghprc.tamu.edu36

Monitor a Running Job See lots of info about your running orrecently completed ( 10 minutes) job gpu jobs are charged 28 SUs per jobregardless of whether 1 or all CPUs areselectedcan add this command at the end of yourjob script to capture job info into thestdout file Texas A&M UniversityHigh Performance Research Computinghprc.tamu.edu37

See Completed Job Efficiency Stats CPU load was at 100%for 88% of the run timemax memory utilizedwas 90% of requestedmemoryTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu38

See Completed Job Stats 36.5 GB max memory used There are three lines for this job: 1 job and 2 steps Texas A&M UniversityHigh Performance Research Computingyou can set an alias in your.bashrc for thesacct --format commandhprc.tamu.edu39

See All Your Terra Job for Current Fiscal Year Texas A&M meStartTimeEndTimeWalltimeTotalSlotsUsedSUsTotal JobsTotal Usage (SUs)High Performance Research Computinghprc.tamu.edu40

Debugging Slurm Jobs If job was not scheduled, check your HPRC account to see if you have enough SUs Look for an out of memory error message; could occur in only one index of a job array Make the necessary adjustments to SBATCH memory parameters in your jobscript and resubmit the jobIf you see an 'Out of disk space' error check your file and disk quotas using the showquota command reduce the number of files you have generated delete any unnecessary or temporary files usein your command if software supports a temporary directory create a .tar.gz package of completed projects to free up disk space request an increase in file and/or disk quota for your projectTexas A&M UniversityHigh Performance Research Computinghprc.tamu.edu41

portal.hprc.tamu.eduNetIDTexas A&M UniversityThe HPRC portal allows users to do the following Browse files on the The Terra filesystem Access the Terra Unix command line no SUs charged for using command line runs on login node; limit your processes to 8 cores Launch jobs SUs charged when launching jobs Compose job scripts Launch interactive GUI apps (SUs charged) Monitor and stop running jobs and interactive sessions Terra ANSYS Workbench Abaqus/CAE JBrowse, IGV LS-PREPOST MATLAB ParaView VNC ImageJ and other imaging sw Jupyter Notebook, JupyterLab BEAUti, Tracer, FigTree RStudioHigh Performance Research Computinghprc.tamu.edu42

For More HPRC Telephone:(979) 845-0219Visit us in person: Henderson Hall, Room 114A(best to email or call in advance and make an appointment)Texas A&M UniversityHigh Performance Research Computinghprc.tamu.edu

Slurm Job Script Parameters Always include the first two lines exactly as they are. In rare cases, line two is not needed. Slurm job parameters begin with and can add comments afterwards as above Name the job script whatever you like run_program_project.sh my_job_script.job my_job_script.sbatch my_job_script.txt 8

Related Documents:

Silver Galaxy Try Galaxy at usegalaxy.org to see if it appropriate for your project Getting Access to HPRC Silver Galaxy Available to Texas A&M students, staff and faculty with a NetID and an HPRC account Apply for an HPRC account first https://hprc.tamu.edu/apply Then send an email request for a Silver Galaxy account

Texas A&M University High Performance Research Computing – https://hprc.tamu.edu Using Jupyter Notebook on HPRC Portal Using Jupyter Notebook-Basic Python Exercise 1 Create a new directory under pip_envs Create a new notebook inside it. Save the file. Print a message- Welcome to HPRC 16

Job Scheduler Terra: Slurm hundreds of more compute nodes 3. Texas A&M University High Performance Research Computing hprc.tamu.edu File Systems and User Directories View usage and quota limits using the command: Quota and file limit increases will only be considered for scratch and tiered directories .

Mechanical Engineering Office Building Suite 200 979.845.1252 mechanical-undergradprogram@tamu.edu Undergraduate Program Director Ms. Heather Lewis, hlewis@tamu.edu Academic Advisors Ms. Jennifer Desai, jdesai@tamu.edu Ms. Madison Heyman, mad97hey@tamu.edu Ms. Regina Muir, rmuir@tamu.edu Dr. Ashley Schmitt, schmittae@tamu.edu

Texas A&M University High Performance Research Computing - http://hprc.tamu.edu Introduction to Python Yang Liu Associate Research Scientist

Cost-conscious Etc. Pathway Once upon a time in the snow Arriving & Planning . Boutros Lab COVID Use Case. 1. 0 Slurm Azure HPC Cache. 1. 14,400-21,600 cores target 2. 1 year. IPH Regeneron. 1: 1 Slurm Azure HPC Cache 1. 5,000-10,000 cores target: MDL HPC 1: 0 Slurm

Structural engineering is the field of engineering particularly concerned with the design of load-bearing structures. The field crosses engineering disciplines, and structural engineering can . Barroso, Luciana 979-845-0290 lbarroso@civil.tamu.edu Birely, Anna 979-862-6603 abirely@civil.tamu.edu Bracci, Joe 979-845-3750 bracci@civil.tamu.edu

Production scheduling methods can be further classified as static scheduling and dynamic scheduling. Static operation scheduling is performed during the compilation of the application. Once an acceptable scheduling solution is found, it is deployed as part of the application image. In dynamic scheduling, a dedicated system component makes

Post-processing CESM model output using Python - HPRC Short Course - Spring 2020 3 Upcoming HPRC short courses Introduction to Entos - QM Simulation Software for PCs, HPC, and the Cloud - Instructor: Thomas Miller (Professor at Caltech and Entos co-founder) and Daniel Smith

Florida Linear Scheduling Program Florida Linear Scheduling Program (FLSP) v1.0 is a linear scheduling software developed by the University of Florida in 1999. The tool has two functions; the first function is scheduling a specific linear construction project by using the Linear Scheduling Method

Apr 10, 2014 · Operating System Concepts! 6.1! Silberschatz, Galvin and Gagne 2002 Chapter 5: CPU Scheduling! Basic Concepts! Scheduling Criteria ! Scheduling Algorithms! Multiple-Processor Scheduling! Real-Time Scheduling! Algorithm Evaluation!

Using Scheduling Center Chapter 1 Getting Started 1 Getting Started Overview Scheduling Center The Scheduling Center is a product used with Recruiting to handle automated and high volume scheduling of candidates. This add-on scheduling functionality allows users to schedule any number of screening functions for candidates, including but not .

Using Scheduling Center Chapter 1 Getting Started 1 Getting Started Overview Scheduling Center The Scheduling Center is a product used with Recruiting to handle automated and high volume scheduling of candidates. This add-on scheduling functionality allows users to schedule any number of screening functions for candidates, including but not .

Job Code Listing May 2022 Job Code Job Title Job Function SuccessFactors Function Job Family Salary Plan Grade FLSA Status Minimum Salary Midpoint Salary Maximum Salary. Job Code Listing May 2022 Job Code Job Title Job Function SuccessFactors Function Job Family Salary Plan Grade FLSA Status Minimum Salary Midpoint Salary

Enhancing 3D spatial skills of engineering students using Augmented Reality Ali Sheharyar, Eyad Masad, Arun Srinivasa ali.sheharyar@qatar.tamu.edu, eyad.masad@qatar.tamu.edu, asrinivasa@tamu.edu Texas A&M University at Qatar, Doha, Qatar Abstract 3D spatial visualization and reasoning skills are of great importance for success in a number of

systems and shared-memory multi-processors, it is used in parallel job scheduling. A large number of studies have addressed the problem of parallel job scheduling (see [38] for a survey of work on this topic), but most of them address nonpreemptive scheduling strategies. Further, most of the work on preemptive scheduling of parallel jobs considers

In linear scheduling methods activities are defined as lines and the scheduling is done by drawing the lines with considering the buffers between the activities. In 1981 linear scheduling method(LSM) was first presented by Johnston [5]. Johnston for the first time used the “linear scheduling

A modied method has been evolved which uses Location Based Scheduling (LBS) in the form of ow line scheduling, which is a combination of CPM and Linear Scheduling Method (LSM) which can be used for planning and scheduling of small, medium and big projects. This study uses the modied LBS method

Scheduling at the Barn: Scheduling@Cornell ERP Guide for the Big Red Barn Purpose The purpose of this document is to help groups reserve space for events at the Big Red Barn using the Scheduling@Cornell reservation system. In order to reserve space at the Barn, your group

of branched rough paths introduced in (J. Differential Equations 248 (2010) 693–721). We first show that branched rough paths can equivalently be defined as γ-Hölder continuous paths in some Lie group, akin to geometric rough paths. We then show that every branched rough path can be encoded in a geometric rough path. More precisely, for every branched rough path Xlying above apathX .