POStER Speed Up Evaluation - Lex Jansen

3y ago
62 Views
2 Downloads
8.51 MB
18 Pages
Last View : 2d ago
Last Download : 2m ago
Upload by : Angela Sonnier
Transcription

POStERSpeed up evaluationby parallelization///////////November 2018Michael Weiss – Bayer AG

POStER - Speed up evaluation by parallelizationWhat is POStER?The BasicsWhy to do parallelization?How to parallelize SAS programs?What is scheduling?What are dependencies?On TopDoes the order matter?Can we mix it?Program InitializationWhat was done?2/// POStER – PhUSE – Michael Weiss – Bayer AG /// November 2018

POStERWhat is POStER?Parallel Optimized Statistical ExecutionRuntimeSAS based Macro SystemParallel execution of multiple SAS programs29 macros (3 user relevant) 4400 lines in 150KB%add job(programFile , logFile , outFile , iniProg default , termProg default , options default , interpreter default , weight )%sync jobs()Focus: easy to use3/// POStER – PhUSE – Michael Weiss – Bayer AG /// November 2018%run jobs(count 4)

POStER – The BasicsWhy to do parallelization?Runtime of programs affect timelinesReduce timelines requires reduced runtime (wall clock time)Common PCs have multiple CPUs / CoresServers often have 10 CPU CoresEven Workstations often have 4 to 8 coresSAS programs are by design linearOne PROC at a timeEither doing calculation or waiting for I/OA study evaluation contains multiple (mostly) independent programs4/// POStER – PhUSE – Michael Weiss – Bayer AG /// November 2018

POStER – The BasicsHow to parallelize SAS programs?Common practice is to use single “run-all” programwith %INCLUDES à serial executionpgm1.saspgm2.sas%INCLUDE "&prgdir./pgm1.sas";%INCLUDE "&prgdir./pgm2.sas";%INCLUDE "&prgdir./pgm3.sas";%INCLUDE "&prgdir./pgm4.sas";pgm3.saspgm4.sasPOStER provides similar API, but supports parallelexecution%add job(programFile &prgdir./pgm1.sas)%add job(programFile &prgdir./pgm2.sas)%add job(programFile &prgdir./pgm3.sas)%add job(programFile &prgdir./pgm4.sas)%run jobs(count 4)5/// POStER – PhUSE – Michael Weiss – Bayer AG /// November 2018pgm1.saspgm2.saspgm3.saspgm4.sas

POStER – The BasicsHow to parallelize SAS programs?SAS programs are linear (“single threaded”)Some procedures are multithreaded, but always execute just one “PROC” at a timeParallel execution can be performed by starting multiple SAS processes at the same timeStart SAS Process from SASMost Execution methods are meant to be used synchronizedSYSTEM Function or CALL RoutineX ' command '%SYSEXEC Macro StatementFILENAME . PIPE ' command 'Synchronized execution is still linear (“single threaded”)6/// POStER – PhUSE – Michael Weiss – Bayer AG /// November 2018

POStER – The BasicsHow to parallelize SAS programs?SYSTASK COMMAND NOWAIT allows asynchronous executionSYSTASK COMMAND "sas -sysin &prgdir./pgm1.sas" NOWAIT;SYSTASK COMMAND "sas -sysin &prgdir./pgm2.sas" NOWAIT;SYSTASK COMMAND "sas -sysin &prgdir./pgm3.sas" NOWAIT;SYSTASK COMMAND "sas -sysin &prgdir./pgm4.sas" NOWAIT;pgm1.sas7/// POStER – PhUSE – Michael Weiss – Bayer AG /// November 2018pgm2.saspgm3.saspgm4.sas

POStER – The BasicsWhat is scheduling?It is good to execute 4 or 8 programs at a time, but not 40, 80 or even .saspgm13.sasBetter would be a configurable number of programs to run in parallel at the same 3.saspgm5.saspgm6.saspgm10.sas pgm8.saspgm12.sas pgm40.sas pgm4.saspgm9.sas à This is scheduling8/// POStER – PhUSE – Michael Weiss – Bayer AG /// November 2018pgm40.sas

POStER – The BasicsWhat is scheduling?SAS Command WAITFOR à Either wait for ALL or wait for ANY processSYSTASK COMMAND "sas -sysin &prgdir./pgm1.sas" NOWAIT TASKNAME n1;SYSTASK COMMAND "sas -sysin &prgdir./pgm2.sas" NOWAIT TASKNAME n2;SYSTASK COMMAND "sas -sysin &prgdir./pgm3.sas" NOWAIT TASKNAME n3;SYSTASK COMMAND "sas -sysin &prgdir./pgm4.sas" NOWAIT TASKNAME n4;WAITFOR ANY n1 n2 n3 n4;SYSTASK COMMAND "sas -sysin &prgdir./pgm5.sas" NOWAIT TASKNAME n5;WAITFOR ANY &running tasks;SYSTASK COMMAND "sas -sysin &prgdir./pgm6.sas" NOWAIT TASKNAME n6;WAITFOR ALL &running FOR ALL9/// POStER – PhUSE – Michael Weiss – Bayer AG /// November 2018pgm4.sas

POStER – The BasicsWhat are dependencies?Not all programs are independent of each otherPrograms creating ADaM should be finished before programs start that use these data sets (TLF)!Some programs require one ADaM data set to generate an other ADaM data set à These are dependenciesPOStER implements synchronization request – WAITFOR ALL%add job(programFile &prgdir./pgm1.sas)%add job(programFile &prgdir./pgm2.sas)%add job(programFile &prgdir./pgm3.sas)%add job(programFile &prgdir./pgm4.sas)%sync jobs()%add job(programFile &prgdir./pgm5.sas)%add job(programFile &prgdir./pgm6.sas)%run jobs(count 4)10/// POStER – PhUSE – Michael Weiss – Bayer AG /// November 2018pgm1.saspgm2.saspgm3.sasWAITFOR ALLpgm5.saspgm6.sasWAITFOR ALLpgm4.sas

POStER - Speed up evaluation by parallelizationWhat is POStER?The BasicsWhy to do parallelization?How to parallelize SAS programs?What is scheduling?What are dependencies?On TopDoes the order matter?Can we mix it?Program InitializationWhat was done?11/// POStER – PhUSE – Michael Weiss – Bayer AG /// November 2018

POStER – On TopDoes the order matter?Does the order of execution matter?pgm1.saspgm2.saspgm1.sasAssume 10 programs to be run in 2 3.saspgm7.saspgm8.sas9 with 5min runtime each1 with 45min m7.sasPOStER allows manual ordering and automatedre-ordering on re-executionpgm8.saspgm10.sas65 min144%12/// POStER – PhUSE – Michael Weiss – Bayer AG /// November 2018pgm9.sas45 min100%

POStER – On TopCan we mix it?%add job(programFile &prgdir./pgm1.sas, interpreter SAS9.2)%add job(programFile &prgdir./pgm2.R, interpreter R3.1)%add job(programFile &prgdir./pgm3.sas, interpreter SAS9.2)%add job(programFile &prgdir./pgm4.sas, interpreter SAS9.4)%sync jobs()%add job(programFile &prgdir./pgm5.R, interpreter R3.1)%add job(programFile &prgdir./pgm6.sas, interpreter SAS9.4)%run jobs(count 4)13/// POStER – PhUSE – Michael Weiss – Bayer AG /// November 2018pgm1.saspgm2.Rpgm3.saspgm4.sasWAITFOR ALLpgm5.Rpgm6.sasWAITFOR ALL

POStER – On TopProgram InitializationPOStER supports INITSTMT and TERMSTMT SAS Options through separate files%add job(programFile &prgdir./pgm1.sas, iniPROG &prgdir./init.sas, termPROG &prgdir./term.sas)%add job(programFile &prgdir./pgm2.sas, iniPROG &prgdir./init.sas, termPROG &prgdir./term.sas)%add job(programFile &prgdir./pgm3.sas, iniPROG &prgdir./init2.sas, termPROG &prgdir./term2.sas)%add job(programFile &prgdir./pgm4.sas, iniPROG &prgdir./init2.sas, termPROG &prgdir./term2.sas)%run jobs(count 4)14/// POStER – PhUSE – Michael Weiss – Bayer AG /// November .sasWAITFOR ALL

POStER – On TopProgram InitializationPOStER supports INITSTMT and TERMSTMT SAS Options through separate files%LET POSTER PARAM INIPROG &prgdir./init.sas;%LET POSTER PARAM TERMPROG &prgdir./term.sas;%add job(programFile &prgdir./pgm1.sas)%add job(programFile &prgdir./pgm2.sas)%LET POSTER PARAM INIPROG &prgdir./init2.sas;%LET POSTER PARAM TERMPROG &prgdir./term2.sas;%add job(programFile &prgdir./pgm3.sas)%add job(programFile &prgdir./pgm4.sas)%run jobs(count 4)15/// POStER – PhUSE – Michael Weiss – Bayer AG /// November .sasWAITFOR ALL

POStER – On TopWhat was done?Automated tracing – Depending on Operating SystemSAS internal à RTRACE:RTRACE ALL RTRACELOC "/path/to/trace.file"Linux / HP-UXOS commands strace / ptrace / tusc:strace -D -f -e trace open,unlink,rename,stat -o "/path/to/trace.file"16/// POStER – PhUSE – Michael Weiss – Bayer AG /// November 2018

POStER – On TopWhat was done?### STARTED:2018-08-13T09:32:04### FINISHED:2018-08-13T09:33:27### JOB-STATUS:0### gms/t adce.sas### ogs/t adce fas.log### esults/t adce fas.lst### ev/pgms/ini 14 shell fas.sas### dev/pgms/term 14.sas### INTERPRETER:sas9.4### JOB-OPTIONS:### csr/dev/pgms/t adce.sasW t adce prod/macros/prepare ster1/dev/macros/prepare ev/pgms/ini 14 shell dev/pgms/term 14.sasW ts/t adce fas.rtf17/// POStER – PhUSE – Michael Weiss – Bayer AG /// November 2018

POStERSpeed up evaluation byparallelization///////////November 2018Michael Weiss – Bayer AGTHANKYOU

POStERallows manual ordering and automated re-ordering on re-execution pgm1.sas pgm2.sas pgm3.sas pgm4.sas pgm5.sas pgm6.sas pgm7.sas pgm8.sas pgm9.sas pgm10.sas pgm1.sas pgm2.sas pgm3.sas pgm4.sas pgm5.sas pgm6.sas pgm7.sas pgm8.sas pgm9.sas pgm10.sas 65 min 45 min 144% 100%

Related Documents:

Lex La-Ray Contact Info Health Science Annex (HSA) 817 S. Bus. Hwy 13, Lexington, MO 64067 660-259-2688 (phone) 660-259-2858 (fax) Monday - Friday 7:30 am -4:00 pm Lex La-Ray Technical Center 2323 High School Drive, Lexington, MO 64067 660-259-2264 Monday - Friday 7:30 am -4:00 pm

Ask your class what the poster’s message is. Who might have created the poster, and when? Ask your students to record any facts they can find about the poster. Be sure they understand the difference between facts (e.g., the poster’s date) and inferences (e.g., that the poster’s purpose was to build acceptance for an integrated military).

The Poster talk - technical (1) The poster talk has two parts: the presentation and the discussion. Consider prompting some questions about things you cannot cover in the time of the talk (4) The poster talk can be using classical poster format (poster board with pictures) or be a PPT. If you use power point it is your duty to make sure it runs.

campaign. The jury awards gold to the poster which uses simple methods to get to the heart of an important concept. Swiss Poster Award Poster of the year Poster of the year Swiss Poster Award 2 012 5 Jugendsession 2012 «Jetzt reden wir!» Client: SAJV Jugendsession, Bern Agency: Spillmann/Felser/Leo Burnett AG, Zürich Design: Niels Schefer

General Guidance For Poster Presentations Learning Development Service . 2 . 1. Oral Presentations and Introduction to Pow-erPoint: 2. Poster Presentations. . If, after consideration, there is too much text on a poster, there are a few points that could be con-sidered in order to trim it down. 1. The text on a poster should tell a story of .

In your poster presentations you are limited to a poster that is 36 inches high and 48 inches wideposter that is 36 inches high and 48 inches wide (Display Boards come in a folded 36 inch by 48 inch board). How to make a poster The purpose of poster presentations is not to have boards upon boards of information.

Scientific Poster Design How to keep your poster from resembling an "abstract painting" . research onto my poster? Your poster is a short story Describe a few major points . 2 main elements . 1) Simple,

VariQuest Poster Maker 3600 1-1 Getting Started Thank you for purchasing the VariQuest Poster Maker 3600! To get the most from your Poster Maker, please read this user's guide thoroughly and follow all instructions carefully. Keep this guide near the Poster Maker so that it is available for reference.