Parallel And Distributed Computing With MATLAB - EAFIT
Parallel and Distributed Computing with MATLABGerardo HernándezManager, Application Engineer 2018 The MathWorks, Inc.1
Practical Application of Parallel Computing Why parallel computing? Need faster insight on more complex problems with larger datasets Computing infrastructure is broadly available (multicore desktops, GPUs, clusters) Why parallel computing with MATLAB Leverage computational power of more hardware Accelerate workflows with minimal to no code changes to your original code Focus on your engineering and research, not the computation2
Steps for Improving Performance First get code working Speed up code with core MATLAB Include compiled languages and additional hardwareWebinar: Optimizing and Accelerating Your MATLAB Code3
Programming Parallel Applications Built-in multithreading– Automatically enabled in MATLAB since R2008a– Multiple threads in a single MATLAB computation engine Parallel computing using explicit techniques– Multiple computation engines controlled by a single session– High-level constructs to let you parallelize MATLAB applications– Perform MATLAB computations on GPUs4
Parallel rWorkerWorker5
Agenda Utilizing multiple cores on a desktop computer Scaling up to cluster and cloud resources Tackling data-intensive problems on desktops and clusters Accelerating applications with NVIDIA GPUs Summary and resources6
Programming Parallel ApplicationsBuilt in supportGreater Control– ., 'UseParallel', true)Ease of Use 7
Demo: Cell Phone Tower OptimizationUsing Parallel-Enabled Functions Parallel-enabled functions in Optimization Toolbox Set flags to run optimization in parallel Use pool of MATLAB workers to enableparallelism8
Predictive Maintenance of Turbofan EngineSensor data from 100 engines of the same modelScenario: No data from failures Performing scheduled maintenance No failures have occurred Maintenance crews tell us most engines could run forlonger Can we be smarter about how to schedulemaintenance without knowing what failure lookslike?Data provided by NASA ic-data-repository/9
Parallel-enabled Toolboxes (MATLAB Product Family)Enable parallel computing support by setting a flag or preferenceImage ProcessingBatch Image Processor, BlockProcessing, GPU-enabled functionsSignal Processing andCommunicationsGPU-enabled FFT filtering,cross correlation, BERsimulationsStatistics and Machine LearningResampling Methods, k-Meansclustering, GPU-enabled functionsComputer VisionParallel-enabled functionsin bag-of-words workflowDeep LearningDeep Learning, Neural Networktraining and simulationOptimizationParallel estimation ofgradientsOther parallel-enabled Toolboxes10
Programming Parallel Applications Built in supportSimple programming constructs– parfor, batchEase of Use Greater Control– ., 'UseParallel', true)12
Embarrassingly Parallel: Independent Tasks or Iterations No dependencies or communications between tasks Examples: parameter sweeps, Monte Carlo simulationsWorkerMATLABDesktop (Client)WorkerWorkerTimeTime13
Example: Parameter Sweep of ODEsParallel for-loops Parameter sweep of ODE system– Damped spring oscillator 5m x b x k x 01, 2 ,.1, 2 ,.– Sweep through different valuesof damping and stiffness– Record peak value for eachsimulation Convert for to parfor Use pool of MATLAB workers14
Mechanics of parfor Loops11223 34 4 55 6678 8 9 910 10Workera(i) i;a zeros(10, 1)parfor i 1:10a(i) i;endaWorkera(i) i;Workera(i) i;Workera(i) i;16
Tips for Leveraging PARFOR Consider creating smaller arrays on each worker versus one large arrayprior to the parfor loop Take advantage of parallel.pool.Constant to establish variableson pool workers prior to the loop Encapsulate blocks as functions when neededUnderstanding parfor17
Programming Parallel Applications Built in supportSimple programming constructs– parfor, batch Full control of parallelizationEase of Use Greater Control– ., 'UseParallel', true)– spmd, parfeval18
Agenda Utilizing multiple cores on a desktop computer Scaling up to cluster and cloud resources Tackling data-intensive problems on desktops and clusters Accelerating applications with NVIDIA GPUs Summary and resources19
Take Advantage of Cluster Hardware Offload computation:Computer Cluster– Free up desktopCluster– Access better computers Scale speed-up:– Use more cores– Go from hours to minutes MATLABDesktop (Client)SchedulerScale memory:– Utilize tall arrays and distributed arrays– Solve larger problems without re-coding algorithms20
Offloading dulerWorkerWorkerWorker21
Offloading Serial Computations job batch(.);WorkMATLABDesktop (Client)WorkerWorkerWorkerResultWorker22
Example: Parameter Sweep of ODEsOffload and Scale Processing Offload processing to workers:batch 5m x b x k x 01, 2 ,.1, 2 ,.Scale offloaded processing:batch( ,'Pool', ) Retrieve results from job:fetchOutputs23
Offloading and Scaling Computations job batch(. , 'Pool', n);WorkMATLABDesktop (Client)WorkerWorkerWorkerResultWorker24
Migrate to Cluster / Cloud Use MATLAB Distributed Computing Server Change hardware without changing algorithm25
Scale your applications beyond the desktopOptionParallel ComputingToolboxMATLAB Parallel CloudMATLAB DistributedComputing Serverfor Amazon EC2MATLAB DistributedComputing Serverfor Custom CloudMATLAB DistributedComputing ServerDescriptionExplicit desktop scalingSingle-user, basic scalingto cloudScale to EC2 with somecustomizationScale to custom cloudScale to clustersMaximumworkersNo limit16256No limitNo limitHardwareDesktopMathWorks ComputeCloudAmazon EC2Amazon EC2,Microsoft Azure,OthersAnyAvailabilityWorldwideUnited States and CanadaUnited States, Canada and otherselect countries in EuropeWorldwideWorldwideLearn More: Parallel Computing on the Cloud26
Agenda Utilizing multiple cores on a desktop computer Scaling up to cluster and cloud resources Tackling data-intensive problems on desktops and clusters Accelerating applications with NVIDIA GPUs Summary and resources27
Tall and Distributed Data Tall Data – Columnar data that does not fit inmemory of a desktop or clusterDistributed Data– Large matrices using the combinedmemory of a clusterMachineMemory Common Actions– Data manipulation, math, statistics– Summary visualizations– Machine learning Common Actions– Matrix Manipulation– Linear Algebra and Signal Processing28
Tall Arrays New data type in MATLAB R2016b Applicable when:MachineMemory– Data is columnar – with many rows– Overall data size is too big to fit into memory– Operations are mathematical/statistical in nature Statistical and machine learning applications– Hundreds of functions supported in MATLAB andStatistics and Machine Learning ToolboxTall Data29
Predictive Maintenance of Turbofan EngineSensor data from 100 engines of the same modelScenario: No data from failures Performing scheduled maintenance No failures have occurred Maintenance crews tell us most engines could run forlonger Can we be smarter about how to schedulemaintenance without knowing what failure lookslike?Data provided by NASA ic-data-repository/30
Execution Environments for Tall ArraysRun on Compute Clusters,or Spark if your data isstored in HDFS, for largescale analysisLocal diskShared foldersDatabasesProcess out-of-memory data onyour Desktop to explore,analyze, gain insights and todevelop analyticsUse Parallel ComputingToolbox for increasedperformanceSpark Hadoop32
Distributed Arrays Distributed Arrays hold data remotely on workers running on a cluster Manipulate directly from client MATLAB (desktop) 200 MATLAB functions overloaded for distributed arraysSupported functions for Distributed Arrays33
Agenda Utilizing multiple cores on a desktop computer Scaling up to cluster and cloud resources Tackling data-intensive problems on desktops and clusters Accelerating applications with NVIDIA GPUs Summary and resources34
Graphics Processing Units (GPUs) For graphics acceleration and scientific computing Many parallel processors Dedicated high speed memory35
GPU Requirements Parallel Computing Toolbox requires NVIDIA GPUs www.nvidia.com/object/cuda gpus.htmlMATLAB ReleaseRequired Compute CapabilityMATLAB R2018a and later releases3.0 or greaterMATLAB R2014b – MATLAB R2017b2.0 or greaterMATLAB R2014a and earlier releases1.3 or greater36
Built in toolbox support Simple programming constructs– gpuArray, gatherGreater Control Ease of UseProgramming with GPUs37
Demo: Wave EquationAccelerating scientific computing in MATLAB with GPUs Objective: Solve 2nd order wave equation with spectral methods Approach:– Develop code for CPU– Modify the code to use GPUcomputing using gpuArray– Compare performance ofthe code using CPU and GPU38
Speed-up MATLAB code with NVIDIA GPUs Ideal Problems Massively Parallel and/or Vectorized operations Computationally Intensive 300 GPU-enabled MATLAB functions Enable existing MATLAB code to run on GPUs Support for sparse matrices on GPUs Additional GPU-enabled Toolboxes Deep Learning Image Processing Signal Processing. Learn More39
Built in toolbox support Simple programming constructs– gpuArray, gather Advanced programming constructs– spmd, arrayfun Greater Control Ease of UseProgramming with GPUsInterface for experts– CUDAKernel, mex40
Agenda Utilizing multiple cores on a desktop computer Scaling up to cluster and cloud resources Tackling data-intensive problems on desktops and clusters Accelerating applications with NVIDIA GPUs Summary and resources41
Summary Easily develop parallel MATLAB applications without being a parallelprogramming expert Speed up the execution of your MATLAB applications using additionalhardware Develop parallel applications on your desktop and easily scale to a clusterwhen needed42
Some Other Valuable Resources MATLAB Documentation– MATLAB Advanced Software Development Performance and Memory– Parallel Computing Toolbox Parallel and GPU Computing Tutorials– nd-gpu-computing-tutorials97719.html Parallel Computing on the Cloud with MATLAB– ng/parallel-computing-on-thecloud/43
MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See www.mathworks.com/trademarks for a list of additional trademarks. Otherproduct or brand names may be trademarks or registered trademarks of their respective holders. 2018 The MathWorks, Inc. 2018 The MathWorks, Inc.44
Practical Application of Parallel Computing Why parallel computing? Need faster insight on more complex problems with larger datasets Computing infrastructure is broadly available (multicore desktops, GPUs, clusters) Why parallel computing with MATLAB Leverage computational power of more hardware
Cloud Computing J.B.I.E.T Page 5 Computing Paradigm Distinctions . The high-technology community has argued for many years about the precise definitions of centralized computing, parallel computing, distributed computing, and cloud computing. In general, distributed computing is the opposite of centralized computing.
Parallel computing, distributed computing, java, ITU-PRP . 1 Introduction . ITU-PRP provides an all-in-one solution for Parallel Programmers, with a Parallel Programming Framework and a . JADE (Java Agent Development Framework) [6] as another specific Framework implemented on Java, provides a framework for Parallel Processing.
In distributed computing we have multiple autonomous computers which seems to the user as single system. In distributed systems there is no shared memory and computers communicate with each other through message passing. In distributed computing a single task is divided among different computers. Difference between Parallel Computing and .
Progress! "DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language", OSDI 2008 "SCOPE: Easy and efficient parallel processing of massive data sets", VLDB 2008 "Distributed Data-Parallel Computing Using a High- Level Programming Language", SIGMOD 2009
Parallel computing is a form of High Performance computing. By using the strength of many smaller computational units, parallel computing can pro-vide a massive speed boost for traditional algorithms.[3] There are multiple programming solutions that o er parallel computing. Traditionally, programs are written to be executed linearly. Languages
Parallel and Distributed Computing Chapter 1: Introduction to Parallel Computing Jun Zhang Laboratory for High Performance Computing & Computer Simulation Department of Computer Science University of Kentucky Lexington, KY 40506. Chapter 1: CS621 2 1.1a: von Neumann Architecture
Parallel Computing Toolbox Ordinary Di erential Equations Partial Di erential Equations Conclusion Lecture 8 Scienti c Computing: Symbolic Math, Parallel Computing, ODEs/PDEs Matthew J. Zahr CME 292 Advanced MATLAB for Scienti c Computing Stanford University 30th April 2015 CME 292: Advanced MATLAB for SC Lecture 8. Symbolic Math Toolbox .
2021 ULI AsiaPac Awards for Excellence P r o j e c t C a t e g o r y / S e c t o r Indicate the main pr oject categor y for your project. You can fur ther describe it in the following sections. Select P r o j e c t T y p e Describe your project type, e.g. new de velopment, refurbishment, redevelopment, repurposing, cultural/industrial heritage
