A Cuda Implementation Of The Hpcg Benchmark - Nvidia
A CUDA IMPLEMENTATION OF THEHPCG BENCHMARKEverett PhillipsMassimiliano Fatica
OUTLINEHigh Performance Conjugate Gradient BenchmarkMotivationOverviewOptimizationPerformance ResultsSingle GPUGPU SupercomputersConclusion
WHY HPCG ?HPL (Linpack) Top500 benchmarkSupercomputer Ranking / EvaluationDense Linear Algebra (Ax b)Compute intensiveDGEMM (Matrix-Matrix Multiply)O(N3)FLOPS / O(N2) Data10-100 Flop/ByteWorkload does not correlate with many modern applications
WHY HPCG?New Benchmark toSupplement HPLCommon ComputationPatterns not addressed byHPLNumerical Solution ofPDEsMemory IntensiveNetwork
HPCG BENCHMARKPreconditioned Conjugate Gradient AlgorithmSparse Linear Algebra (Ax b), Iterative solverBandwidth Intensive: 1/6 Flop/ByteSimple Problem (sparsity pattern of Matrix A)Simplifies matrix generation/solution validationRegular 3D grid, 27-point stencilNx x Ny x Nz local domain / Px x Py x Pz ProcessorsCommunications: boundary global reduction
HPCG ALGORITHMMulti-Grid PreconditionerSymmetric-Gauss-Seidel Smoother (SYMGS)Sparse Matrix Vector Multiply (SPMV)Dot Product – MPI Allreduce()
HPCG BENCHMARKProblem Setup – initialize data structuresOptimization (required to expose parallelism in SYMGS smoother)Matrix analysis / reordering / data layoutTime counted against final performance resultReference Run – 50 iterations with reference code – Record ResidualOptimized Run – converge to Reference ResidualMatrix re-ordering slows convergence (55-60 iterations)Additional iterations counted against final performance resultRepeat to fill target execution time (few minutes typical, 1 hour for official run )
HPCGSPMV (y Ax)Exchange Halo(x) //neighbor communicationsfor row 0 to nrowssum 0for j 0 to nonzeros in row[ row ]col A col[ j ]val A val[ j ]sum sum val * x[ col ]y[ row ] sumNo dependencies between rows, safe to process rows in parallel
HPCGSYMGS (Ax y, smooth x)Exchange Halo(x) //neighbor communicationsfor row 0 to nrows (Fwd Sweep, then Backward Sweep for row nrows to 0)sum b[ row ]for j 0 to nonzeros in row[ row ]col A col[ j ]val A val[ j ]if( col ! row )sum sum – val * x[ col ]x[ row ] sum / A diag[ row ]if col row, must wait for x[col] to be updated
MATRIX REORDERING (COLORING)SYMGS - order requirementPrevious rows must have new valuereorder by color (independent rows)2D example: 5-point stencil - red-black3D 27-point stencil 8 colors
MATRIX REORDERING (COLORING)Coloring to extract parallelismAssignment of “color” (integer) to vertices (rows), with no twoadjacent vertices the same color“Efficient Graph Matching and Coloring on the GPU” – (Jon Cohen)Luby / Jones-Plassman based algorithmCompare hash of row index with neighborsAssign color if local extremaOptional: recolor to reduce # of colors
MORE OPTIMIZATIONSOverlap Computation with neighbor communicationOverlap 1/3 MPI Allreduce with ComputationLDG loads for irregular access patterns (SPMV SYMGS)
OPTIMIZATIONSSPMV Overlap Computation with communicationsGather to GPU send bufferCopy send buffer to CPUMPI send / MPI recvCopy recv buffer to GPULaunch SPMV KernelGPUCPUTime
OPTIMIZATIONSSPMV Overlap Computation with communicationsGather to GPU send bufferCopy send buffer to CPULaunch SPMV interior KernelMPI send / MPI recvCopy recv buffer to GPULaunch SPMV boundary KernelGPU Stream AGPU Stream BCPUTime
RESULTS – SINGLE GPU
RESULTS – SINGLE GPU
RESULTS – SINGLE GPU
RESULTS – SINGLE GPU
RESULTS – GPU SUPERCOMPUTERSTitan @ ORNLCray XK7, 18688 Nodes16-core AMD Interlagos K20XGemini Network - 3D Torus TopologyPiz Daint @ CSCSCray XC30, 5272 Nodes8-core Xeon E5 K20XAries Network – Dragonfly Topology
RESULTS – GPU SUPERCOMPUTERS1 GPU 20.8 GFLOPS (ECC ON) 7% iteration overhead at scaleTitan @ ORNL322 TFLOPS (18648 K20X)89% efficiency (17.3 GF per GPU)Piz Daint @ CSCS97 TFLOPS (5265 K20X)97% efficiency (19.0 GF per GPU)
RESULTS – GPU SUPERCOMPUTERSDDOT (-10%)MPI Allreduce()Scales as Log(#nodes)MG (-2%)Exchange Halo (neighbor)SPMV (-0%)Overlapped w/Compute
SUPERCOMPUTER COMPARISON
CONCLUSIONSGPUs proven effective for HPL, especially for power efficiencyHigh flop rateGPUs also very effective for HPCGHigh memory bandwidthStacked memory will give a huge boostFuture work will add CPU GPU
ACKNOWLEDGMENTSOak Ridge Leadership Computing Facility (ORNL)Buddy Bland, Jack Wells and Don MaxwellSwiss National Supercomputing Center (CSCS)Gilles Fourestey and Thomas SchulthessNVIDIALung Scheng Chien and Jonathan Cohen
High Performance Conjugate Gradient Benchmark . . 10-100 Flop/Byte Workload does not correlate with many modern applications HPL (Linpack) Top500 benchmark . WHY HPCG? New Benchmark to Supplement HPL Common Computation Patterns not addressed by . GPUs proven effective for HPL, especially for power efficiency High flop rate GPUs also very .
May 02, 2018 · D. Program Evaluation ͟The organization has provided a description of the framework for how each program will be evaluated. The framework should include all the elements below: ͟The evaluation methods are cost-effective for the organization ͟Quantitative and qualitative data is being collected (at Basics tier, data collection must have begun)
Silat is a combative art of self-defense and survival rooted from Matay archipelago. It was traced at thé early of Langkasuka Kingdom (2nd century CE) till thé reign of Melaka (Malaysia) Sultanate era (13th century). Silat has now evolved to become part of social culture and tradition with thé appearance of a fine physical and spiritual .
On an exceptional basis, Member States may request UNESCO to provide thé candidates with access to thé platform so they can complète thé form by themselves. Thèse requests must be addressed to esd rize unesco. or by 15 A ril 2021 UNESCO will provide thé nomineewith accessto thé platform via their émail address.
̶The leading indicator of employee engagement is based on the quality of the relationship between employee and supervisor Empower your managers! ̶Help them understand the impact on the organization ̶Share important changes, plan options, tasks, and deadlines ̶Provide key messages and talking points ̶Prepare them to answer employee questions
Dr. Sunita Bharatwal** Dr. Pawan Garga*** Abstract Customer satisfaction is derived from thè functionalities and values, a product or Service can provide. The current study aims to segregate thè dimensions of ordine Service quality and gather insights on its impact on web shopping. The trends of purchases have
CUDA-GDB runs on Linux and Mac OS X, 32-bit and 64-bit. CUDA-GDB is based on GDB 7.6 on both Linux and Mac OS X. 1.2. Supported Features CUDA-GDB is designed to present the user with a seamless debugging environment that allows simultaneous debugging of both GPU and CPU code within the same application.
www.nvidia.com CUDA Debugger DU-05227-042 _v5.5 3 Chapter 2. RELEASE NOTES 5.5 Release Kernel Launch Stack Two new commands, info cuda launch stack and info cuda launch children, are introduced to display the kernel launch stack and the children k
CUDA Toolkit Major Components www.nvidia.com NVIDIA CUDA Toolkit 10.0.153 RN-06722-001 _v10.0 2 ‣ cudadevrt (CUDA Device Runtime) ‣ cudart (CUDA Runtime) ‣ cufft (Fast Fourier Transform [FFT]) ‣ cupti (Profiling Tools Interface) ‣ curand (Random Number Generation) ‣ cusolver (Dense and Sparse Direct Linear Solvers and Eigen Solvers) ‣ cusparse (Sparse Matrix)
