Parallel Manual - New Jersey Institute Of Technology
Compaq FortranParallel Processing Manual forTru64 UNIX SystemsJanuary 2002This manual provides information about the Compaq Fortran parallelprogram development and run-time environment on Compaq Tru64 UNIXsystems.Note: The NUMA parallel processing feature described in this manual isavailable in the Compaq Fortran software but is not supported.Revision/Update Information:This is a new manual.Software Version:Compaq Fortran Version 5.5 or higherfor Tru64 UNIX SystemsCompaq Computer CorporationHouston, Texas
First Printing, January 2002 2002 Compaq Information Technologies Group, L.P.Compaq, the Compaq logo, AlphaServer, and Tru64 are trademarks of Compaq InformationTechnologies Group, L.P. in the U.S. and/or other countries.UNIX is a trademark of The Open Group in the U.S. and/or other countries.All other product names mentioned herein may be trademarks of their respective companies.Confidential computer software. Valid license from Compaq required for possession, use orcopying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, ComputerSoftware Documentation, and Technical Data for Commercial Items are licensed to the U.S.Government under vendor’s standard commercial license.Compaq shall not be liable for technical or editorial errors or omissions contained herein. Theinformation is provided ‘‘as is’’ without warranty of any kind and is subject to change withoutnotice. The warranties for Compaq products are set forth in the express limited warrantystatements accompanying such products. Nothing herein should be construed as constituting anadditional warranty.This document is available on CD–ROM.This document was prepared using DECdocument, Version 3.3-1n.
ContentsPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xiii1 Compaq Fortran Parallel Processing: An 1.6.1Overview of Parallel Processing . . . . . . . . . . . . . . . . . . . . . . .Applying Amdahl’s Law To Determine Whether To Convert aSerial Program to a Parallel One . . . . . . . . . . . . . . . . . . . . . .Example of Selecting Serial or Parallel Processing . . . . . . . .Serial Example Program red black 10 . . . . . . . . . . . . . . . . . .Analysis of Serial Example Program red black 10 . . . . .Architectures That Can Implement Parallel Processing . . . . .Symmetric Multiprocessor (SMP) Architecture . . . . . . . .Non-Uniform Memory Access (NUMA) Architecture . . . .Distributed Memory Architecture . . . . . . . . . . . . . . . . . .OpenMP Directives on an SMP System: Parallel Programred black 20 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Explanation of Parallel Program red black 20 . . . . . . . 18.1–191–242 Data Layout: An Introduction2.12.22.2.12.2.2Overview of Data Layout Principles . . . . . . . . . . . . . . . . .User-Directed Data Migration . . . . . . . . . . . . . . . . . . . . .MIGRATE NEXT TOUCH Directive . . . . . . . . . . . . .MIGRATE NEXT TOUCH NOPRESERVE �123 NUMA Architecture: An Introduction3.13.1.13.23.2.1OpenMP and Page Migration Directives on a NUMA System:Parallel Program red black 30 . . . . . . . . . . . . . . . . . . . . . . . .Explanation of Parallel Program red black 30 . . . . . . . . .OpenMP and Data Layout Directives on a NUMA System:Parallel Program red black 40 . . . . . . . . . . . . . . . . . . . . . . . .Explanation of Parallel Program red black 40 . . . . . . . . .iii
3.33.43.4.13.4.23.5Restrictions on OpenMP Features With the !DEC OMP NUMADirective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Two Short but Complete Example Programs . . . . . . . . . . . . . . . .Program TWELVE BILLION A . . . . . . . . . . . . . . . . . . . . . . .Program TWELVE BILLION B . . . . . . . . . . . . . . . . . . . . . . .Specifying Memories and Threads per Memory . . . . . . . . . . . . . .3–133–153–153–163–164 High Performance Fortran (HPF) Software: An Introduction4.14.1.14.24.34.3.14.3.2HPF Directives on a Distributed Memory System: ParallelProgram red black 50 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Explanation of Parallel Program red black 50 . . . . . . .What is HPF? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Parallel Programming Models . . . . . . . . . . . . . . . . . . . . . . .Data Parallel Programming . . . . . . . . . . . . . . . . . . . . . .HPF and Data Parallelism . . . . . . . . . . . . . . . . . . . . . . 235–245 HPF 5.15.5.25.5.35.5.45.5.5ivHPF Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . .When to Use HPF . . . . . . . . . . . . . . . . . . . . .Existing Code . . . . . . . . . . . . . . . . . . . . . .New Code . . . . . . . . . . . . . . . . . . . . . . . . .HPF Directives . . . . . . . . . . . . . . . . . . . . . . . . . . .Minimum Requirements for Parallel Execution . .Data Parallel Array Operations . . . . . . . . . . . . . .Array Terminology . . . . . . . . . . . . . . . . . . . . .Fortran 90 Array Assignment . . . . . . . . . . . .Whole Array Assignment . . . . . . . . . . . . .Array Subsections . . . . . . . . . . . . . . . . . .FORALL . . . . . . . . . . . . . . . . . . . . . . . . . . . .INDEPENDENT Directive . . . . . . . . . . . . . . .Vector-Valued Subscripts . . . . . . . . . . . . . . . .Entity-Oriented Declaration Syntax . . . . . . . .SEQUENCE and NOSEQUENCE Directives .Out of Range Subscripts . . . . . . . . . . . . . . . .Data Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . .Data Mapping Basics . . . . . . . . . . . . . . . . . . .Illustrated Summary of HPF Data Mapping .ALIGN Directive . . . . . . . . . . . . . . . . . . . . . .TEMPLATE Directive . . . . . . . . . . . . . . . . . .PROCESSORS Directive . . . . . . . . . . . . . . . .
DISTRIBUTE Directive . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Explanation of the Distribution Figures . . . . . . . . . . . . . .BLOCK Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . .CYCLIC Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . .BLOCK, BLOCK Distribution . . . . . . . . . . . . . . . . . . . . .CYCLIC, CYCLIC Distribution . . . . . . . . . . . . . . . . . . . .CYCLIC, BLOCK Distribution . . . . . . . . . . . . . . . . . . . . .BLOCK, CYCLIC Distribution . . . . . . . . . . . . . . . . . . . . .Asterisk Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . .Visual Technique for Computing Two-DimensionalDistributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.5.6.10Using DISTRIBUTE Without an Explicit Template . . . . .5.5.6.11Using DISTRIBUTE Without an Explicit PROCESSORSDirective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.5.6.12Deciding on a Distribution . . . . . . . . . . . . . . . . . . . . . . . .5.5.7SHADOW Directive for Nearest-Neighbor Algorithms . . . . . .5.6Subprograms in HPF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.6.1Assumed-Size Array Specifications . . . . . . . . . . . . . . . . . . . .5.6.2Explicit Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.6.3Module Program Units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.6.4PURE Attribute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.6.5Transcriptive Distributions and the INHERIT Directive . . . .5.7Intrinsic and Library Procedures . . . . . . . . . . . . . . . . . . . . . . . . .5.7.1Intrinsic Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.7.2Library Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.8Extrinsic Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.8.1Programming Models and How They Are Specified . . . . . . . .5.8.2Who Can Call Whom . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.8.2.1Calling Non-HPF Subprograms fromEXTRINSIC(HPF LOCAL) Routines . . . . . . . . . . . . . . . .5.8.3Requirements on the Called EXTRINSIC Procedure . . . . . . .5.8.4Calling C Subprograms from HPF Programs . . . . . . . . . . . . 35–635–645–645–675–685–695–696 Compiling and Running HPF ProgramsCompiling HPF Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Compile-Time Options for High Performance FortranPrograms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6.1.1.1-hpf [nn] Option — Compile for Parallel Execution . . . .6.1.1.2-assume bigarrays Option — Assume Nearest-NeighborArrays are Large . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6.1.1.3-assume nozsize Option — Omit Zero-Sized ArrayChecking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6.16.1.1.6–1.6–26–2.6–3.6–3v
-fast Option — Set Options to Improve Run-TimePerformance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6.1.1.5-nearest neighbor [nn] and -nonearest neighbor Options— Nearest Neighbor Optimization . . . . . . . . . . . . . . . . . .6.1.1.6-nohpf main Option — Compiling Parallel Objects toLink with a Non-Parallel Main Program . . . . . . . . . . . . .6.1.1.7-show hpf—Show Parallelization Information . . . . . . . . .6.1.2Consistency of Number of Peers . . . . . . . . . . . . . . . . . . . . . . .6.2HPF Programs with MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6.2.1Overview of HPF and MPI . . . . . . . . . . . . . . . . . . . . . . . . . . .6.2.2Compiling HPF Programs for MPI . . . . . . . . . . . . . . . . . . . . .6.2.3Linking HPF Programs with MPI . . . . . . . . . . . . . . . . . . . . .6.2.4Running HPF Programs Linked with MPI . . . . . . . . . . . . . . .6.2.5Cleaning Up After Running HPF Programs Linked withMPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6.2.6Changing HPF Programs for MPI . . . . . . . . . . . . . . . . . . . . ��86–96–106–107 Optimizing HPF st Compile-Time Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Converting Fortran 77 Programs to HPF . . . . . . . . . . . . . . . . . . .Explicit Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Nonparallel Execution of Code and Data Mapping Removal . . . .Compile Speed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Nearest-Neighbor Optimization . . . . . . . . . . . . . . . . . . . . . . . . . .Compiling for a Specific Number of Processors . . . . . . . . . . . . . .Avoiding Unnecessary Communications Setup for Allocatable orPointer Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .USE Statements HPF LIBRARY andHPF LOCAL LIBRARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Forcing Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Input/Output in HPF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .General Guidelines for I/O . . . . . . . . . . . . . . . . . . . . . . . . . . .Specifying a Particular Processor as Peer 0 . . . . . . . . . . . . . .Printing Large Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Reading and Writing to Variables Stored Only on Peer 0 . . . .Use Array Assignment Syntax instead of Implied DO . . . . . .IOSTAT and I/O with Error Exits–Localizing to Peer 0 . . . . .Stack and Data Space Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . .-show hpf Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Timing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Spelling of the HPF Directives . . . . . . . . . . . . . . . . . . . . . . . . . . 57–157–167–17
A HPF Tutorials: IntroductionB HPF Tutorial: LU DecompositionUsing LU Decomposition to Solve a System of SimultaneousEquations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .B.2Coding the Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .B.2.1Fortran 77 Style Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .B.2.2Parallelizing the DO Loops . . . . . . . . . . . . . . . . . . . . . . . . . .B.2.3Comparison of Array Syntax, FORALL, and INDEPENDENTDO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .B.3Directives Needed for Parallel Execution . . . . . . . . . . . . . . . . . . .B.3.1DISTRIBUTE Directive . . . . . . . . . . . . . . . . . . . . . . . . . . . . .B.3.2Deciding on a Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . .B.3.3Distribution for LU Decomposition . . . . . . . . . . . . . . . . . . . .B.3.3.1Parallel Speed-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .B.4Packaging the Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5B–17B–18C HPF Tutorial: Solving Nearest-Neighbor l Heat Flow Problem . . . . . . . .Jacobi’s Method . . . . . . . . . . . . . . . . . . . . . . . . .Coding the Algorithm . . . . . . . . . . . . . . . . . . . . .Illustration of the Results . . . . . . . . . . . . . . . . .Distributing the Data for Parallel Performance .Deciding on a Distribution . . . . . . . . . . . . . .Optimization of Nearest-Neighbor ProblemsPackaging the Code . . . . . . . . . . . . . . . . . . . . . �2D–2D–3D–4D–5D–6D–7D HPF Tutorial: Visualizing the Mandelbrot SetD.1D.1.1D.1.2D.2D.2.1D.2.2D.2.3D.2.4What Is the Mandelbrot Set? . . . . . . . . . . .How Is the Mandelbrot Set Visualized?Electrostatic Potential of the Set . . . . .Mandelbrot Example Program . . . . . . . . . .Developing the Algorithm . . . . . . . . . .Computing the Entire Grid . . . . . . . . .Converting to HPF . . . . . . . . . . . . . . . .PURE Attribute . . . . . . . . . . . . . . . . . .vii
E HPF Tutorial: Simulating Network Striped FilesE.1E.1.1E.2E.3E.3.1E.4E.5Why Simulate Network Striped Files? . . . . . . . . . . . . . .Constructing a Module for Parallel Temporary FilesSubroutine parallel open . . . . . . . . . . . . . . . . . . . . . . . .Subroutine parallel write . . . . . . . . . . . . . . . . . . . . . . .Passing Data Through the Interface . . . . . . . . . . . .Subroutines parallel read, parallel close, andparallel rewind . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Module parallel temporary files . . . . . . . . . . . . . . . . . .E–1E–2E–3E–4E–4.E–5E–5Serial Program red black 10.f90 . . . . . . . . . . . . . . . . . . . . . .Parallel Program red black 20.f90, Using OpenMPDirectives on an SMP System . . . . . . . . . . . . . . . . . . . . . . . .Program red black 30.f90 . . . . . . . . . . . . . . . . . . . . . . . . . . .Parallel Program red black 40.f90 . . . . . . . . . . . . . . . . . . . . .Parallel Program red black 50.f90 . . . . . . . . . . . . . . . . . . . . .Code Fragment for Mapping Illustrations . . . . . . . . . . . . . . .Avoiding Communication Set-up with Allocatable Arrays . . .Iteration of the Function z 2 c . . . . . . . . . . . . . . . . . . . . . . .Using a DO Loop to Compute the Grid . . . . . . . . . . . . . . . . .Using a FORALL Structure to Compute the Grid . . . . . . . . .PURE Function escape time . . . . . . . . . . . . . . . . . . . . . . . . .Test Program for Parallel Temporary Files . . . . . . . . . . . . . ��6D–7E–6Amdahl’s Law: Potential Speedup of Serial ProgramsMetal Cube with Initial Temperatures . . . . . . . . . . . .Upper Left Portion of Metal Cube . . . . . . . . . . . . . . .A Typical SMP System . . . . . . . . . . . . . . . . . . . . . . . .A Typical NUMA System . . . . . . . . . . . . . . . . . . . . . .A Typical Distributed Memory System . . . . . . . . . . . .BLOCK Distribution — Array View . . . . . . . . . . . . . .BLOCK Distribution — Processor View . . . . . . . . . . 51–65–15–2viii.
LIC Distribution — Array View . . . . . . . . . . . . . .CYCLIC Distribution — Processor View . . . . . . . . . . .BLOCK, BLOCK Distribution — Array View . . . . . . . .BLOCK, BLOCK Distribution — Processor View . . . . .CYCLIC, CYCLIC Distribution — Array View . . . . . . .CYCLIC, CYCLIC Distribution — Processor View . . . .CYCLIC, BLOCK Distribution — Array View . . . . . . .CYCLIC, BLOCK Distribution — Processor View . . . .BLOCK, CYCLIC Distribution — Array View . . . . . . .BLOCK, CYCLIC Distribution — Processor View . . . .BLOCK,* Distribution — Array View . . . . . . . . . . . . .BLOCK, * Distribution — Processor View . . . . . . . . . .CYCLIC, * Distribution — Array View . . . . . . . . . . . .CYCLIC, * Distribution — Processor View . . . . . . . . . .*, BLOCK Distribution — Array View . . . . . . . . . . . . .*, BLOCK Distribution — Processor View . . . . . . . . . .*, CYCLIC Distribution — Array View . . . . . . . . . . . .*, CYCLIC Distribution — Processor View . . . . . . . . . .Visual Technique for Computing Two-DimensionalDistributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Distributing an Array (*, BLOCK) . . . . . . . . . . . . . . . .Distributing an Array (*, CYCLIC) . . . . . . . . . . . . . . .Distributing an Array (BLOCK, CYCLIC) . . . . . . . . . .Distributing an Array (BLOCK, BLOCK) . . . . . . . . . .LU Decomposition with (*, BLOCK) Distribution . . . .LU Decomposition with (*, CYCLIC) Distribution . . . .Three-Dimensional Problem and Its Two-DimensionalModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Shadow Edges for Nearest-Neighbor Optimization . . 17.C–2C–8Conventions Used in This Document . . . . . .HPF Directives and HPF-Specific Attribute .Summary of MPI Versions . . . . . . . . . . . . . .Explanation of Example 7–1 . . . . . . . . . . . .xviii5–46–87–9Tables15–16–17–1.ix
PrefaceThis manual describes the Compaq Fortran parallel processing environment.This environment comprises coding, compiling, linking, and executing CompaqFortran parallel programs using the Compaq Tru64 UNIX operating systemon Alpha hardware.This manual brings together explanations of Fortran parallel processingthat have appeared in other Compaq Fortran manuals. Chapter 5 containsdescriptions of HPF directives that first appeared in the Digital HighPerformance Fortran 90 HPF and PSE Manual.This manual also contains new material not previously published.Intended AudienceThis manual makes the following assumptions about you, the reader: You already have a basic understanding of the Fortran 95/90 language.Tutorial Fortran 95/90 language information is widely available incommercially published books (see the Preface of the Compaq FortranLanguage Reference Manual). You are familiar with the operating system shell commands used duringprogram development and a text editor, such as emacs or vi. Suchinformation is available in your operating system documentation set orcommercially published books. You have access to the Compaq Fortran Language Reference Manual, whichdescribes the Compaq Fortran language. You have access to the Compaq Fortran User Manual for Tru64 UNIX andLinux Alpha Systems, which describes the Compaq Fortran programmingenvironment including the compiler options, performance guidelines,run-time I/O, error handling support, and data types.xiii
Structure of This ManualThis manual consists of the following chapters and appendixes:xiv Chapter 1, Compaq Fortran Parallel Processing: An Introduction,introduces you to Compaq Fortran parallel processing. Chapter 2, Data Layout: An Introduction, introduces directives thatdistribute data — usually in large arrays — among the memories ofprocessors that share computations. Chapter 3, NUMA Architecture: An Introduction, introduces Non UniformMemory Access (NUMA) architecture. Chapter 4, High Performance Fortran (HPF) Software: An Introduction,introduces High Performance Fortran (HPF) software. Chapter 5, HPF Essentials, shows ways to distribute data among thememories of processors that share computations. Chapter 6, Compiling and Running HPF Programs, explains how tocompile and execute HPF programs. Chapter 7, Optimizing HPF Programs, explains how to write HPFprograms that execute quickly. Appendix A, HPF Tutorials: Introduction, introduces the four tutorials inthe appendixes. Appendix B, HPF Tutorial: LU Decomposition, contains an HPF exampleprogram from linear algebra. Appendix C, HPF Tutorial: Solving Nearest-Neighbor Problems, containsan HPF example program related to the transfer of heat in a rectangularsolid. Appendix D, HPF Tutorial: Visualizing the Mandelbrot Set, contains anHPF example program related to the Mandelbrot Set. Appendix E, HPF Tutorial: Simulating Network Striped Files, contains anHPF example program that illustrates input/output by simulating networkstriped files.
Associated DocumentsThe following documents may also be useful to Compaq Fortran programmers: Compaq Fortran User Manual for Tru64 UNIX and Linux Alpha SystemsDescribes compiling, linking, running, and debugging Compaq Fortranprograms, performance guidelines, run-time I/O, error-handling support,data types, numeric data conversion, calling other procedures and libraryroutines, and compatibility with Compaq Fortran 77 (formerly DECFortran).In particular, you should see the chapter on ‘‘Parallel Compiler Directivesand Their Programming Environment’’ and the appendix on ‘‘ParallelLibrary Routines.’’ Compaq Fortran Language Reference ManualDescribes the Compaq Fortran 95/90 source language for referencepurposes, including the format and use of statements, intrinsic procedures,and other language elements. Compaq Fortran Installation Guide for Tru64 UNIX SystemsExplains how to install Compaq Fortran on the Compaq Tru64 UNIXoperating system, including prerequisites and requirements. Compaq Fortran Release NotesProvide more information on this version of Compaq Fortran, includingknown problems and a summary of the Compaq Fortran run-time errormessages.The Release Notes are located in:/usr/lib/cmplrs/fort90/relnotes90 Compaq Fortran online reference pagesDescribe the Compaq Fortran software components, including f95(1),f90(1), f77(1), fpr(1), fsplit(1), intro(3f), numerous Fortran libraryroutines listed in intro(3f), and numerous parallel Fortran library routineslisted in intro(3hpf). Compaq Tru64 UNIX operating system documentationThe operating system documentation set includes reference pages foroperating system components and a programmer’s subkit, in which certainxv
documents describe the commands, tools, libraries, and other aspects of theprogramming environment:For programming information, see the Compaq Tru64 UNIXProgrammer’s Guide and the Compaq Tru64 UNIX Using ProgrammingSupport Tools.For performance information, see the Compaq Tru64 UNIX SystemTuning and Performance.For an overview of Compaq Tru64 UNIX documentation, see theCompaq Tru64 UNIX Reader’s Guide.For more information, see the Compaq Tru64 UNIX Web site at:http://www.tru64unix.compaq.com/ Other layered product documentationIf you are using a programming-related layered product package fromCompaq, consult the appropriate documentation for the layered productpackage for use of that product. High Performance Fortran (HPF)See the High Performance Fortran Language Specification, availablewithout charge at the following locations:The HPF Web site /index.cfmAnonymous FTP at ftp.cs.rice.edu in /public/HPFF/draft Third-party documentationIf you are unfamiliar with OpenMP software and will be using OpenMPdirectives to control parallel execution of your program, Compaqrecommends this book: Parallel Programming in OpenMP (Rohit Chandraet. al., Morgan Kaufmann, 2000) is a comprehensive introduction to thecompiler directives, run-time library routines, and environment variablesthat comprise OpenMP software. Its International Standard Book Number(ISBN) is 1-55860-671-8. More information about this book is on the Website for Morgan Kaufmann Publishers at http://www.mkp.com.xvi
Compaq Fortran Web PageThe Compaq Fortran home page is at:http://www.compaq.com/fortranThis Web site contains information about software patch kits, exampleprograms, and additional product information.Communicating with CompaqIf you have a customer support contract and have comments or questionsabout Compaq Fortran software, you can contact our Customer Support Center(CSC), preferably using electronic means (such as DSNlink). Customers in theUnited States can call the CSC at 1-800-354-9000.You can also send comments, questions, and suggestions about the CompaqFortran product to the following e-mail address: fortran@compaq.com. Notethat this address is for informational inquiries only and is not a formal supportchannel.Conventions Used in This DocumentThis manual uses the conventions listed in Table 1. Also, example code — suchas program red black 10.f90 in Section 1.4 — is usually in free source form(where a statement does not have to begin in position 7 of a line).xvii
Table 1 Conventions Used in This DocumentConventionMeaning%This manual uses a percent sign ( % ) to represent the Tru64UNIX system prompt. The actual user prompt varies withthe shell in use.% pwd/usr/usrc/jonesThis manual displays system prompts and responses usinga monospaced font. Typed user input is displayed in a boldmonospaced font.monospacedThis typeface indicates the name of a command, option,pathname, file name, directory path, or partition. Thistypeface is also used in examples of program code, interactiveexamples, and other screen displays.cat(1)A shell command name followed by the number 1 inparentheses refers to a command reference page. Similarly, aroutine name followed by the number 2 or 3 in parenthesesrefers to a system call or library routine reference page. (Thenumber in parentheses indicates the section containing thereference page.) To read online reference pages, use the man(1) command. Your operating system documentation alsoincludes reference page descriptions.newtermBold type indicates the introduction of a new term in text.variableItalic type indicates important information, a complete titleof a manual, or variable information, such as user-suppliedinformation in command or option syntax.UPPERCASElowercaseThe operating system shell differentiates between lowercaseand uppercase characters. Literal strings that appear in text,examples, syntax descriptions, and function definitions mustbe typed exactly as shown.8 9Large braces enclose lists from which you must choose oneitem. For example:([]STATUSDISPOSEDISP)Square brackets enclose items that are optional. For example:BLOCK DATA [nam](continued on next page)xviii
Table 1 (Cont.) Conventions Used in This DocumentConvention.MeaningA horizontal ellipsis means that the item preceding theellipsis can be repeated. For example:s[,s] . . .A vertical ellipsis in a figure or example means that not all ofthe statements are shown.realThis term refers to all floating-point intrinsic data types as agroup.complexThis term refers to all complex floating-point intrinsic datatypes as a group.logicalThis term refers to all logical data types as a group.integerThis term refers to all integer data types as a group.AlphaAlpha systemsThe terms Alpha and Alpha systems refer to the Alpha architecture or systems equipped with this 64-bit architecture.Compaq Tru64 UNIXCompaq Tru64 UNIXsystemsThe terms Compaq Tru64 UNIX and Compaq Tru64 UNIXsystems refer to the Compaq Tru64 UNIX (formerly DIGITALUNIX) operating system running on Alpha processorhardware.FortranThis term refers to language information that is common toANSI FORTRAN 77, ANSI/ISO Fortran 95/90, and CompaqFortran.Fortran 95/90This term refers to language information that is common toANSI/ISO Fortran 95 and ANSI/ISO Fortran 90.f90This command invokes the Compaq Fortran compiler onTru64 UNIX Alpha systems.Compaq Fortran 77DEC FortranThe term Compaq Fortran 77 (formerly DEC Fortran) refersto language information that is common to the FORTRAN-77standard and any Compaq Fortran extensions.Compaq FortranThe term Compaq For
This manual provides information about the Compaq Fortran parallel program development and run-time environment on Compaq Tru64 UNIX systems. Note: The NUMA parallel processing feature described in this manual is available in the Compaq Fortran software but is not supported. Revision/Update Information: This is a new manual.
Double-Master-Degree Program, University of Passau & New Jersey Institute of Technology 15 Contact –New Jersey Institute of Technology Program Coordinator and Academic Coordinator Prof. Layek Abdel Malek Department of Mechanical and Industrial Engineering New Jersey Institute of Technology University Height
Jul 01, 2019 · New Jersey Is An Equal Opportunity Employer State of New Jersey NEW JERSEY STATE PAROLE BOARD P.O. BOX 862 PHILIP D. MURPHY TRENTON, NEW JERSEY 08625 Governor TELEPHONE NUMBER: (609) 292-0845 SHEILA Y. OLIVER SAMUEL J. PLUMERI, JR. Lt. Governor
A Publication of New Jersey - The New Jersey Society of Architects New Jersey 2007 Design Awards Winners AIA New Jersey's Michael Graves Lifetime Achievement Award J. Robert Hillier, FAIA 12 19 Member News New Licensee Distinguished Service Award David DelVecchio, AIA Architect of the Year Hugh Boyd, FAIA Architectural Firm of the Year NK .
Figure 1-10 New Jersey Rail System–Post Conrail 1-18 Figure 1-11 New Jersey Rail System–The Aldene Plan 1-21 Figure 1-12 New Jersey Transit Rail System 1-24 Figure 1-13 U.S. Greenhouse Gas Emissions in 2009 by Economic Sector 1-26 Figure 2-1 New Jersey Rail System Ownership 2
Society of Interventional Pain Physicians, American Academy of Family Physicians, New Jersey Society of Gastroenterology and Endoscopy, New Jersey Chapter of the American College of Cardiology, the New Jersey Rheumatology Association, the New Jersey Academy of Otolaryngology - Head & Neck Surgery and the New Jersey Academy of Facial Plastic .
White, M. Campo, M. Kaplan, J. Herb, and L. Auermuller. New Jersey's Rising Seas and Changing Coastal Storms: Report of the 2019 Science and Technical Advisory Panel. Rutgers, The State University of New Jersey. Prepared for the New Jersey Department of Environmental Protection. Trenton, New Jersey.
transact insurance in New Jersey. 17. Drive New Jersey is and was, at all times relevant to this lawsuit, an Ohio corporation with its principal place of business in Ohio. Drive New Jersey is authorized to transact insurance in New Jersey. FACTUAL ALLEGATIONS A. The Progressive Policy 18.
Edison, New Jersey February 17-20, 2022 Exhibitor Kit . NJBSE 2022 - 2 GENERAL INFORMATION Dear New Jersey Boat Sale & Expo Exhibitor: On behalf of the MTA/NJ, we would like to welcome you to the New Jersey Boat Sale & Expo and thank you for participating. The New Jersey Boat Sale & Expo is organized by the Marine Trades Association of New
