GAWK: Effective AWK Programming - GNU/Linux

iv GAWK: Effective AWK Programming5Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 735.1Constant Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.1.1 Numeric and String Constants . . . . . . . . . . . . . . . . . . . . . . . . . .5.1.2 Octal and Hexadecimal Numbers. . . . . . . . . . . . . . . . . . . . . . . .5.1.3 Regular Expression Constants . . . . . . . . . . . . . . . . . . . . . . . . . .5.2 Using Regular Expression Constants . . . . . . . . . . . . . . . . . . . . . . . . .5.3 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.3.1 Using Variables in a Program . . . . . . . . . . . . . . . . . . . . . . . . . . .5.3.2 Assigning Variables on the Command Line . . . . . . . . . . . . . . .5.4 Conversion of Strings and Numbers . . . . . . . . . . . . . . . . . . . . . . . . . .5.5 Arithmetic Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.6 String Concatenation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.7 Assignment Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.8 Increment and Decrement Operators . . . . . . . . . . . . . . . . . . . . . . . . .5.9 True and False in awk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.10 Variable Typing and Comparison Expressions . . . . . . . . . . . . . . .5.11 Boolean Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.12 Conditional Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.13 Function Calls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.14 Operator Precedence (How Operators Nest) . . . . . . . . . . . . . . . . .673737374747676767778808183858588898990Patterns, Actions, and Variables . . . . . . . . . . . 936.1Pattern Elements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 936.1.1 Regular Expressions as Patterns . . . . . . . . . . . . . . . . . . . . . . . . 936.1.2 Expressions as Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 936.1.3 Specifying Record Ranges with Patterns . . . . . . . . . . . . . . . . . 956.1.4 The BEGIN and END Special Patterns . . . . . . . . . . . . . . . . . . . . 966.1.4.1 Startup and Cleanup Actions . . . . . . . . . . . . . . . . . . . . . . 966.1.4.2 Input/Output from BEGIN and END Rules . . . . . . . . . . . 976.1.5 The Empty Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 976.2 Using Shell Variables in Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . 976.3 Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 986.4 Control Statements in Actions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 996.4.1 The if-else Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 996.4.2 The while Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1006.4.3 The do-while Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1016.4.4 The for Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1016.4.5 The switch Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1026.4.6 The break Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1036.4.7 The continue Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1046.4.8 The next Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1056.4.9 Using gawk’s nextfile Statement . . . . . . . . . . . . . . . . . . . . . . 1066.4.10 The exit Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1066.5 Built-in Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1076.5.1 Built-in Variables That Control awk . . . . . . . . . . . . . . . . . . . . 1076.5.2 Built-in Variables That Convey Information . . . . . . . . . . . . 1106.5.3 Using ARGC and ARGV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

v7Arrays in awk . . . . . . . . . . . . . . . . . . . . . . . . . . . . n to Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Referring to an Array Element . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Assigning Array Elements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Basic Array Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Scanning All Elements of an Array . . . . . . . . . . . . . . . . . . . . . . . . .The delete Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Using Numbers to Subscript Arrays. . . . . . . . . . . . . . . . . . . . . . . . .Using Uninitialized Variables as Subscripts . . . . . . . . . . . . . . . . . .Multidimensional Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Scanning Multidimensional Arrays . . . . . . . . . . . . . . . . . . . . . . . . .Sorting Array Values and Indices with gawk . . . . . . . . . . . . . . . .Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1278.1Built-in Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8.1.1 Calling Built-in Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8.1.2 Numeric Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8.1.3 String-Manipulation Functions . . . . . . . . . . . . . . . . . . . . . . . . .8.1.3.1 More About ‘\’ and ‘&’ with sub, gsub, and gensub.8.1.4 Input/Output Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8.1.5 Using gawk’s Timestamp Functions . . . . . . . . . . . . . . . . . . . .8.1.6 Bit-Manipulation Functions of gawk . . . . . . . . . . . . . . . . . . . .8.1.7 Using gawk’s String-Translation Functions . . . . . . . . . . . . . .8.2 User-Defined Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8.2.1 Function Definition Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . .8.2.2 Function Definition Examples. . . . . . . . . . . . . . . . . . . . . . . . . .8.2.3 Calling User-Defined Functions . . . . . . . . . . . . . . . . . . . . . . . .8.2.4 The return Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8.2.5 Functions and Their Effects on Variable Typing . . . . . . . . 140142147149149149151152154155Internationalization with gawk . . . . . . . . . . . . 1569.19.29.39.4Internationalization and Localization . . . . . . . . . . . . . . . . . . . . . . .GNU gettext . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Internationalizing awk Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . .Translating awk Programs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9.4.1 Extracting Marked Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . .9.4.2 Rearranging printf Arguments . . . . . . . . . . . . . . . . . . . . . . . .9.4.3 awk Portability Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9.5 A Simple Internationalization Example . . . . . . . . . . . . . . . . . . . . .9.6 gawk Can Speak Your Language . . . . . . . . . . . . . . . . . . . . . . . . . . . .156156158159160160161162163

vi GAWK: Effective AWK Programming10Advanced Features of gawk . . . . . . . . . . . . . . 16510.110.210.310.410.511Invoking awk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Command-Line Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Other Command-Line Arguments . . . . . . . . . . . . . . . . . . . . . . . . .The AWKPATH Environment Variable . . . . . . . . . . . . . . . . . . . . . . .Obsolete Options and/or Features . . . . . . . . . . . . . . . . . . . . . . . . .Undocumented Options and Features . . . . . . . . . . . . . . . . . . . . . .Known Bugs in gawk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .173173178178179180180A Library of awk Functions . . . . . . . . . . . . . . 18112.1 Naming Library Function Global Variables . . . . . . . . . . . . . . . . .12.2 General Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12.2.1 Implementing nextfile as a Function . . . . . . . . . . . . . . . . .12.2.2 Converting Strings To Numbers . . . . . . . . . . . . . . . . . . . . . . .12.2.3 Assertions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12.2.4 Rounding Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12.2.5 The Cliff Random Number Generator . . . . . . . . . . . . . . . . .12.2.6 Translating Between Characters and Numbers . . . . . . . . .12.2.7 Merging an Array into a String . . . . . . . . . . . . . . . . . . . . . . .12.2.8 Managing the Time of Day . . . . . . . . . . . . . . . . . . . . . . . . . . .12.3 Data File Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12.3.1 Noting Data File Boundaries . . . . . . . . . . . . . . . . . . . . . . . . .12.3.2 Rereading the Current File . . . . . . . . . . . . . . . . . . . . . . . . . . .12.3.3 Checking for Readable Data Files . . . . . . . . . . . . . . . . . . . . .12.3.4 Checking For Zero-length Files. . . . . . . . . . . . . . . . . . . . . . . .12.3.5 Treating Assignments as File Names . . . . . . . . . . . . . . . . . .12.4 Processing Command-Line Options . . . . . . . . . . . . . . . . . . . . . . . .12.5 Reading the User Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12.6 Reading the Group Database. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13165166168169169Running awk and gawk . . . . . . . . . . . . . . . . . . . 17311.111.211.311.411.511.611.712Allowing Nondecimal Input Data . . . . . . . . . . . . . . . . . . . . . . . . . .Two-Way Communications with Another Process . . . . . . . . . . .Using gawk for Network Programming . . . . . . . . . . . . . . . . . . . . .Using gawk with BSD Portals . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Profiling Your awk Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96201205Practical awk Programs . . . . . . . . . . . . . . . . . 21013.1 Running the Example Programs . . . . . . . . . . . . . . . . . . . . . . . . . . .13.2 Reinventing Wheels for Fun and Profit . . . . . . . . . . . . . . . . . . . . .13.2.1 Cutting out Fields and Columns . . . . . . . . . . . . . . . . . . . . . .13.2.2 Searching for Regular Expressions in Files . . . . . . . . . . . . .13.2.3 Printing out User Information . . . . . . . . . . . . . . . . . . . . . . . .13.2.4 Splitting a Large File into Pieces. . . . . . . . . . . . . . . . . . . . . .13.2.5 Duplicating Output into Multiple Files . . . . . . . . . . . . . . . .13.2.6 Printing Nonduplicated Lines of Text . . . . . . . . . . . . . . . . .13.2.7 Counting Things . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .210210210215219221223224228

vii13.3 A Grab Bag of awk Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13.3.1 Finding Duplicated Words in a Document . . . . . . . . . . . . .13.3.2 An Alarm Clock Program . . . . . . . . . . . . . . . . . . . . . . . . . . . .13.3.3 Transliterating Characters . . . . . . . . . . . . . . . . . . . . . . . . . . . .13.3.4 Printing Mailing Labels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13.3.5 Generating Word-Usage Counts . . . . . . . . . . . . . . . . . . . . . . .13.3.6 Removing Duplicates from Unsorted Text . . . . . . . . . . . . .13.3.7 Extracting Programs from Texinfo Source Files . . . . . . . .13.3.8 A Simple Stream Editor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13.3.9 An Easy Way to Use Library Functions . . . . . . . . . . . . . . .230230231233235237239240243244Appendix A The Evolution of the awk Language. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252A.1A.2A.3A.4A.5A.6Major Changes Between V7 and SVR3.1 . . . . . . . . . . . . . . . . . . .Changes Between SVR3.1 and SVR4 . . . . . . . . . . . . . . . . . . . . . . .Changes Between SVR4 and POSIX awk . . . . . . . . . . . . . . . . . . .Extensions in the Bell Laboratories awk . . . . . . . . . . . . . . . . . . . .Extensions in gawk Not in POSIX awk. . . . . . . . . . . . . . . . . . . . . .Major Contributors to gawk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Appendix B252253253254255258Installing gawk . . . . . . . . . . . . . . . . 260B.1 The gawk Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .B.1.1 Getting the gawk Distribution . . . . . . . . . . . . . . . . . . . . . . . . .B.1.2 Extracting the Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . .B.1.3 Contents of the gawk Distribution . . . . . . . . . . . . . . . . . . . . .B.2 Compiling and Installing gawk on Unix . . . . . . . . . . . . . . . . . . . . .B.2.1 Compiling gawk for Unix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .B.2.2 Additional Configuration Options . . . . . . . . . . . . . . . . . . . . .B.2.3 The Configuration Process . . . . . . . . . . . . . . . . . . . . . . . . . . . .B.3 Installation on Other Operating Systems . . . . . . . . . . . . . . . . . . .B.3.1 Installing gawk on an Amiga . . . . . . . . . . . . . . . . . . . . . . . . . .B.3.2 Installing gawk on BeOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .B.3.3 Installation on PC Operating Systems . . . . . . . . . . . . . . . . .B.3.3.1 Installing a Prepared Distribution for PC Systems . .B.3.3.2 Compiling gawk for PC Operating Systems . . . . . . . .B.3.3.3 Compiling gawk For Dynamic Libraries . . . . . . . . . . . .B.3.3.4 Using gawk on PC Operating Systems . . . . . . . . . . . . .B.3.3.5 Using gawk In The Cygwin Environment . . . . . . . . . .B.3.4 How to Compile and Install gawk on VMS. . . . . . . . . . . . . .B.3.4.1 Compiling gawk on VMS . . . . . . . . . . . . . . . . . . . . . . . . .B.3.4.2 Installing gawk on VMS . . . . . . . . . . . . . . . . . . . . . . . . . .B.3.4.3 Running gawk on VMS . . . . . . . . . . . . . . . . . . . . . . . . . . .B.3.4.4 Building and Using gawk on VMS POSIX . . . . . . . . .B.4 Unsupported Operating System Ports . . . . . . . . . . . . . . . . . . . . . .B.4.1 Installing gawk on the Atari ST . . . . . . . . . . . . . . . . . . . . . . .B.4.1.1 Compiling gawk on the Atari ST . . . . . . . . . . . . . . . . . .B.4.1.2 Running gawk on the Atari ST . . . . . . . . . . . . . . . . . . . 71271271272273273273274274274

viiiGAWK: Effective AWK ProgrammingB.4.2 Installing gawk on a Tandem . . . . . . . . . . . . . . . . . . . . . . . . . . 275B.5 Reporting Problems and Bugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276B.6 Other Freely Available awk Implementations . . . . . . . . . . . . . . . . 277Appendix CImplementation Notes . . . . . . . . 279C.1 Downward Compatibility and Debugging . . . . . . . . . . . . . . . . . . .C.2 Making Additions to gawk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .C.2.1 Adding New Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .C.2.2 Porting gawk to a New Operating System . . . . . . . . . . . . . .C.3 Adding New Built-in Functions to gawk . . . . . . . . . . . . . . . . . . . .C.3.1 A Minimal Introduction to gawk Internals . . . . . . . . . . . . . .C.3.2 Directory and File Operation Built-ins . . . . . . . . . . . . . . . . .C.3.2.1 Using chdir and stat . . . . . . . . . . . . . . . . . . . . . . . . . . .C.3.2.2 C Code for chdir and stat . . . . . . . . . . . . . . . . . . . . . .C.3.2.3 Integrating the Extensions. . . . . . . . . . . . . . . . . . . . . . . .C.4 Probable Future Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Appendix DD.1D.2D.3279279279281282282286286287290291Basic Programming Concepts . . 294What a Program Does . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294Data Values in a Computer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295Floating-Point Number Caveats . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299GNU General Public License . . . . . . . . . . . . . . . . 309Preamble . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309Terms and Conditions for Copying, Distribution and Modification . . 310How to Apply These Terms to Your New Programs . . . . . . . . . . . . . . . 314GNU Free Documentation License . . . . . . . . . . . 315ADDENDUM: How to use this License for your documents . . . . . . . . 320Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322

1ForewordArnold Robbins and I are good friends. We were introduced 11 years ago by circumstances—and our favorite programming language, AWK. The circumstances started a couple of yearsearlier. I was working at a new job and noticed an unplugged Unix computer sitting in thecorner. No one knew how to use it, and neither did I. However, a couple of days later itwas running, and I was root and the one-and-only user. That day, I began the transitionfrom statistician to Unix programmer.On one of many trips to the library or bookstore in search of books on Unix, I foundthe gray AWK book, a.k.a. Aho, Kernighan and Weinberger, The AWK ProgrammingLanguage, Addison-Wesley, 1988. AWK’s simple programming paradigm—find a pattern inthe input and then perform an action—often reduced complex or tedious data manipulationsto few lines of code. I was excited to try my hand at programming in AWK.Alas, the awk on my computer was a limited version of the language described in theAWK book. I discovered that my computer had “old awk” and the AWK book described“new awk.” I learned that this was typical; the old version refused to step aside or relinquishits name. If a system had a new awk, it was invariably called nawk, and few systems had it.The best way to get a new awk was to ftp the source code for gawk from prep.ai.mit.edu.gawk was a version of new awk written by David Trueman and Arnold, and available underthe GNU General Public License.(Incidentally, it’s no longer difficult to find a new awk. gawk ships with Linux, and youcan download binaries or source code for almost any system; my wife uses gawk on her VMSbox.)My Unix system started out unplugged from the wall; it certainly was not plugged intoa network. So, oblivious to the existence of gawk and the Unix community in general, anddesiring a new awk, I wrote my own, called mawk. Before I was finished I knew about gawk,but it was too late to stop, so I eventually posted to a comp.sources newsgroup.A few days after my posting, I got a friendly email from Arnold introducing himself.He suggested we share design and algorithms and attached a draft of the POSIX standardso that I could update mawk to support language extensions added after publication of theAWK book.Frankly, if our roles had been reversed, I would not have been so open and we probablywould have never met. I’m glad we did meet. He is an AWK expert’s AWK exper

This is Edition 3 of GAWK: Effective AWK Programming: A User's Guide for GNU Awk, for the 3.1.4 (or later) version of the GNU implementation of AWK. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the