• • Foreword to the Third Edition
  • Foreword to the Fourth Edition
  • Preface
• Part I:
  The awk Language
  • 1 Getting Started with awk
  • 2 Running awk and gawk
  • 3 Regular Expressions
  • 4 Reading Input Files
  • 5 Printing Output
  • 6 Expressions
  • 7 Patterns, Actions, and Variables
  • 8 Arrays in awk
  • 9 Functions
• Part II:
  Problem Solving with awk
  • 10 A Library of awk Functions
  • 11 Practical awk Programs
• Part III:
  Moving Beyond Standard awk with gawk
  • 12 Advanced Features of gawk
  • 13 Internationalization with gawk
  • 14 Debugging awk Programs
  • 15 Namespaces in gawk
  • 16 Arithmetic and Arbitrary-Precision Arithmetic with gawk
  • 17 Writing Extensions for gawk
• Part IV:
  Appendices
  • Appendix A The Evolution of the awk Language
  • Appendix B Installing gawk
  • Appendix C Implementation Notes
  • Appendix D Basic Programming Concepts
  • Glossary
  • GNU General Public License
  • GNU Free Documentation License
  • Index

• • Foreword to the Third Edition
  • Foreword to the Fourth Edition
  • Preface
    • History of awk and gawk
    • A Rose by Any Other Name
    • Using This Book
    • Typographical Conventions
      • Dark Corners
    • The GNU Project and This Book
    • How to Contribute
    • Acknowledgments
• Part I:
  The awk Language
  • 1 Getting Started with awk
    • 1.1 How to Run awk Programs
      • 1.1.1 One-Shot Throwaway awk Programs
      • 1.1.2 Running awk Without Input Files
      • 1.1.3 Running Long Programs
      • 1.1.4 Executable awk Programs
      • 1.1.5 Comments in awk Programs
      • 1.1.6 Shell Quoting Issues
        • 1.1.6.1 Quoting in MS-Windows Batch Files
    • 1.2 Data files for the Examples
    • 1.3 Some Simple Examples
    • 1.4 An Example with Two Rules
    • 1.5 A More Complex Example
    • 1.6 awk Statements Versus Lines
    • 1.7 Other Features of awk
    • 1.8 When to Use awk
    • 1.9 Summary
  • 2 Running awk and gawk
    • 2.1 Invoking awk
    • 2.2 Command-Line Options
    • 2.3 Other Command-Line Arguments
    • 2.4 Naming Standard Input
    • 2.5 The Environment Variables gawk Uses
      • 2.5.1 The AWKPATH Environment Variable
      • 2.5.2 The AWKLIBPATH Environment Variable
      • 2.5.3 Other Environment Variables
    • 2.6 gawk’s Exit Status
    • 2.7 Including Other Files into Your Program
    • 2.8 Loading Dynamic Extensions into Your Program
    • 2.9 Obsolete Options and/or Features
    • 2.10 Undocumented Options and Features
    • 2.11 Summary
  • 3 Regular Expressions
    • 3.1 How to Use Regular Expressions
    • 3.2 Escape Sequences
    • 3.3 Regular Expression Operators
      • 3.3.1 Regexp Operators in awk
      • 3.3.2 Some Notes On Interval Expressions
      • 3.3.3 Shortest Match, or “Non-greedy” Regexp Operators
    • 3.4 Using Bracket Expressions
    • 3.5 How Much Text Matches?
    • 3.6 Using Dynamic Regexps
    • 3.7 gawk-Specific Regexp Operators
    • 3.8 Case Sensitivity in Matching
    • 3.9 Selecting the Regexp Matching Engine
    • 3.10 Summary
  • 4 Reading Input Files
    • 4.1 How Input Is Split into Records
      • 4.1.1 Record Splitting with Standard awk
      • 4.1.2 Record Splitting with gawk
    • 4.2 Examining Fields
    • 4.3 Nonconstant Field Numbers
    • 4.4 Changing the Contents of a Field
    • 4.5 Specifying How Fields Are Separated
      • 4.5.1 Whitespace Normally Separates Fields
      • 4.5.2 Using Regular Expressions to Separate Fields
      • 4.5.3 Making Each Character a Separate Field
      • 4.5.4 Working With Comma Separated Value Files
      • 4.5.5 Setting FS from the Command Line
      • 4.5.6 Making the Full Line Be a Single Field
      • 4.5.7 Field-Splitting Summary
    • 4.6 Reading Fixed-Width Data
      • 4.6.1 Processing Fixed-Width Data
      • 4.6.2 Skipping Intervening Fields
      • 4.6.3 Capturing Optional Trailing Data
      • 4.6.4 Field Values With Fixed-Width Data
    • 4.7 Defining Fields by Content
      • 4.7.1 FS Versus FPAT: A Subtle Difference
    • 4.8 Checking How gawk Is Splitting Records
    • 4.9 Multiple-Line Records
    • 4.10 Explicit Input with getline
      • 4.10.1 Using getline with No Arguments
      • 4.10.2 Using getline into a Variable
      • 4.10.3 Using getline from a File
      • 4.10.4 Using getline into a Variable from a File
      • 4.10.5 Using getline from a Pipe
      • 4.10.6 Using getline into a Variable from a Pipe
      • 4.10.7 Using getline from a Coprocess
      • 4.10.8 Using getline into a Variable from a Coprocess
      • 4.10.9 Points to Remember About getline
      • 4.10.10 Summary of getline Variants
    • 4.11 Reading Input with a Timeout
    • 4.12 Retrying Reads After Certain Input Errors
    • 4.13 Directories on the Command Line
    • 4.14 Summary
    • 4.15 Exercises
  • 5 Printing Output
    • 5.1 The print Statement
    • 5.2 print Statement Examples
    • 5.3 Output Separators
    • 5.4 Controlling Numeric Output with print
    • 5.5 Using printf Statements for Fancier Printing
      • 5.5.1 Introduction to the printf Statement
      • 5.5.2 Format-Control Letters
      • 5.5.3 Modifiers for printf Formats
      • 5.5.4 Examples Using printf
    • 5.6 Redirecting Output of print and printf
    • 5.7 Special Files for Standard Preopened Data Streams
    • 5.8 Special File names in gawk
      • 5.8.1 Accessing Other Open Files with gawk
      • 5.8.2 Special Files for Network Communications
      • 5.8.3 Special File name Caveats
    • 5.9 Closing Input and Output Redirections
      • 5.9.1 Using close()’s Return Value
    • 5.10 Speeding Up Pipe Output
    • 5.11 Enabling Nonfatal Output
    • 5.12 Summary
    • 5.13 Exercises
  • 6 Expressions
    • 6.1 Constants, Variables, and Conversions
      • 6.1.1 Constant Expressions
        • 6.1.1.1 Numeric and String Constants
        • 6.1.1.2 Octal and Hexadecimal Numbers
        • 6.1.1.3 Regular Expression Constants
      • 6.1.2 Using Regular Expression Constants
        • 6.1.2.1 Standard Regular Expression Constants
        • 6.1.2.2 Strongly Typed Regexp Constants
      • 6.1.3 Variables
        • 6.1.3.1 Using Variables in a Program
        • 6.1.3.2 Assigning Variables on the Command Line
      • 6.1.4 Conversion of Strings and Numbers
        • 6.1.4.1 How awk Converts Between Strings and Numbers
        • 6.1.4.2 Locales Can Influence Conversion
    • 6.2 Operators: Doing Something with Values
      • 6.2.1 Arithmetic Operators
      • 6.2.2 String Concatenation
      • 6.2.3 Assignment Expressions
      • 6.2.4 Increment and Decrement Operators
    • 6.3 Truth Values and Conditions
      • 6.3.1 True and False in awk
      • 6.3.2 Variable Typing and Comparison Expressions
        • 6.3.2.1 String Type versus Numeric Type
        • 6.3.2.2 Comparison Operators
        • 6.3.2.3 String Comparison Based on Locale Collating Order
      • 6.3.3 Boolean Expressions
      • 6.3.4 Conditional Expressions
    • 6.4 Function Calls
    • 6.5 Operator Precedence (How Operators Nest)
    • 6.6 Where You Are Makes a Difference
    • 6.7 Summary
  • 7 Patterns, Actions, and Variables
    • 7.1 Pattern Elements
      • 7.1.1 Regular Expressions as Patterns
      • 7.1.2 Expressions as Patterns
      • 7.1.3 Specifying Record Ranges with Patterns
      • 7.1.4 The BEGIN and END Special Patterns
        • 7.1.4.1 Startup and Cleanup Actions
        • 7.1.4.2 Input/Output from BEGIN and END Rules
      • 7.1.5 The BEGINFILE and ENDFILE Special Patterns
      • 7.1.6 The Empty Pattern
    • 7.2 Using Shell Variables in Programs
    • 7.3 Actions
    • 7.4 Control Statements in Actions
      • 7.4.1 The if-else Statement
      • 7.4.2 The while Statement
      • 7.4.3 The do-while Statement
      • 7.4.4 The for Statement
      • 7.4.5 The switch Statement
      • 7.4.6 The break Statement
      • 7.4.7 The continue Statement
      • 7.4.8 The next Statement
      • 7.4.9 The nextfile Statement
      • 7.4.10 The exit Statement
    • 7.5 Predefined Variables
      • 7.5.1 Built-in Variables That Control awk
      • 7.5.2 Built-in Variables That Convey Information
      • 7.5.3 Using ARGC and ARGV
    • 7.6 Summary
  • 8 Arrays in awk
    • 8.1 The Basics of Arrays
      • 8.1.1 Introduction to Arrays
      • 8.1.2 Referring to an Array Element
      • 8.1.3 Assigning Array Elements
      • 8.1.4 Basic Array Example
      • 8.1.5 Scanning All Elements of an Array
      • 8.1.6 Using Predefined Array Scanning Orders with gawk
    • 8.2 Using Numbers to Subscript Arrays
    • 8.3 Using Uninitialized Variables as Subscripts
    • 8.4 The delete Statement
    • 8.5 Multidimensional Arrays
      • 8.5.1 Scanning Multidimensional Arrays
    • 8.6 Arrays of Arrays
    • 8.7 Summary
  • 9 Functions
    • 9.1 Built-in Functions
      • 9.1.1 Calling Built-in Functions
      • 9.1.2 Generating Boolean Values
      • 9.1.3 Numeric Functions
      • 9.1.4 String-Manipulation Functions
        • 9.1.4.1 More about ‘\’ and ‘&’ with sub(), gsub(), and gensub()
      • 9.1.5 Input/Output Functions
      • 9.1.6 Time Functions
      • 9.1.7 Bit-Manipulation Functions
      • 9.1.8 Getting Type Information
      • 9.1.9 String-Translation Functions
    • 9.2 User-Defined Functions
      • 9.2.1 Function Definition Syntax
      • 9.2.2 Function Definition Examples
      • 9.2.3 Calling User-Defined Functions
        • 9.2.3.1 Writing a Function Call
        • 9.2.3.2 Controlling Variable Scope
        • 9.2.3.3 Passing Function Arguments by Value Or by Reference
        • 9.2.3.4 Other Points About Calling Functions
      • 9.2.4 The return Statement
    • 9.3 Variable Typing Is Dynamic
      • 9.3.1 Dynamic Typing In Standard awk
      • 9.3.2 Dynamic Typing In gawk
    • 9.4 A Note On Shadowed Variables
    • 9.5 Indirect Function Calls
    • 9.6 Summary
• Part II:
  Problem Solving with awk
  • 10 A Library of awk Functions
    • 10.1 Naming Library Function Global Variables
    • 10.2 General Programming
      • 10.2.1 Converting Strings to Numbers
      • 10.2.2 Assertions
      • 10.2.3 Rounding Numbers
      • 10.2.4 The Cliff Random Number Generator
      • 10.2.5 Translating Between Characters and Numbers
      • 10.2.6 Merging an Array into a String
      • 10.2.7 Managing the Time of Day
      • 10.2.8 Reading a Whole File at Once
      • 10.2.9 Quoting Strings to Pass to the Shell
      • 10.2.10 Checking Whether A Value Is Numeric
      • 10.2.11 Producing CSV Data
    • 10.3 Data file Management
      • 10.3.1 Noting Data file Boundaries
      • 10.3.2 Rereading the Current File
      • 10.3.3 Checking for Readable Data files
      • 10.3.4 Checking for Zero-Length Files
      • 10.3.5 Treating Assignments as File names
    • 10.4 Processing Command-Line Options
    • 10.5 Reading the User Database
    • 10.6 Reading the Group Database
    • 10.7 Traversing Arrays of Arrays
    • 10.8 Summary
    • 10.9 Exercises
  • 11 Practical awk Programs
    • 11.1 Running the Example Programs
    • 11.2 Reinventing Wheels for Fun and Profit
      • 11.2.1 Cutting Out Fields and Columns
      • 11.2.2 Searching for Regular Expressions in Files
      • 11.2.3 Printing Out User Information
      • 11.2.4 Splitting a Large File into Pieces
      • 11.2.5 Duplicating Output into Multiple Files
      • 11.2.6 Printing Nonduplicated Lines of Text
      • 11.2.7 Counting Things
        • 11.2.7.1 Modern Character Sets
        • 11.2.7.2 A Brief Introduction To Extensions
        • 11.2.7.3 Code for wc.awk
    • 11.3 A Grab Bag of awk Programs
      • 11.3.1 Finding Duplicated Words in a Document
      • 11.3.2 An Alarm Clock Program
      • 11.3.3 Transliterating Characters
      • 11.3.4 Printing Mailing Labels
      • 11.3.5 Generating Word-Usage Counts
      • 11.3.6 Removing Duplicates from Unsorted Text
      • 11.3.7 Extracting Programs from Texinfo Source Files
      • 11.3.8 A Simple Stream Editor
      • 11.3.9 An Easy Way to Use Library Functions
      • 11.3.10 Finding Anagrams from a Dictionary
      • 11.3.11 And Now for Something Completely Different
      • 11.3.12 Demonstrating Shortest and Longest Match Operators
    • 11.4 Summary
    • 11.5 Exercises
• Part III:
  Moving Beyond Standard awk with gawk
  • 12 Advanced Features of gawk
    • 12.1 Allowing Nondecimal Input Data
    • 12.2 Boolean Typed Values
    • 12.3 Controlling Array Traversal and Array Sorting
      • 12.3.1 Controlling Array Traversal
      • 12.3.2 Sorting Array Values and Indices with gawk
    • 12.4 Two-Way Communications with Another Process
    • 12.5 Using gawk for Network Programming
    • 12.6 Profiling Your awk Programs
    • 12.7 Preserving Data Between Runs
    • 12.8 Builtin Features versus Extensions
    • 12.9 Summary
  • 13 Internationalization with gawk
    • 13.1 Internationalization and Localization
    • 13.2 GNU gettext
    • 13.3 Internationalizing awk Programs
    • 13.4 Translating awk Programs
      • 13.4.1 Extracting Marked Strings
      • 13.4.2 Rearranging printf Arguments
      • 13.4.3 awk Portability Issues
    • 13.5 A Simple Internationalization Example
    • 13.6 gawk Can Speak Your Language
    • 13.7 Summary
  • 14 Debugging awk Programs
    • 14.1 Introduction to the gawk Debugger
      • 14.1.1 Debugging in General
      • 14.1.2 Debugging Concepts
      • 14.1.3 awk Debugging
    • 14.2 Sample gawk Debugging Session
      • 14.2.1 How to Start the Debugger
      • 14.2.2 Finding the Bug
    • 14.3 Main Debugger Commands
      • 14.3.1 Control of Breakpoints
      • 14.3.2 Control of Execution
      • 14.3.3 Viewing and Changing Data
      • 14.3.4 Working with the Stack
      • 14.3.5 Obtaining Information About the Program and the Debugger State
      • 14.3.6 Miscellaneous Commands
    • 14.4 Readline Support
    • 14.5 Limitations
    • 14.6 Summary
  • 15 Namespaces in gawk
    • 15.1 Standard awk’s Single Namespace
    • 15.2 Qualified Names
    • 15.3 The Default Namespace
    • 15.4 Changing The Namespace
    • 15.5 Namespace and Component Naming Rules
    • 15.6 Internal Name Management
    • 15.7 Namespace Example
    • 15.8 Including A File Without Changing The Namespace
    • 15.9 Namespaces and Other gawk Features
    • 15.10 Summary
  • 16 Arithmetic and Arbitrary-Precision Arithmetic with gawk
    • 16.1 A General Description of Computer Arithmetic
    • 16.2 Other Stuff to Know
    • 16.3 Arbitrary-Precision Arithmetic Features in gawk
      • 16.3.1 Arbitrary Precision Arithmetic is On Parole!
      • 16.3.2 Arbitrary Precision Introduction
    • 16.4 Floating-Point Arithmetic: Caveat Emptor!
      • 16.4.1 Floating-Point Arithmetic Is Not Exact
        • 16.4.1.1 Many Numbers Cannot Be Represented Exactly
        • 16.4.1.2 Be Careful Comparing Values
        • 16.4.1.3 Errors Accumulate
        • 16.4.1.4 Floating Point Values They Didn’t Talk About In School
      • 16.4.2 Getting the Accuracy You Need
      • 16.4.3 Try a Few Extra Bits of Precision and Rounding
      • 16.4.4 Setting the Precision
      • 16.4.5 Setting the Rounding Mode
    • 16.5 Arbitrary-Precision Integer Arithmetic with gawk
    • 16.6 How To Check If MPFR Is Available
    • 16.7 Standards Versus Existing Practice
    • 16.8 Summary
  • 17 Writing Extensions for gawk
    • 17.1 Introduction
    • 17.2 Extension Licensing
    • 17.3 How It Works at a High Level
    • 17.4 API Description
      • 17.4.1 Introduction
      • 17.4.2 General-Purpose Data Types
      • 17.4.3 Memory Allocation Functions and Convenience Macros
      • 17.4.4 Constructor Functions
      • 17.4.5 Managing MPFR and GMP Values
      • 17.4.6 Registration Functions
        • 17.4.6.1 Registering An Extension Function
        • 17.4.6.2 Registering An Exit Callback Function
        • 17.4.6.3 Registering An Extension Version String
        • 17.4.6.4 Customized Input Parsers
        • 17.4.6.5 Customized Output Wrappers
        • 17.4.6.6 Customized Two-way Processors
      • 17.4.7 Printing Messages
      • 17.4.8 Updating ERRNO
      • 17.4.9 Requesting Values
      • 17.4.10 Accessing and Updating Parameters
      • 17.4.11 Symbol Table Access
        • 17.4.11.1 Variable Access and Update by Name
        • 17.4.11.2 Variable Access and Update by Cookie
        • 17.4.11.3 Creating and Using Cached Values
      • 17.4.12 Array Manipulation
        • 17.4.12.1 Array Data Types
        • 17.4.12.2 Array Functions
        • 17.4.12.3 Working With All The Elements of an Array
        • 17.4.12.4 How To Create and Populate Arrays
      • 17.4.13 Accessing and Manipulating Redirections
      • 17.4.14 API Variables
        • 17.4.14.1 API Version Constants and Variables
        • 17.4.14.2 GMP and MPFR Version Information
        • 17.4.14.3 Informational Variables
      • 17.4.15 Boilerplate Code
      • 17.4.16 Changes From Version 1 of the API
    • 17.5 How gawk Finds Extensions
    • 17.6 Example: Some File Functions
      • 17.6.1 Using chdir() and stat()
      • 17.6.2 C Code for chdir() and stat()
      • 17.6.3 Integrating the Extensions
    • 17.7 The Sample Extensions in the gawk Distribution
      • 17.7.1 File-Related Functions
      • 17.7.2 Interface to fnmatch()
      • 17.7.3 Interface to fork(), wait(), and waitpid()
      • 17.7.4 Enabling In-Place File Editing
      • 17.7.5 Character and Numeric values: ord() and chr()
      • 17.7.6 Reading Directories
      • 17.7.7 Reversing Output
      • 17.7.8 Two-Way I/O Example
      • 17.7.9 Dumping and Restoring an Array
      • 17.7.10 Reading an Entire File
      • 17.7.11 Extension Time Functions
      • 17.7.12 API Tests
    • 17.8 The gawkextlib Project
    • 17.9 Summary
    • 17.10 Exercises
• Part IV:
  Appendices
  • Appendix A The Evolution of the awk Language
    • A.1 Major Changes Between V7 and SVR3.1
    • A.2 Changes Between SVR3.1 and SVR4
    • A.3 Changes Between SVR4 and POSIX awk
    • A.4 Extensions in Brian Kernighan’s awk
    • A.5 Extensions in gawk Not in POSIX awk
    • A.6 History of gawk Features
    • A.7 Common Extensions Summary
    • A.8 Regexp Ranges and Locales: A Long Sad Story
    • A.9 Major Contributors to gawk
    • A.10 Summary
  • Appendix B Installing gawk
    • B.1 The gawk Distribution
      • B.1.1 Getting the gawk Distribution
      • B.1.2 Extracting the Distribution
      • B.1.3 Contents of the gawk Distribution
    • B.2 Compiling and Installing gawk on Unix-Like Systems
      • B.2.1 Compiling gawk for Unix-Like Systems
        • B.2.1.1 Building With MPFR
      • B.2.2 Shell Startup Files
      • B.2.3 Additional Configuration Options
      • B.2.4 The Configuration Process
      • B.2.5 Compiling from Git
      • B.2.6 Building the Documentation
    • B.3 Installation on Other Operating Systems
      • B.3.1 Installation on MS-Windows
        • B.3.1.1 Installing a Prepared Distribution for MS-Windows Systems
        • B.3.1.2 Compiling gawk for PC Operating Systems
        • B.3.1.3 Using gawk on PC Operating Systems
        • B.3.1.4 Using gawk In The Cygwin Environment
        • B.3.1.5 Using gawk In The MSYS Environment
      • B.3.2 Compiling and Installing gawk on OpenVMS
        • B.3.2.1 Compiling gawk on OpenVMS
        • B.3.2.2 Compiling gawk Dynamic Extensions on OpenVMS
        • B.3.2.3 Installing gawk on OpenVMS
        • B.3.2.4 Running gawk on OpenVMS
        • B.3.2.5 The OpenVMS GNV Project
    • B.4 Reporting Problems and Bugs
      • B.4.1 Defining What Is and What Is Not A Bug
      • B.4.2 Submitting Bug Reports
      • B.4.3 Please Don’t Post Bug Reports to USENET
      • B.4.4 What To Do If You Think There Is A Performance Issue
      • B.4.5 Where To Send Non-bug Questions
      • B.4.6 Reporting Problems with Non-Unix Ports
    • B.5 Other Freely Available awk Implementations
    • B.6 Summary
  • Appendix C Implementation Notes
    • C.1 Downward Compatibility and Debugging
    • C.2 Making Additions to gawk
      • C.2.1 Accessing The gawk Git Repository
      • C.2.2 Adding New Features
      • C.2.3 Porting gawk to a New Operating System
      • C.2.4 Why Generated Files Are Kept In Git
    • C.3 Probable Future Extensions
    • C.4 Some Limitations of the Implementation
    • C.5 Extension API Design
      • C.5.1 Problems With The Old Mechanism
      • C.5.2 Goals For A New Mechanism
      • C.5.3 Other Design Decisions
      • C.5.4 Room For Future Growth
    • C.6 Summary
  • Appendix D Basic Programming Concepts
    • D.1 What a Program Does
    • D.2 Data Values in a Computer
  • Glossary
  • GNU General Public License
  • GNU Free Documentation License
    • ADDENDUM: How to use this License for your documents
  • Index

Michael Brennan
Author of mawk
March 2001

Michael Brennan
Author of mawk
October 2014

Next: A Rose by Any Other Name, Up: Preface   [Contents][Index]

History of awk and gawk ¶

The name awk comes from the initials of its designers: Alfred V. Aho, Peter J. Weinberger, and Brian W. Kernighan. The original version of awk was written in 1977 at AT&T Bell Laboratories. In 1985, a new version made the programming language more powerful, introducing user-defined functions, multiple input streams, and computed regular expressions. This new version became widely available with Unix System V Release 3.1 (1987). The version in System V Release 4 (1989) added some new features and cleaned up the behavior in some of the “dark corners” of the language. The specification for awk in the POSIX Command Language and Utilities standard further clarified the language. Both the gawk designers and the original awk designers at Bell Laboratories provided feedback for the POSIX specification.

Paul Rubin wrote gawk in 1986. Jay Fenlason completed it, with advice from Richard Stallman. John Woods contributed parts of the code as well. In 1988 and 1989, David Trueman, with help from me, thoroughly reworked gawk for compatibility with the newer awk. Circa 1994, I became the primary maintainer. Current development focuses on bug fixes, performance improvements, standards compliance, and, occasionally, new features.

In May 1997, Jürgen Kahrs felt the need for network access from awk, and with a little help from me, set about adding features to do this for gawk. At that time, he also wrote the bulk of TCP/IP Internetworking with gawk (a separate document, available as part of the gawk distribution). His code finally became part of the main gawk distribution with gawk version 3.1.

John Haque rewrote the gawk internals, in the process providing an awk-level debugger. This version became available as gawk version 4.0 in 2011.

See Major Contributors to gawk for a full list of those who have made important contributions to gawk.

Next: Using This Book, Previous: History of awk and gawk, Up: Preface   [Contents][Index]

A Rose by Any Other Name ¶

The awk language has evolved over the years. Full details are provided in The Evolution of the awk Language. The language described in this Web page is often referred to as “new awk.” By analogy, the original version of awk is referred to as “old awk.”

On most current systems, when you run the awk utility you get some version of new awk.⁴ If your system’s standard awk is the old one, you will see something like this if you try the following test program:

$ awk 1 /dev/null
error→ awk: syntax error near line 1
error→ awk: bailing out near line 1

In this case, you should find a version of new awk, or just install gawk!

Throughout this Web page, whenever we refer to a language feature that should be available in any complete implementation of POSIX awk, we simply use the term awk. When referring to a feature that is specific to the GNU implementation, we use the term gawk.

Next: Typographical Conventions, Previous: A Rose by Any Other Name, Up: Preface   [Contents][Index]

Using This Book ¶

The term awk refers to a particular program as well as to the language you use to tell this program what to do. When we need to be careful, we call the language “the awk language,” and the program “the awk utility.” This Web page explains both how to write programs in the awk language and how to run the awk utility. The term “awk program” refers to a program written by you in the awk programming language.

Primarily, this Web page explains the features of awk as defined in the POSIX standard. It does so in the context of the gawk implementation. While doing so, it also attempts to describe important differences between gawk and other awk implementations.⁵ Finally, it notes any gawk features that are not in the POSIX standard for awk.

This Web page has the difficult task of being both a tutorial and a reference. If you are a novice, feel free to skip over details that seem too complex. You should also ignore the many cross-references; they are for the expert user and for the Info and HTML versions of the Web page.

There are sidebars scattered throughout the Web page. They add a more complete explanation of points that are relevant, but not likely to be of interest on first reading. All appear in the index, under the heading “sidebar.”

Most of the time, the examples use complete awk programs. Some of the more advanced sections show only the part of the awk program that illustrates the concept being described.

Although this Web page is aimed principally at people who have not been exposed to awk, there is a lot of information here that even the awk expert should find useful. In particular, the description of POSIX awk and the example programs in A Library of awk Functions, and in Practical awk Programs, should be of interest.

This Web page is split into several parts, as follows:

• Part I describes the awk language and the gawk program in detail. It starts with the basics, and continues through all of the features of awk. It contains the following chapters:
  • Getting Started with awk, provides the essentials you need to know to begin using awk.
  • Running awk and gawk, describes how to run gawk, the meaning of its command-line options, and how it finds awk program source files.
  • Regular Expressions, introduces regular expressions in general, and in particular the flavors supported by POSIX awk and gawk.
  • Reading Input Files, describes how awk reads your data. It introduces the concepts of records and fields, as well as the getline function. I/O redirection is first described here. Network I/O is also briefly introduced here.
  • Printing Output, describes how awk programs can produce output with print and printf.
  • Expressions, describes expressions, which are the basic building blocks for getting most things done in a program.
  • Patterns, Actions, and Variables, describes how to write patterns for matching records, actions for doing something when a record is matched, and the predefined variables awk and gawk use.
  • Arrays in awk, covers awk’s one-and-only data structure: the associative array. Deleting array elements and whole arrays is described, as well as sorting arrays in gawk. The chapter also describes how gawk provides arrays of arrays.
  • Functions, describes the built-in functions awk and gawk provide, as well as how to define your own functions. It also discusses how gawk lets you call functions indirectly.
• Part II shows how to use awk and gawk for problem solving. There is lots of code here for you to read and learn from. This part contains the following chapters:
  • A Library of awk Functions, provides a number of functions meant to be used from main awk programs.
  • Practical awk Programs, provides many sample awk programs.

  Reading these two chapters allows you to see awk solving real problems.

• Part III focuses on features specific to gawk. It contains the following chapters:
  • Advanced Features of gawk, describes a number of advanced features. Of particular note are the abilities to control the order of array traversal, have two-way communications with another process, perform TCP/IP networking, and profile your awk programs.
  • Internationalization with gawk, describes special features for translating program messages into different languages at runtime.
  • Debugging awk Programs, describes the gawk debugger.
  • Namespaces in gawk, describes how gawk allows variables and/or functions of the same name to be in different namespaces.
  • Arithmetic and Arbitrary-Precision Arithmetic with gawk, describes advanced arithmetic facilities.
  • Writing Extensions for gawk, describes how to add new variables and functions to gawk by writing extensions in C or C++.
• Part IV provides the appendices, the Glossary, and two licenses that cover the gawk source code and this Web page, respectively. It contains the following appendices:
  • The Evolution of the awk Language, describes how the awk language has evolved since its first release to the present. It also describes how gawk has acquired features over time.
  • Installing gawk, describes how to get gawk, how to compile it on POSIX-compatible systems, and how to compile and use it on different non-POSIX systems. It also describes how to report bugs in gawk and where to get other freely available awk implementations.
  • Implementation Notes, describes how to disable gawk’s extensions, as well as how to contribute new code to gawk, and some possible future directions for gawk development.
  • Basic Programming Concepts, provides some very cursory background material for those who are completely unfamiliar with computer programming.
  • The Glossary, defines most, if not all, of the significant terms used throughout the Web page. If you find terms that you aren’t familiar with, try looking them up here.
  • GNU General Public License, and GNU Free Documentation License, present the licenses that cover the gawk source code and this Web page, respectively.

Next: The GNU Project and This Book, Previous: Using This Book, Up: Preface   [Contents][Index]

Typographical Conventions ¶

This Web page is written in Texinfo, the GNU documentation formatting language. A single Texinfo source file is used to produce both the printed and online versions of the documentation. Because of this, the typographical conventions are slightly different than in other books you may have read.

Examples you would type at the command line are preceded by the common shell primary and secondary prompts, ‘$’ and ‘>’, respectively. Input that you type is shown like this. Output from the command is preceded by the glyph “-|”. This typically represents the command’s standard output. Error messages and other output on the command’s standard error are preceded by the glyph “error→”. For example:

$ echo hi on stdout
-| hi on stdout
$ echo hello on stderr 1>&2
error→ hello on stderr

In the text, almost anything related to programming, such as command names, variable and function names, and string, numeric and regexp constants appear in this font. Code fragments appear in the same font and quoted, ‘like this’. Things that are replaced by the user or programmer appear in this font. Options look like this: -f. File names are indicated like this: /path/to/ourfile. Some things are emphasized like this, and if a point needs to be made strongly, it is done like this. The first occurrence of a new term is usually its definition and appears in the same font as the previous occurrence of “definition” in this sentence.

Characters that you type at the keyboard look like this. In particular, there are special characters called “control characters.” These are characters that you type by holding down both the CONTROL key and another key, at the same time. For example, a Ctrl-d is typed by first pressing and holding the CONTROL key, next pressing the d key, and finally releasing both keys.

For the sake of brevity, throughout this Web page, we refer to Brian Kernighan’s version of awk as “BWK awk.” (See Other Freely Available awk Implementations for information on his and other versions.)

• Dark Corners

Next: How to Contribute, Previous: Typographical Conventions, Up: Preface   [Contents][Index]

The GNU Project and This Book ¶

The Free Software Foundation (FSF) is a nonprofit organization dedicated to the production and distribution of freely distributable software. It was founded by Richard M. Stallman, the author of the original Emacs editor. GNU Emacs is the most widely used version of Emacs today.

The GNU⁶ Project is an ongoing effort on the part of the Free Software Foundation to create a complete, freely distributable, POSIX-compliant computing environment. The FSF uses the GNU General Public License (GPL) to ensure that its software’s source code is always available to the end user. A copy of the GPL is included in this Web page for your reference (see GNU General Public License). The GPL applies to the C language source code for gawk. To find out more about the FSF and the GNU Project online, see the GNU Project’s home page. This Web page may also be read from GNU’s website.

A shell, an editor (Emacs), highly portable optimizing C, C++, and Objective-C compilers, a symbolic debugger and dozens of large and small utilities (such as gawk), have all been completed and are freely available. The GNU operating system kernel (the HURD), has been released but remains in an early stage of development.

Until the GNU operating system is more fully developed, you should consider using GNU/Linux, a freely distributable, Unix-like operating system for Intel, Power Architecture, Sun SPARC, IBM S/390, and other systems.⁷ Many GNU/Linux distributions are available for download from the Internet.

The Web page you are reading is actually free—at least, the information in it is free to anyone. The machine-readable source code for the Web page comes with gawk. (Take a moment to check the Free Documentation License in GNU Free Documentation License.)

The Web page itself has gone through multiple previous editions. Paul Rubin wrote the very first draft of The GAWK Manual; it was around 40 pages long. Diane Close and Richard Stallman improved it, yielding a version that was around 90 pages and barely described the original, “old” version of awk.

I started working with that version in the fall of 1988. As work on it progressed, the FSF published several preliminary versions (numbered 0.x). In 1996, edition 1.0 was released with gawk 3.0.0. The FSF published the first two editions under the title The GNU Awk User’s Guide.

This edition maintains the basic structure of the previous editions. For FSF edition 4.0, the content was thoroughly reviewed and updated. All references to gawk versions prior to 4.0 were removed. Of significant note for that edition was the addition of Debugging awk Programs.

For FSF edition 5.0, the content has been reorganized into parts, and the major new additions are Arithmetic and Arbitrary-Precision Arithmetic with gawk, and Writing Extensions for gawk.

This Web page will undoubtedly continue to evolve. If you find an error in the Web page, please report it! See Reporting Problems and Bugs for information on submitting problem reports electronically.

Next: Acknowledgments, Previous: The GNU Project and This Book, Up: Preface   [Contents][Index]

How to Contribute ¶

As the maintainer of GNU awk, I once thought that I would be able to manage a collection of publicly available awk programs and I even solicited contributions. Making things available on the Internet helps keep the gawk distribution down to manageable size.

The initial collection of material, such as it is, is still available at ftp://ftp.freefriends.org/arnold/Awkstuff.

In the hopes of doing something broader, I acquired the awklang.org domain. Late in 2017, a volunteer took on the task of managing it.

If you have written an interesting awk program that you would like to share with the rest of the world, please see http://www.awklang.org and use the “Contact” link.

If you have written a gawk extension, please see The gawkextlib Project.

Previous: How to Contribute, Up: Preface   [Contents][Index]

Acknowledgments ¶

The initial draft of The GAWK Manual had the following acknowledgments:

Many people need to be thanked for their assistance in producing this manual. Jay Fenlason contributed many ideas and sample programs. Richard Mlynarik and Robert Chassell gave helpful comments on drafts of this manual. The paper A Supplemental Document for AWK by John W. Pierce of the Chemistry Department at UC San Diego, pinpointed several issues relevant both to awk implementation and to this manual, that would otherwise have escaped us.

I would like to acknowledge Richard M. Stallman, for his vision of a better world and for his courage in founding the FSF and starting the GNU Project.

Earlier editions of this Web page had the following acknowledgements:

The following people (in alphabetical order) provided helpful comments on various versions of this book: Rick Adams, Dr. Nelson H.F. Beebe, Karl Berry, Dr. Michael Brennan, Rich Burridge, Claire Cloutier, Diane Close, Scott Deifik, Christopher (“Topher”) Eliot, Jeffrey Friedl, Dr. Darrel Hankerson, Michal Jaegermann, Dr. Richard J. LeBlanc, Michael Lijewski, Pat Rankin, Miriam Robbins, Mary Sheehan, and Chuck Toporek.

Robert J. Chassell provided much valuable advice on the use of Texinfo. He also deserves special thanks for convincing me not to title this Web page How to Gawk Politely. Karl Berry helped significantly with the TeX part of Texinfo.

I would like to thank Marshall and Elaine Hartholz of Seattle and Dr. Bert and Rita Schreiber of Detroit for large amounts of quiet vacation time in their homes, which allowed me to make significant progress on this Web page and on gawk itself.

Phil Hughes of SSC contributed in a very important way by loaning me his laptop GNU/Linux system, not once, but twice, which allowed me to do a lot of work while away from home.

David Trueman deserves special credit; he has done a yeoman job of evolving gawk so that it performs well and without bugs. Although he is no longer involved with gawk, working with him on this project was a significant pleasure.

The intrepid members of the GNITS mailing list, and most notably Ulrich Drepper, provided invaluable help and feedback for the design of the internationalization features.

Chuck Toporek, Mary Sheehan, and Claire Cloutier of O’Reilly & Associates contributed significant editorial help for this Web page for the 3.1 release of gawk.

Dr. Nelson Beebe, Andreas Buening, Dr. Manuel Collado, Antonio Colombo, Stephen Davies, Scott Deifik, Akim Demaille, Daniel Richard G., Juan Manuel Guerrero, Darrel Hankerson, Michal Jaegermann, Jürgen Kahrs, Stepan Kasal, John Malmberg, Chet Ramey, Pat Rankin, Andrew Schorr, Corinna Vinschen, and Eli Zaretskii (in alphabetical order) make up the current gawk “crack portability team.” Without their hard work and help, gawk would not be nearly the robust, portable program it is today. It has been and continues to be a pleasure working with this team of fine people.

Notable code and documentation contributions were made by a number of people. See Major Contributors to gawk for the full list.

Thanks to Michael Brennan for the Forewords.

Thanks to Patrice Dumas for the new makeinfo program. Thanks to Karl Berry for his past work on Texinfo, and to Gavin Smith, who continues to work to improve the Texinfo markup language.

Robert P.J. Day, Michael Brennan, and Brian Kernighan kindly acted as reviewers for the 2015 edition of this Web page. Their feedback helped improve the final work.

I would also like to thank Brian Kernighan for his invaluable assistance during the testing and debugging of gawk, and for his ongoing help and advice in clarifying numerous points about the language. We could not have done nearly as good a job on either gawk or its documentation without his help.

Brian is in a class by himself as a programmer and technical author. I have to thank him (yet again) for his ongoing friendship and for being a role model to me for over 30 years! Having him as a reviewer is an exciting privilege. It has also been extremely humbling...

I must thank my wonderful wife, Miriam, for her patience through the many versions of this project, for her proofreading, and for sharing me with the computer. I would like to thank my parents for their love, and for the grace with which they raised and educated me. Finally, I also must acknowledge my gratitude to G-d, for the many opportunities He has sent my way, as well as for the gifts He has given me with which to take advantage of those opportunities.

Arnold Robbins
Nof Ayalon
Israel
March, 2020

pattern { action }
pattern { action }
...

Next: Data files for the Examples, Up: Getting Started with awk   [Contents][Index]

1.1 How to Run awk Programs ¶

There are several ways to run an awk program. If the program is short, it is easiest to include it in the command that runs awk, like this:

awk 'program' input-file1 input-file2 ...

When the program is long, it is usually more convenient to put it in a file and run it with a command like this:

awk -f program-file input-file1 input-file2 ...

This section discusses both mechanisms, along with several variations of each.

• One-Shot Throwaway awk Programs
• Running awk Without Input Files
• Running Long Programs
• Executable awk Programs
• Comments in awk Programs
• Shell Quoting Issues

awk 'program' input-file1 input-file2 ...

awk 'program'

$ awk 'BEGIN { print "Don\47t Panic!" }'
-| Don't Panic!

$ awk '{ print }'
Now is the time for all good men
-| Now is the time for all good men
to come to the aid of their country.
-| to come to the aid of their country.
Four score and seven years ago, ...
-| Four score and seven years ago, ...
What, me worry?
-| What, me worry?
Ctrl-d

awk -f source-file input-file1 input-file2 ...

BEGIN { print "Don't Panic!" }

awk -f advice

awk 'BEGIN { print "Don\47t Panic!" }'

#! /bin/awk -f
BEGIN { print "Don't Panic!" }

$ chmod +x advice
$ ./advice
-| Don't Panic!

awk 'program text' input-file1 input-file2 ...

$ echo ""

$ awk 'BEGIN { print "Here is a single quote <'"'"'>" }'
-| Here is a single quote <'>

$ awk 'BEGIN { print "Here is a single quote <'\''>" }'
-| Here is a single quote <'>

$ awk "BEGIN { print \"Here is a single quote <'>\" }"
-| Here is a single quote <'>

$ awk 'BEGIN { print "Here is a single quote <\47>" }'
-| Here is a single quote <'>
$ awk 'BEGIN { print "Here is a double quote <\42>" }'
-| Here is a double quote <">

$ awk -v sq="'" 'BEGIN { print "Here is a single quote <" sq ">" }'
-| Here is a single quote <'>

{ print "\"" $0 "\"" }

gawk "{ print \"\042\" $0 \"\042\" }" file

gawk "{ print \"\\\"\" $0 \"\\\"\" }" file

Next: Some Simple Examples, Previous: How to Run awk Programs, Up: Getting Started with awk   [Contents][Index]

1.2 Data files for the Examples ¶

Many of the examples in this Web page take their input from two sample data files. The first, mail-list, represents a list of peoples’ names together with their email addresses and information about those people. The second data file, called inventory-shipped, contains information about monthly shipments. In both files, each line is considered to be one record.

In mail-list, each record contains the name of a person, his/her phone number, his/her email address, and a code for his/her relationship with the author of the list. The columns are aligned using spaces. An ‘A’ in the last column means that the person is an acquaintance. An ‘F’ in the last column means that the person is a friend. An ‘R’ means that the person is a relative:

Amelia       555-5553     amelia.zodiacusque@gmail.com    F
Anthony      555-3412     anthony.asserturo@hotmail.com   A
Becky        555-7685     becky.algebrarum@gmail.com      A
Bill         555-1675     bill.drowning@hotmail.com       A
Broderick    555-0542     broderick.aliquotiens@yahoo.com R
Camilla      555-2912     camilla.infusarum@skynet.be     R
Fabius       555-1234     fabius.undevicesimus@ucb.edu    F
Julie        555-6699     julie.perscrutabor@skeeve.com   F
Martin       555-6480     martin.codicibus@hotmail.com    A
Samuel       555-3430     samuel.lanceolis@shu.edu        A
Jean-Paul    555-2127     jeanpaul.campanorum@nyu.edu     R

The data file inventory-shipped represents information about shipments during the year. Each record contains the month, the number of green crates shipped, the number of red boxes shipped, the number of orange bags shipped, and the number of blue packages shipped, respectively. There are 16 entries, covering the 12 months of last year and the first four months of the current year. An empty line separates the data for the two years:

Jan  13  25  15 115
Feb  15  32  24 226
Mar  15  24  34 228
Apr  31  52  63 420
May  16  34  29 208
Jun  31  42  75 492
Jul  24  34  67 436
Aug  15  34  47 316
Sep  13  55  37 277
Oct  29  54  68 525
Nov  20  87  82 577
Dec  17  35  61 401
Jan  21  36  64 620
Feb  26  58  80 652
Mar  24  75  70 495
Apr  21  70  74 514

The sample files are included in the gawk distribution, in the directory awklib/eg/data.

Next: An Example with Two Rules, Previous: Data files for the Examples, Up: Getting Started with awk   [Contents][Index]

1.3 Some Simple Examples ¶

The following command runs a simple awk program that searches the input file mail-list for the character string ‘li’ (a grouping of characters is usually called a string; the term string is based on similar usage in English, such as “a string of pearls” or “a string of cars in a train”):

awk '/li/ { print $0 }' mail-list

When lines containing ‘li’ are found, they are printed because ‘print $0’ means print the current line. (Just ‘print’ by itself means the same thing, so we could have written that instead.)

You will notice that slashes (‘/’) surround the string ‘li’ in the awk program. The slashes indicate that ‘li’ is the pattern to search for. This type of pattern is called a regular expression, which is covered in more detail later (see Regular Expressions). The pattern is allowed to match parts of words. There are single quotes around the awk program so that the shell won’t interpret any of it as special shell characters.

Here is what this program prints:

$ awk '/li/ { print $0 }' mail-list
-| Amelia       555-5553     amelia.zodiacusque@gmail.com    F
-| Broderick    555-0542     broderick.aliquotiens@yahoo.com R
-| Julie        555-6699     julie.perscrutabor@skeeve.com   F
-| Samuel       555-3430     samuel.lanceolis@shu.edu        A

In an awk rule, either the pattern or the action can be omitted, but not both. If the pattern is omitted, then the action is performed for every input line. If the action is omitted, the default action is to print all lines that match the pattern.

Thus, we could leave out the action (the print statement and the braces) in the previous example and the result would be the same: awk prints all lines matching the pattern ‘li’. By comparison, omitting the print statement but retaining the braces makes an empty action that does nothing (i.e., no lines are printed).

Many practical awk programs are just a line or two long. Following is a collection of useful, short programs to get you started. Some of these programs contain constructs that haven’t been covered yet. (The description of the program will give you a good idea of what is going on, but you’ll need to read the rest of the Web page to become an awk expert!) Most of the examples use a data file named data. This is just a placeholder; if you use these programs yourself, substitute your own file names for data.

Some of the following examples use the output of ‘ls -l’ as input. ls is a system command that gives you a listing of the files in a directory. With the -l option, this listing includes each file’s size and the date the file was last modified. Its output looks like this:

-rw-r--r--  1 arnold   user   1933 Nov  7 13:05 Makefile
-rw-r--r--  1 arnold   user  10809 Nov  7 13:03 awk.h
-rw-r--r--  1 arnold   user    983 Apr 13 12:14 awk.tab.h
-rw-r--r--  1 arnold   user  31869 Jun 15 12:20 awkgram.y
-rw-r--r--  1 arnold   user  22414 Nov  7 13:03 awk1.c
-rw-r--r--  1 arnold   user  37455 Nov  7 13:03 awk2.c
-rw-r--r--  1 arnold   user  27511 Dec  9 13:07 awk3.c
-rw-r--r--  1 arnold   user   7989 Nov  7 13:03 awk4.c

The first field contains read-write permissions, the second field contains the number of links to the file, and the third field identifies the file’s owner. The fourth field identifies the file’s group. The fifth field contains the file’s size in bytes. The sixth, seventh, and eighth fields contain the month, day, and time, respectively, that the file was last modified. Finally, the ninth field contains the file name.

For future reference, note that there is often more than one way to do things in awk. At some point, you may want to look back at these examples and see if you can come up with different ways to do the same things shown here:

• Print every line that is longer than 80 characters:

  awk 'length($0) > 80' data
  

  The sole rule has a relational expression as its pattern and has no action—so it uses the default action, printing the record.

• Print the length of the longest input line:

  awk '{ if (length($0) > max) max = length($0) }
       END { print max }' data
  

  The code associated with END executes after all input has been read; it’s the other side of the coin to BEGIN.

• Print the length of the longest line in data:

  expand data | awk '{ if (x < length($0)) x = length($0) }
                     END { print "maximum line length is " x }'
  

  This example differs slightly from the previous one: the input is processed by the expand utility to change TABs into spaces, so the widths compared are actually the right-margin columns, as opposed to the number of input characters on each line.

• Print every line that has at least one field:

  awk 'NF > 0' data
  

  This is an easy way to delete blank lines from a file (or rather, to create a new file similar to the old file but from which the blank lines have been removed).

• Print seven random numbers from 0 to 100, inclusive:

  awk 'BEGIN { for (i = 1; i <= 7; i++)
                   print int(101 * rand()) }'
  

• Print the total number of bytes used by files:

  ls -l files | awk '{ x += $5 }
                     END { print "total bytes: " x }'
  

• Print the total number of kilobytes used by files:

  ls -l files | awk '{ x += $5 }
     END { print "total K-bytes:", x / 1024 }'
  

• Print a sorted list of the login names of all users:

  awk -F: '{ print $1 }' /etc/passwd | sort
  

• Count the lines in a file:

  awk 'END { print NR }' data
  

• Print the even-numbered lines in the data file:

  awk 'NR % 2 == 0' data
  

  If you used the expression ‘NR % 2 == 1’ instead, the program would print the odd-numbered lines.

Next: A More Complex Example, Previous: Some Simple Examples, Up: Getting Started with awk   [Contents][Index]

1.4 An Example with Two Rules ¶

The awk utility reads the input files one line at a time. For each line, awk tries the patterns of each rule. If several patterns match, then several actions execute in the order in which they appear in the awk program. If no patterns match, then no actions run.

After processing all the rules that match the line (and perhaps there are none), awk reads the next line. (However, see The next Statement and also see The nextfile Statement.) This continues until the program reaches the end of the file. For example, the following awk program contains two rules:

/12/  { print $0 }
/21/  { print $0 }

The first rule has the string ‘12’ as the pattern and ‘print $0’ as the action. The second rule has the string ‘21’ as the pattern and also has ‘print $0’ as the action. Each rule’s action is enclosed in its own pair of braces.

This program prints every line that contains the string ‘12’ or the string ‘21’. If a line contains both strings, it is printed twice, once by each rule.

This is what happens if we run this program on our two sample data files, mail-list and inventory-shipped:

$ awk '/12/ { print $0 }
>      /21/ { print $0 }' mail-list inventory-shipped
-| Anthony      555-3412     anthony.asserturo@hotmail.com   A
-| Camilla      555-2912     camilla.infusarum@skynet.be     R
-| Fabius       555-1234     fabius.undevicesimus@ucb.edu    F
-| Jean-Paul    555-2127     jeanpaul.campanorum@nyu.edu     R
-| Jean-Paul    555-2127     jeanpaul.campanorum@nyu.edu     R
-| Jan  21  36  64 620
-| Apr  21  70  74 514

Note how the line beginning with ‘Jean-Paul’ in mail-list was printed twice, once for each rule.

Next: awk Statements Versus Lines, Previous: An Example with Two Rules, Up: Getting Started with awk   [Contents][Index]

1.5 A More Complex Example ¶

Now that we’ve mastered some simple tasks, let’s look at what typical awk programs do. This example shows how awk can be used to summarize, select, and rearrange the output of another utility. It uses features that haven’t been covered yet, so don’t worry if you don’t understand all the details:

ls -l | awk '$6 == "Nov" { sum += $5 }
             END { print sum }'

This command prints the total number of bytes in all the files in the current directory that were last modified in November (of any year).

As a reminder, the output of ‘ls -l’ gives you a listing of the files in a directory, including each file’s size and the date the file was last modified. The first field contains read-write permissions, the second field contains the number of links to the file, and the third field identifies the file’s owner. The fourth field identifies the file’s group. The fifth field contains the file’s size in bytes. The sixth, seventh, and eighth fields contain the month, day, and time, respectively, that the file was last modified. Finally, the ninth field contains the file name.

The ‘$6 == "Nov"’ in our awk program is an expression that tests whether the sixth field of the output from ‘ls -l’ matches the string ‘Nov’. Each time a line has the string ‘Nov’ for its sixth field, awk performs the action ‘sum += $5’. This adds the fifth field (the file’s size) to the variable sum. As a result, when awk has finished reading all the input lines, sum is the total of the sizes of the files whose lines matched the pattern. (This works because awk variables are automatically initialized to zero.)

After the last line of output from ls has been processed, the END rule executes and prints the value of sum. In this example, the value of sum is 80600.

These more advanced awk techniques are covered in later sections (see Actions). Before you can move on to more advanced awk programming, you have to know how awk interprets your input and displays your output. By manipulating fields and using print statements, you can produce some very useful and impressive-looking reports.

Next: Other Features of awk, Previous: A More Complex Example, Up: Getting Started with awk   [Contents][Index]

1.6 awk Statements Versus Lines ¶

Most often, each line in an awk program is a separate statement or separate rule, like this:

awk '/12/  { print $0 }
     /21/  { print $0 }' mail-list inventory-shipped

However, gawk ignores newlines after any of the following symbols and keywords:

,    {    ?    :    ||    &&    do    else

A newline at any other point is considered the end of the statement.⁹

If you would like to split a single statement into two lines at a point where a newline would terminate it, you can continue it by ending the first line with a backslash character (‘\’). The backslash must be the final character on the line in order to be recognized as a continuation character. A backslash followed by a newline is allowed anywhere in the statement, even in the middle of a string or regular expression. For example:

awk '/This regular expression is too long, so continue it\
 on the next line/ { print $1 }'

We have generally not used backslash continuation in our sample programs. gawk places no limit on the length of a line, so backslash continuation is never strictly necessary; it just makes programs more readable. For this same reason, as well as for clarity, we have kept most statements short in the programs presented throughout the Web page.

Backslash continuation is most useful when your awk program is in a separate source file instead of entered from the command line. You should also note that many awk implementations are more particular about where you may use backslash continuation. For example, they may not allow you to split a string constant using backslash continuation. Thus, for maximum portability of your awk programs, it is best not to split your lines in the middle of a regular expression or a string.

CAUTION: Backslash continuation does not work as described with the C shell. It works for awk programs in files and for one-shot programs, provided you are using a POSIX-compliant shell, such as the Unix Bourne shell or Bash. But the C shell behaves differently! There you must use two backslashes in a row, followed by a newline. Note also that when using the C shell, every newline in your awk program must be escaped with a backslash. To illustrate:

% awk 'BEGIN { \
?   print \\
?       "hello, world" \
? }'
-| hello, world

Here, the ‘%’ and ‘?’ are the C shell’s primary and secondary prompts, analogous to the standard shell’s ‘$’ and ‘>’.

Compare the previous example to how it is done with a POSIX-compliant shell:

$ awk 'BEGIN {
>   print \
>       "hello, world"
> }'
-| hello, world

awk is a line-oriented language. Each rule’s action has to begin on the same line as the pattern. To have the pattern and action on separate lines, you must use backslash continuation; there is no other option.

Another thing to keep in mind is that backslash continuation and comments do not mix. As soon as awk sees the ‘#’ that starts a comment, it ignores everything on the rest of the line. For example:

$ gawk 'BEGIN { print "don\47t panic" # a friendly \
>                                    BEGIN rule
> }'
error→ gawk: cmd. line:2:                BEGIN rule
error→ gawk: cmd. line:2:                ^ syntax error

In this case, it looks like the backslash would continue the comment onto the next line. However, the backslash-newline combination is never even noticed because it is “hidden” inside the comment. Thus, the BEGIN is noted as a syntax error.

Backslash continuation comes into play in an additional, unexpected situation. Consider:

gawk -F'\
a' '...'

This command line assigns a value to FS. But what value? There are several possibilities, and in fact different versions of awk do different things. gawk treats this as if it were written:

BEGIN { FS = "\
a"
}
...

In short, the backslash and newline are removed, assigning "a" to FS. This same treatment applies to variable assignments made with the -v option (see Command-Line Options) and to regular command-line variable assignments (see Assigning Variables on the Command Line).

If you’re interested, see https://lists.gnu.org/archive/html/bug-gawk/2022-10/msg00025.html for a source code patch that allows lines to be continued when inside parentheses. This patch was not added to gawk since it would quietly decrease the portability of awk programs.

When awk statements within one rule are short, you might want to put more than one of them on a line. This is accomplished by separating the statements with a semicolon (‘;’). This also applies to the rules themselves. Thus, the program shown at the start of this section could also be written this way:

/12/ { print $0 } ; /21/ { print $0 }

NOTE: The requirement that states that rules on the same line must be separated with a semicolon was not in the original awk language; it was added for consistency with the treatment of statements within an action.

Next: When to Use awk, Previous: awk Statements Versus Lines, Up: Getting Started with awk   [Contents][Index]

1.7 Other Features of awk ¶

The awk language provides a number of predefined, or built-in, variables that your programs can use to get information from awk. There are other variables your program can set as well to control how awk processes your data.

In addition, awk provides a number of built-in functions for doing common computational and string-related operations. gawk provides built-in functions for working with timestamps, performing bit manipulation, for runtime string translation (internationalization), determining the type of a variable, and array sorting.

As we develop our presentation of the awk language, we will introduce most of the variables and many of the functions. They are described systematically in Predefined Variables and in Built-in Functions.

Next: Summary, Previous: Other Features of awk, Up: Getting Started with awk   [Contents][Index]

1.8 When to Use awk ¶

Now that you’ve seen some of what awk can do, you might wonder how awk could be useful for you. By using utility programs, advanced patterns, field separators, arithmetic statements, and other selection criteria, you can produce much more complex output. The awk language is very useful for producing reports from large amounts of raw data, such as summarizing information from the output of other utility programs like ls. (See A More Complex Example.)

Programs written with awk are usually much smaller than they would be in other languages. This makes awk programs easy to compose and use. Often, awk programs can be quickly composed at your keyboard, used once, and thrown away. Because awk programs are interpreted, you can avoid the (usually lengthy) compilation part of the typical edit-compile-test-debug cycle of software development.

Complex programs have been written in awk, including a complete retargetable assembler for eight-bit microprocessors (see Glossary, for more information), and a microcode assembler for a special-purpose Prolog computer. The original awk’s capabilities were strained by tasks of such complexity, but modern versions are more capable.

If you find yourself writing awk scripts of more than, say, a few hundred lines, you might consider using a different programming language. The shell is good at string and pattern matching; in addition, it allows powerful use of the system utilities. Python offers a nice balance between high-level ease of programming and access to system facilities.¹⁰

Previous: When to Use awk, Up: Getting Started with awk   [Contents][Index]

1.9 Summary ¶

• Programs in awk consist of pattern–action pairs.
• An action without a pattern always runs. The default action for a pattern without one is ‘{ print $0 }’.
• Use either ‘awk 'program' files’ or ‘awk -f program-file files’ to run awk.
• You may use the special ‘#!’ header line to create awk programs that are directly executable.
• Comments in awk programs start with ‘#’ and continue to the end of the same line.
• Be aware of quoting issues when writing awk programs as part of a larger shell script (or MS-Windows batch file).
• You may use backslash continuation to continue a source line. Lines are automatically continued after a comma, open brace, question mark, colon, ‘||’, ‘&&’, do, and else.

Next: Command-Line Options, Up: Running awk and gawk   [Contents][Index]

2.1 Invoking awk ¶

There are two ways to run awk—with an explicit program or with one or more program files. Here are templates for both of them; items enclosed in […] in these templates are optional:

awk [options] -f progfile [--] file ...
awk [options] [--] 'program' file ...

In addition to traditional one-letter POSIX-style options, gawk also supports GNU long options.

It is possible to invoke awk with an empty program:

awk '' datafile1 datafile2

Doing so makes little sense, though; awk exits silently when given an empty program. (d.c.) If --lint has been specified on the command line, gawk issues a warning that the program is empty.

Next: Other Command-Line Arguments, Previous: Invoking awk, Up: Running awk and gawk   [Contents][Index]

2.2 Command-Line Options ¶

Options begin with a dash and consist of a single character. GNU-style long options consist of two dashes and a keyword. The keyword can be abbreviated, as long as the abbreviation allows the option to be uniquely identified. If the option takes an argument, either the keyword is immediately followed by an equals sign (‘=’) and the argument’s value, or the keyword and the argument’s value are separated by whitespace (spaces or TABs). If a particular option with a value is given more than once, it is (usually) the last value that counts.

Each long option for gawk has a corresponding POSIX-style short option. The long and short options are interchangeable in all contexts. The following list describes options mandated by the POSIX standard:

-F fs ¶

--field-separator fs

Set the FS variable to fs (see Specifying How Fields Are Separated).

-f source-file ¶

--file source-file

Read the awk program source from source-file instead of in the first nonoption argument. This option may be given multiple times; the awk program consists of the concatenation of the contents of each specified source-file.

Files named with -f are treated as if they had ‘@namespace "awk"’ at their beginning. See Changing The Namespace, for more information on this advanced feature.

-v var=val ¶

--assign var=val

Set the variable var to the value val before execution of the program begins. Such variable values are available inside the BEGIN rule (see Other Command-Line Arguments).

The -v option can only set one variable, but it can be used more than once, setting another variable each time, like this: ‘awk -v foo=1 -v bar=2 …’.

CAUTION: Using -v to set the values of the built-in variables may lead to surprising results. awk will reset the values of those variables as it needs to, possibly ignoring any initial value you may have given.

-W gawk-opt ¶

Provide an implementation-specific option. This is the POSIX convention for providing implementation-specific options. These options also have corresponding GNU-style long options. Note that the long options may be abbreviated, as long as the abbreviations remain unique. The full list of gawk-specific options is provided next.

-- ¶

Signal the end of the command-line options. The following arguments are not treated as options even if they begin with ‘-’. This interpretation of -- follows the POSIX argument parsing conventions.

This is useful if you have file names that start with ‘-’, or in shell scripts, if you have file names that will be specified by the user that could start with ‘-’. It is also useful for passing options on to the awk program; see Processing Command-Line Options.

The following list describes gawk-specific options:

-b ¶

--characters-as-bytes

Cause gawk to treat all input data as single-byte characters. In addition, all output written with print or printf is treated as single-byte characters.

Normally, gawk follows the POSIX standard and attempts to process its input data according to the current locale (see Where You Are Makes a Difference). This can often involve converting multibyte characters into wide characters (internally), and can lead to problems or confusion if the input data does not contain valid multibyte characters. This option is an easy way to tell gawk, “Hands off my data!”

-c ¶

--traditional

Specify compatibility mode, in which the GNU extensions to the awk language are disabled, so that gawk behaves just like BWK awk. See Extensions in gawk Not in POSIX awk, which summarizes the extensions. Also see Downward Compatibility and Debugging.

-C ¶

--copyright

Print the short version of the General Public License and then exit.

-d[file] ¶

--dump-variables[=file]

Print a sorted list of global variables, their types, and final values to file. If no file is provided, print this list to a file named awkvars.out in the current directory. No space is allowed between the -d and file, if file is supplied.

Having a list of all global variables is a good way to look for typographical errors in your programs. You would also use this option if you have a large program with a lot of functions, and you want to be sure that your functions don’t inadvertently use global variables that you meant to be local. (This is a particularly easy mistake to make with simple variable names like i, j, etc.)

-D[file] ¶

--debug[=file]

Enable debugging of awk programs (see Introduction to the gawk Debugger). By default, the debugger reads commands interactively from the keyboard (standard input). The optional file argument allows you to specify a file with a list of commands for the debugger to execute noninteractively. No space is allowed between the -D and file, if file is supplied.

-e program-text ¶

--source program-text

Provide program source code in the program-text. This option allows you to mix source code in files with source code that you enter on the command line. This is particularly useful when you have library functions that you want to use from your command-line programs (see The AWKPATH Environment Variable).

Note that gawk treats each string as if it ended with a newline character (even if it doesn’t). This makes building the total program easier.

CAUTION: Prior to version 5.0, there was no requirement that each program-text be a full syntactic unit. I.e., the following worked:

$ gawk -e 'BEGIN { a = 5 ;' -e 'print a }'
-| 5

However, this is no longer true. If you have any scripts that rely upon this feature, you should revise them.

This is because each program-text is treated as if it had ‘@namespace "awk"’ at its beginning. See Changing The Namespace, for more information.

-E file ¶

--exec file

Similar to -f, read awk program text from file. There are two differences from -f:

• This option terminates option processing; anything else on the command line is passed on directly to the awk program.
• Command-line variable assignments of the form ‘var=value’ are disallowed.

This option is particularly necessary for World Wide Web CGI applications that pass arguments through the URL; using this option prevents a malicious (or other) user from passing in options, assignments, or awk source code (via -e) to the CGI application.¹¹ This option should be used with ‘#!’ scripts (see Executable awk Programs), like so:

#! /usr/local/bin/gawk -E
awk program here ...

-g ¶

--gen-pot

Analyze the source program and generate a GNU gettext portable object template file on standard output for all string constants that have been marked for translation. See Internationalization with gawk, for information about this option.

-h ¶

--help

Print a “usage” message summarizing the short- and long-style options that gawk accepts and then exit.

-i source-file ¶

--include source-file

Read an awk source library from source-file. This option is completely equivalent to using the @include directive inside your program. It is very similar to the -f option, but there are two important differences. First, when -i is used, the program source is not loaded if it has been previously loaded, whereas with -f, gawk always loads the file. Second, because this option is intended to be used with code libraries, gawk does not recognize such files as constituting main program input. Thus, after processing an -i argument, gawk still expects to find the main source code via the -f option or on the command line.

Files named with -i are treated as if they had ‘@namespace "awk"’ at their beginning. See Changing The Namespace, for more information.

-I ¶

--trace

Print the internal byte code names as they are executed when running the program. The trace is printed to standard error. Each “op code” is preceded by a + sign in the output.

-k ¶

--csv

Enable special processing for files with comma separated values (CSV). See Working With Comma Separated Value Files. This option cannot be used with --posix. Attempting to do causes a fatal error.

-l ext ¶

--load ext

Load a dynamic extension named ext. Extensions are stored as system shared libraries. This option searches for the library using the AWKLIBPATH environment variable. The correct library suffix for your platform will be supplied by default, so it need not be specified in the extension name. The extension initialization routine should be named dl_load(). An alternative is to use the @load directive inside the program to load a shared library. This advanced feature is described in detail in Writing Extensions for gawk.

-L[value] ¶

--lint[=value]

Warn about constructs that are dubious or nonportable to other awk implementations. No space is allowed between the -L and value, if value is supplied. Some warnings are issued when gawk first reads your program. Others are issued at runtime, as your program executes. The optional argument may be one of the following:

fatal

Cause lint warnings become fatal errors. This may be drastic, but its use will certainly encourage the development of cleaner awk programs.

invalid

Only issue warnings about things that are actually invalid are issued. (This is not fully implemented yet.)

no-ext

Disable warnings about gawk extensions.

Some warnings are only printed once, even if the dubious constructs they warn about occur multiple times in your awk program. Thus, when eliminating problems pointed out by --lint, you should take care to search for all occurrences of each inappropriate construct. As awk programs are usually short, doing so is not burdensome.

-M ¶

--bignum

Select arbitrary-precision arithmetic on numbers. This option has no effect if gawk is not compiled to use the GNU MPFR and MP libraries (see Arithmetic and Arbitrary-Precision Arithmetic with gawk).

As of version 5.2, the arbitrary precision arithmetic features in gawk are “on parole.” The primary maintainer is no longer willing to support this feature, but another member of the development team has stepped up to take it over. As long as this situation remains stable, MPFR will be supported. If it changes, the MPFR support will be removed from gawk.

-n ¶

--non-decimal-data

Enable automatic interpretation of octal and hexadecimal values in input data (see Allowing Nondecimal Input Data).

CAUTION: This option can severely break old programs. Use with care.

-N ¶

--use-lc-numeric

Force the use of the locale’s decimal point character when parsing numeric input data (see Where You Are Makes a Difference).

-o[file] ¶

--pretty-print[=file]

Enable pretty-printing of awk programs. Implies --no-optimize. By default, the output program is created in a file named awkprof.out (see Profiling Your awk Programs). The optional file argument allows you to specify a different file name for the output. No space is allowed between the -o and file, if file is supplied.

NOTE: In the past, this option would also execute your program. This is no longer the case.

-O ¶

--optimize

Enable gawk’s default optimizations on the internal representation of the program. At the moment, this includes just simple constant folding.

Optimization is enabled by default. This option remains primarily for backwards compatibility. However, it may be used to cancel the effect of an earlier -s option (see later in this list).

-p[file] ¶

--profile[=file]

Enable profiling of awk programs (see Profiling Your awk Programs). Implies --no-optimize. By default, profiles are created in a file named awkprof.out. The optional file argument allows you to specify a different file name for the profile file. No space is allowed between the -p and file, if file is supplied.

The profile contains execution counts for each statement in the program in the left margin, and function call counts for each function.

-P ¶

--posix

Operate in strict POSIX mode. This disables all gawk extensions (just like --traditional) and disables all extensions not allowed by POSIX. See Common Extensions Summary for a summary of the extensions in gawk that are disabled by this option. Also, the following additional restrictions apply:

• Newlines are not allowed after ‘?’ or ‘:’ (see Conditional Expressions).
• Specifying ‘-Ft’ on the command line does not set the value of FS to be a single TAB character (see Specifying How Fields Are Separated).
• The locale’s decimal point character is used for parsing input data (see Where You Are Makes a Difference).

If you supply both --traditional and --posix on the command line, --posix takes precedence. gawk issues a warning if both options are supplied.

-r ¶

--re-interval

Allow interval expressions (see Regular Expression Operators) in regexps. This is now gawk’s default behavior. Nevertheless, this option remains for backward compatibility.

-s ¶

--no-optimize

Disable gawk’s default optimizations on the internal representation of the program.

-S ¶

--sandbox

Disable the system() function, input redirections with getline, output redirections with print and printf, and dynamic extensions. Also, disallow adding file names to ARGV that were not there when gawk started running. This is particularly useful when you want to run awk scripts from questionable sources and need to make sure the scripts can’t access your system (other than the specified input data files).

-t ¶

--lint-old

Warn about constructs that are not available in the original version of awk from Version 7 Unix (see Major Changes Between V7 and SVR3.1).

-V ¶

--version

Print version information for this particular copy of gawk. This allows you to determine if your copy of gawk is up to date with respect to whatever the Free Software Foundation is currently distributing. It is also useful for bug reports (see Reporting Problems and Bugs).

--

Mark the end of all options. Any command-line arguments following -- are placed in ARGV, even if they start with a minus sign.

In compatibility mode, as long as program text has been supplied, any other options are flagged as invalid with a warning message but are otherwise ignored.

In compatibility mode, as a special case, if the value of fs supplied to the -F option is ‘t’, then FS is set to the TAB character ("\t"). This is true only for --traditional and not for --posix (see Specifying How Fields Are Separated).

The -f option may be used more than once on the command line. If it is, awk reads its program source from all of the named files, as if they had been concatenated together into one big file. This is useful for creating libraries of awk functions. These functions can be written once and then retrieved from a standard place, instead of having to be included in each individual program. The -i option is similar in this regard. (As mentioned in Function Definition Syntax, function names must be unique.)

With standard awk, library functions can still be used, even if the program is entered at the keyboard, by specifying ‘-f /dev/tty’. After typing your program, type Ctrl-d (the end-of-file character) to terminate it. (You may also use ‘-f -’ to read program source from the standard input, but then you will not be able to also use the standard input as a source of data.)

Because it is clumsy using the standard awk mechanisms to mix source file and command-line awk programs, gawk provides the -e option. This does not require you to preempt the standard input for your source code, and it allows you to easily mix command-line and library source code (see The AWKPATH Environment Variable). As with -f, the -e and -i options may also be used multiple times on the command line.

If no -f option (or -e option for gawk) is specified, then awk uses the first nonoption command-line argument as the text of the program source code. Arguments on the command line that follow the program text are entered into the ARGV array; awk does not continue to parse the command line looking for options.

If the environment variable POSIXLY_CORRECT exists, then gawk behaves in strict POSIX mode, exactly as if you had supplied --posix. Many GNU programs look for this environment variable to suppress extensions that conflict with POSIX, but gawk behaves differently: it suppresses all extensions, even those that do not conflict with POSIX, and behaves in strict POSIX mode. If --lint is supplied on the command line and gawk turns on POSIX mode because of POSIXLY_CORRECT, then it issues a warning message indicating that POSIX mode is in effect. You would typically set this variable in your shell’s startup file. For a Bourne-compatible shell (such as Bash), you would add these lines to the .profile file in your home directory:

POSIXLY_CORRECT=true
export POSIXLY_CORRECT

For a C shell-compatible shell,¹² you would add this line to the .login file in your home directory:

setenv POSIXLY_CORRECT true

Having POSIXLY_CORRECT set is not recommended for daily use, but it is good for testing the portability of your programs to other environments.

Next: Naming Standard Input, Previous: Command-Line Options, Up: Running awk and gawk   [Contents][Index]

2.3 Other Command-Line Arguments ¶

Any additional arguments on the command line are normally treated as input files to be processed in the order specified. However, an argument that has the form var=value, assigns the value value to the variable var—it does not specify a file at all. (See Assigning Variables on the Command Line.) In the following example, ‘count=1’ is a variable assignment, not a file name:

awk -f program.awk file1 count=1 file2

As a side point, should you really need to have awk process a file named count=1 (or any file whose name looks like a variable assignment), precede the file name with ‘./’, like so:

awk -f program.awk file1 ./count=1 file2

All the command-line arguments are made available to your awk program in the ARGV array (see Predefined Variables). Command-line options and the program text (if present) are omitted from ARGV. All other arguments, including variable assignments, are included. As each element of ARGV is processed, gawk sets ARGIND to the index in ARGV of the current element. (gawk makes the full command line, including program text and options, available in PROCINFO["argv"]; see Built-in Variables That Convey Information.)

Changing ARGC and ARGV in your awk program lets you control how awk processes the input files; this is described in more detail in Using ARGC and ARGV.

The distinction between file name arguments and variable-assignment arguments is made when awk is about to open the next input file. At that point in execution, it checks the file name to see whether it is really a variable assignment; if so, awk sets the variable instead of reading a file.

Therefore, the variables actually receive the given values after all previously specified files have been read. In particular, the values of variables assigned in this fashion are not available inside a BEGIN rule (see The BEGIN and END Special Patterns), because such rules are run before awk begins scanning the argument list.

The variable values given on the command line are processed for escape sequences (see Escape Sequences). (d.c.)

In some very early implementations of awk, when a variable assignment occurred before any file names, the assignment would happen before the BEGIN rule was executed. awk’s behavior was thus inconsistent; some command-line assignments were available inside the BEGIN rule, while others were not. Unfortunately, some applications came to depend upon this “feature.” When awk was changed to be more consistent, the -v option was added to accommodate applications that depended upon the old behavior.

The variable assignment feature is most useful for assigning to variables such as RS, OFS, and ORS, which control input and output formats, before scanning the data files. It is also useful for controlling state if multiple passes are needed over a data file. For example:

awk 'pass == 1  { pass 1 stuff }
     pass == 2  { pass 2 stuff }' pass=1 mydata pass=2 mydata

Given the variable assignment feature, the -F option for setting the value of FS is not strictly necessary. It remains for historical compatibility.

f=""
awk '{ print("hi") }' $f

$ f=""
$ awk 'BEGIN { print ARGC }' $f
-| 1
$ awk 'BEGIN { print ARGC }' "$f"
-| 2

Next: The Environment Variables gawk Uses, Previous: Other Command-Line Arguments, Up: Running awk and gawk   [Contents][Index]

2.4 Naming Standard Input ¶

Often, you may wish to read standard input together with other files. For example, you may wish to read one file, read standard input coming from a pipe, and then read another file.

The way to name the standard input, with all versions of awk, is to use a single, standalone minus sign or dash, ‘-’. For example:

some_command | awk -f myprog.awk file1 - file2

Here, awk first reads file1, then it reads the output of some_command, and finally it reads file2.

You may also use "-" to name standard input when reading files with getline (see Using getline from a File). And, you can even use "-" with the -f option to read program source code from standard input (see Command-Line Options).

In addition, gawk allows you to specify the special file name /dev/stdin, both on the command line and with getline. Some other versions of awk also support this, but it is not standard. (Some operating systems provide a /dev/stdin file in the filesystem; however, gawk always processes this file name itself.)

Next: gawk’s Exit Status, Previous: Naming Standard Input, Up: Running awk and gawk   [Contents][Index]

2.5 The Environment Variables gawk Uses ¶

A number of environment variables influence how gawk behaves.

• The AWKPATH Environment Variable
• The AWKLIBPATH Environment Variable
• Other Environment Variables

Next: Including Other Files into Your Program, Previous: The Environment Variables gawk Uses, Up: Running awk and gawk   [Contents][Index]

2.6 gawk’s Exit Status ¶

If the exit statement is used with a value (see The exit Statement), then gawk exits with the numeric value given to it.

Otherwise, if there were no problems during execution, gawk exits with the value of the C constant EXIT_SUCCESS. This is usually zero.

If an error occurs, gawk exits with the value of the C constant EXIT_FAILURE. This is usually one.

If gawk exits because of a fatal error, the exit status is two. On non-POSIX systems, this value may be mapped to EXIT_FAILURE.

Next: Loading Dynamic Extensions into Your Program, Previous: gawk’s Exit Status, Up: Running awk and gawk   [Contents][Index]

2.7 Including Other Files into Your Program ¶

This section describes a feature that is specific to gawk. It discusses how one source file may include another.

The @include directive can be used to read external awk source files. This gives you the ability to split large awk source files into smaller, more manageable pieces, and also lets you reuse common awk code from various awk scripts. In other words, you can group together awk functions used to carry out specific tasks into external files. These files can be used just like function libraries, using the @include directive in conjunction with the AWKPATH environment variable. Note that source files may also be included using the -i option.

Let’s see an example. We’ll start with two (trivial) awk scripts, namely test1 and test2. Here is the test1 script:

BEGIN {
    print "This is script test1."
}

and here is test2:

@include "test1"
BEGIN {
    print "This is script test2."
}

Running gawk with test2 produces the following result:

$ gawk -f test2
-| This is script test1.
-| This is script test2.

gawk runs the test2 script, which includes test1 using the @include directive. So, to include external awk source files, you just use @include followed by the name of the file to be included, enclosed in double quotes.

NOTE: Keep in mind that this is a language construct and the file name cannot be a string variable, but rather just a literal string constant in double quotes.

The files to be included may be nested; e.g., given a third script, namely test3:

@include "test2"
BEGIN {
    print "This is script test3."
}

Running gawk with the test3 script produces the following results:

$ gawk -f test3
-| This is script test1.
-| This is script test2.
-| This is script test3.

The file name can, of course, be a pathname. For example:

@include "../io_funcs"

and:

@include "/usr/awklib/network"

are both valid. The AWKPATH environment variable can be of great value when using @include. The same rules for the use of the AWKPATH variable in command-line file searches (see The AWKPATH Environment Variable) apply to @include also.

This is very helpful in constructing gawk function libraries. If you have a large script with useful, general-purpose awk functions, you can break it down into library files and put those files in a special directory. You can then include those “libraries,” either by using the full pathnames of the files, or by setting the AWKPATH environment variable accordingly and then using @include with just the file part of the full pathname. Of course, you can keep library files in more than one directory; the more complex the working environment is, the more directories you may need to organize the files to be included.

Given the ability to specify multiple -f options, the @include mechanism is not strictly necessary. However, the @include directive can help you in constructing self-contained gawk programs, thus reducing the need for writing complex and tedious command lines. In particular, @include is very useful for writing CGI scripts to be run from web pages.

The @include directive and the -i/--include command line option are completely equivalent. An included program source is not loaded if it has been previously loaded.

The rules for finding a source file described in The AWKPATH Environment Variable also apply to files loaded with @include.

Finally, files included with @include are treated as if they had ‘@namespace "awk"’ at their beginning. See Changing The Namespace, for more information. However, there is a separate @nsinclude command which does not change the namespace; see Including A File Without Changing The Namespace.

Next: Undocumented Options and Features, Previous: Loading Dynamic Extensions into Your Program, Up: Running awk and gawk   [Contents][Index]

2.9 Obsolete Options and/or Features ¶

This section describes features and/or command-line options from previous releases of gawk that either are not available in the current version or are still supported but deprecated (meaning that they will not be in a future release).

As of gawk version 5.2, the arbitrary precision arithmetic feature is “on parole.” This feature is now being supported by a volunteer in the development team and not by the primary maintainer. If this situation changes, then the feature will be removed. For more information see Arbitrary Precision Arithmetic is On Parole!.

Next: Summary, Previous: Obsolete Options and/or Features, Up: Running awk and gawk   [Contents][Index]

2.10 Undocumented Options and Features ¶

Use the Source, Luke!

— Obi-Wan

This section intentionally left blank.

Previous: Undocumented Options and Features, Up: Running awk and gawk   [Contents][Index]

2.11 Summary ¶

• gawk parses arguments on the command line, left to right, to determine if they should be treated as options or as non-option arguments.
• gawk recognizes several options which control its operation, as described in Command-Line Options. All options begin with ‘-’.
• Any argument that is not recognized as an option is treated as a non-option argument, even if it begins with ‘-’.
  • However, when an option itself requires an argument, and the option is separated from that argument on the command line by at least one space, the space is ignored, and the argument is considered to be related to the option. Thus, in the invocation, ‘gawk -F x’, the ‘x’ is treated as belonging to the -F option, not as a separate non-option argument.
• Once gawk finds a non-option argument, it stops looking for options. Therefore, all following arguments are also non-option arguments, even if they resemble recognized options.
• If no -e or -f options are present, gawk expects the program text to be in the first non-option argument.
• All non-option arguments, except program text provided in the first non-option argument, are placed in ARGV as explained in Using ARGC and ARGV, and are processed as described in Other Command-Line Arguments. Adjusting ARGC and ARGV affects how awk processes input.
• The three standard options for all versions of awk are -f, -F, and -v. gawk supplies these and many others, as well as corresponding GNU-style long options.
• Nonoption command-line arguments are usually treated as file names, unless they have the form ‘var=value’, in which case they are taken as variable assignments to be performed at that point in processing the input.
• You can use a single minus sign (‘-’) to refer to standard input on the command line. gawk also lets you use the special file name /dev/stdin.
• gawk pays attention to a number of environment variables. AWKPATH, AWKLIBPATH, and POSIXLY_CORRECT are the most important ones.
• gawk’s exit status conveys information to the program that invoked it. Use the exit statement from within an awk program to set the exit status.
• gawk allows you to include other awk source files into your program using the @include directive and/or the -i and -f command-line options.
• gawk allows you to load additional functions written in C or C++ using the @load directive and/or the -l option. (This advanced feature is described later, in Writing Extensions for gawk.)

Next: Escape Sequences, Up: Regular Expressions   [Contents][Index]

3.1 How to Use Regular Expressions ¶

A regular expression can be used as a pattern by enclosing it in slashes. Then the regular expression is tested against the entire text of each record. (Normally, it only needs to match some part of the text in order to succeed.) For example, the following prints the second field of each record where the string ‘li’ appears anywhere in the record:

$ awk '/li/ { print $2 }' mail-list
-| 555-5553
-| 555-0542
-| 555-6699
-| 555-3430

Regular expressions can also be used in matching expressions. These expressions allow you to specify the string to match against; it need not be the entire current input record. The two operators ‘~’ and ‘!~’ perform regular expression comparisons. Expressions using these operators can be used as patterns, or in if, while, for, and do statements. (See Control Statements in Actions.) For example, the following is true if the expression exp (taken as a string) matches regexp:

exp ~ /regexp/

This example matches, or selects, all input records with the uppercase letter ‘J’ somewhere in the first field:

$ awk '$1 ~ /J/' inventory-shipped
-| Jan  13  25  15 115
-| Jun  31  42  75 492
-| Jul  24  34  67 436
-| Jan  21  36  64 620

So does this:

awk '{ if ($1 ~ /J/) print }' inventory-shipped

This next example is true if the expression exp (taken as a character string) does not match regexp:

exp !~ /regexp/

The following example matches, or selects, all input records whose first field does not contain the uppercase letter ‘J’:

$ awk '$1 !~ /J/' inventory-shipped
-| Feb  15  32  24 226
-| Mar  15  24  34 228
-| Apr  31  52  63 420
-| May  16  34  29 208
...

When a regexp is enclosed in slashes, such as /foo/, we call it a regexp constant, much like 5.27 is a numeric constant and "foo" is a string constant.

Next: Regular Expression Operators, Previous: How to Use Regular Expressions, Up: Regular Expressions   [Contents][Index]

3.2 Escape Sequences ¶

Some characters cannot be included literally in string constants ("foo") or regexp constants (/foo/). Instead, they should be represented with escape sequences, which are character sequences beginning with a backslash (‘\’). One use of an escape sequence is to include a double quote character in a string constant. Because a plain double quote ends the string, you must use ‘\"’ to represent an actual double quote character as a part of the string. For example:

$ awk 'BEGIN { print "He said \"hi!\" to her." }'
-| He said "hi!" to her.

The backslash character itself is another character that cannot be included normally; you must write ‘\\’ to put one backslash in the string or regexp. Thus, the string whose contents are the two characters ‘"’ and ‘\’ must be written "\"\\".

Other escape sequences represent unprintable characters such as TAB or newline. There is nothing to stop you from entering most unprintable characters directly in a string constant or regexp constant, but they may look ugly.

The following list presents all the escape sequences used in awk and what they represent. Unless noted otherwise, all these escape sequences apply to both string constants and regexp constants:

\\

A literal backslash, ‘\’.

\a

The “alert” character, Ctrl-g, ASCII code 7 (BEL). (This often makes some sort of audible noise.)

\b

Backspace, Ctrl-h, ASCII code 8 (BS).

\f

Formfeed, Ctrl-l, ASCII code 12 (FF).

\n

Newline, Ctrl-j, ASCII code 10 (LF).

\r

Carriage return, Ctrl-m, ASCII code 13 (CR).

\t

Horizontal TAB, Ctrl-i, ASCII code 9 (HT).

\v

Vertical TAB, Ctrl-k, ASCII code 11 (VT).

\nnn

The octal value nnn, where nnn stands for 1 to 3 digits between ‘0’ and ‘7’. For example, the code for the ASCII ESC (escape) character is ‘\033’.

\xhh…

The hexadecimal value hh, where hh stands for a sequence of hexadecimal digits (‘0’–‘9’, and either ‘A’–‘F’ or ‘a’–‘f’). A maximum of two digits are allowed after the ‘\x’. Any further hexadecimal digits are treated as simple letters or numbers. (c.e.) (The ‘\x’ escape sequence is not allowed in POSIX awk.)

CAUTION: In ISO C, the escape sequence continues until the first nonhexadecimal digit is seen. For many years, gawk would continue incorporating hexadecimal digits into the value until a non-hexadecimal digit or the end of the string was encountered. However, using more than two hexadecimal digits produced undefined results. As of version 4.2, only two digits are processed.

\uhh…

The hexadecimal value hh, where hh stands for a sequence of one or more hexadecimal digits (‘0’–‘9’, and either ‘A’–‘F’ or ‘a’–‘f’). A maximum of eight digits are allowed after the ‘\u’. Any further hexadecimal digits are treated as simple letters or numbers. (c.e.) (The ‘\u’ escape sequence is not allowed in POSIX awk.)

This escape sequence is intended for designating a character in the current locale’s character set.¹⁶ gawk first converts the given digits into an integer and then translates the given “wide character” value into the current locale’s multibyte encoding. If the wide character value does not represent a valid character, or if the character is valid but cannot be encoded into the current locale’s multibyte encoding, the value becomes "?". gawk issues a warning message when this happens.

\/

A literal slash (should be used for regexp constants only). This sequence is used when you want to write a regexp constant that contains a slash (such as /.*:\/home\/[[:alnum:]]+:.*/; the ‘[[:alnum:]]’ notation is discussed in Using Bracket Expressions). Because the regexp is delimited by slashes, you need to escape any slash that is part of the pattern, in order to tell awk to keep processing the rest of the regexp.

\"

A literal double quote (should be used for string constants only). This sequence is used when you want to write a string constant that contains a double quote (such as "He said \"hi!\" to her."). Because the string is delimited by double quotes, you need to escape any double quote that is part of the string, in order to tell awk to keep processing the rest of the string.

In gawk, a number of additional two-character sequences that begin with a backslash have special meaning in regexps. See gawk-Specific Regexp Operators.

In a regexp, a backslash before any character that is not in the previous list and not listed in gawk-Specific Regexp Operators means that the next character should be taken literally, even if it would normally be a regexp operator. For example, /a\+b/ matches the three characters ‘a+b’.

For complete portability, do not use a backslash before any character not shown in the previous list or that is not a regular expression operator. (The 2024 POSIX standard explicitly lists the operators that can be escaped, leaving it undefined as to what happens for any other escaped character. But the bottom line is as described previously.)

To summarize:

• The escape sequences in the preceding list are always processed first, for both string constants and regexp constants. This happens very early, as soon as awk reads your program.
• gawk processes both regexp constants and dynamic regexps (see Using Dynamic Regexps), for the special operators listed in gawk-Specific Regexp Operators.
• A backslash before any other character means to treat that character literally.

Next: Using Bracket Expressions, Previous: Escape Sequences, Up: Regular Expressions   [Contents][Index]

3.3 Regular Expression Operators ¶

You can combine regular expressions with special characters, called regular expression operators or metacharacters, to increase the power and versatility of regular expressions.

• Regexp Operators in awk
• Some Notes On Interval Expressions
• Shortest Match, or “Non-greedy” Regexp Operators

$ awk '
> BEGIN {
>     x = "ABC_CBA"
>     gsub(/B/, "bb", x)
>     print x
> }'
-| AbbC_CbbA

$ awk '
> BEGIN {
>     x = "ABC"
>     gsub(//, "x", x)
>     print x
> }'
-| xAxBxCx

"aaaxxxzzz"
    shortpat: /x+?/, result: "aaaXxxzzz"
        0: (s: 4, l: 1) -> "x"
    longpat: /x+/, result: "aaaXzzz"
        0: (s: 4, l: 3) -> "xxx"

"aaaxxxyzzz"
    shortpat: /x+?y/, result: "aaaXzzz"
        0: (s: 4, l: 4) -> "xxxy"
    longpat: /x+y/, result: "aaaXzzz"
        0: (s: 4, l: 4) -> "xxxy"

"aaaxxxxxxxxxxxxxxxxzzz"
    shortpat: /(x+?)(x+)(x+?)(x+)/, result: "aaaXzzz"
        0: (s: 4, l: 16) -> "xxxxxxxxxxxxxxxx"
        1: (s: 4, l: 1) -> "x"
        2: (s: 5, l: 13) -> "xxxxxxxxxxxxx"
        3: (s: 18, l: 1) -> "x"
        4: (s: 19, l: 1) -> "x"
    longpat: /(x+)(x+)(x+)(x+)/, result: "aaaXzzz"
        0: (s: 4, l: 16) -> "xxxxxxxxxxxxxxxx"
        1: (s: 4, l: 13) -> "xxxxxxxxxxxxx"
        2: (s: 17, l: 1) -> "x"
        3: (s: 18, l: 1) -> "x"
        4: (s: 19, l: 1) -> "x"

"aaaxyxxyxxyxzzz"
    shortpat: /((x+)(y+?)(x+))+/, result: "aaaXyxxyxzzz"
        0: (s: 4, l: 4) -> "xyxx"
        1: (s: 4, l: 4) -> "xyxx"
        2: (s: 4, l: 1) -> "x"
        3: (s: 5, l: 1) -> "y"
        4: (s: 6, l: 2) -> "xx"
    longpat: /((x+)(y+)(x+))+/, result: "aaaXzzz"
        0: (s: 4, l: 9) -> "xyxxyxxyx"
        1: (s: 10, l: 3) -> "xyx"
        2: (s: 10, l: 1) -> "x"
        3: (s: 11, l: 1) -> "y"
        4: (s: 12, l: 1) -> "x"

"aaaxyxxyxxyxzzz"
    shortpat: /((x+)(y+?)(x+)){2}/, result: "aaaXyxzzz"
        0: (s: 4, l: 7) -> "xyxxyxx"
        1: (s: 7, l: 4) -> "xyxx"
        2: (s: 7, l: 1) -> "x"
        3: (s: 8, l: 1) -> "y"
        4: (s: 9, l: 2) -> "xx"
    longpat: /((x+)(y+)(x+)){2}/, result: "aaaXyxzzz"
        0: (s: 4, l: 7) -> "xyxxyxx"
        1: (s: 7, l: 4) -> "xyxx"
        2: (s: 7, l: 1) -> "x"
        3: (s: 8, l: 1) -> "y"
        4: (s: 9, l: 2) -> "xx"

Next: How Much Text Matches?, Previous: Regular Expression Operators, Up: Regular Expressions   [Contents][Index]

3.4 Using Bracket Expressions ¶

As mentioned earlier, a bracket expression matches any character among those listed between the opening and closing square brackets.

Within a bracket expression, a range expression consists of two characters separated by a hyphen. It matches any single character that sorts between the two characters, based upon the system’s native character set. For example, ‘[0-9]’ is equivalent to ‘[0123456789]’. (See Regexp Ranges and Locales: A Long Sad Story for an explanation of how the POSIX standard and gawk have changed over time. This is mainly of historical interest.)

With the increasing popularity of the Unicode character standard, there is an additional wrinkle to consider. Octal and hexadecimal escape sequences inside bracket expressions are taken to represent only single-byte characters (characters whose values fit within the range 0–255). To match a range of characters where the endpoints of the range are larger than 255, enter the multibyte encodings of the characters directly, or use the \u escape sequence.

To include one of the characters ‘\’, ‘]’, ‘-’, or ‘^’ in a bracket expression, put a ‘\’ in front of it. For example:

[d\]]

matches either ‘d’ or ‘]’. Additionally, if you place ‘]’ right after the opening ‘[’ (‘[]d]’), the closing bracket is treated as one of the characters to be matched. Inside bracket expressions, it’s not necessary to escape the other standard regular expression operators, such as ‘*’ and ‘?’, and for full portability you should not.

NOTE: Note that the additional regular expression operators that begin with a backslash, such as ‘\<’, or ‘\w’, have no meaning when used inside a bracket expression. There, the backslash is taken to mean escape the following character, so ‘[\w]’ is the same as ‘[w]’, and ‘[\<]’ is the same as ‘[<]’.

The treatment of ‘\’ in bracket expressions is compatible with other awk implementations and is also mandated by POSIX.¹⁹ The regular expressions in awk are a superset of the POSIX specification for Extended Regular Expressions (EREs). POSIX EREs are based on the regular expressions accepted by the traditional egrep utility.

Character classes are a feature introduced in the POSIX standard. A character class is a special notation for describing lists of characters that have a specific attribute, but the actual characters can vary from country to country and/or from character set to character set. For example, the notion of what is an alphabetic character differs between the United States and France.

A character class is only valid in a regexp inside the brackets of a bracket expression. Character classes consist of ‘[:’, a keyword denoting the class, and ‘:]’. Table 3.1 lists the character classes defined by the POSIX standard.

For example, before the POSIX standard, you had to write /[A-Za-z0-9]/ to match alphanumeric characters. If your character set had other alphabetic characters in it, this would not match them. With the POSIX character classes, you can write /[[:alnum:]]/ to match the alphabetic and numeric characters in your character set.

Some utilities that match regular expressions provide a nonstandard ‘[:ascii:]’ character class; awk does not. However, you can simulate such a construct using ‘[\x00-\x7F]’. This matches all values numerically between zero and 127, which is the defined range of the ASCII character set. Use a complemented character list (‘[^\x00-\x7F]’) to match any single-byte characters that are not in the ASCII range.

NOTE: Some older versions of Unix awk treat [:blank:] like [:space:], incorrectly matching more characters than they should. Caveat Emptor.

Two additional special sequences can appear in bracket expressions. These apply to non-ASCII character sets, which can have single symbols (called collating elements) that are represented with more than one character. They can also have several characters that are equivalent for collating, or sorting, purposes. (For example, in French, a plain “e” and a grave-accented “è” are equivalent.) These sequences are:

Collating symbols ¶

Multicharacter collating elements enclosed between ‘[.’ and ‘.]’. For example, if ‘ch’ is a collating element, then ‘[[.ch.]]’ is a regexp that matches this collating element, whereas ‘[ch]’ is a regexp that matches either ‘c’ or ‘h’.

Equivalence classes

Locale-specific names for a list of characters that are equal. The name is enclosed between ‘[=’ and ‘=]’. For example, the name ‘e’ might be used to represent all of “e,” “ê,” “è,” and “é.” In this case, ‘[[=e=]]’ is a regexp that matches any of ‘e’, ‘ê’, ‘é’, or ‘è’.

These features are very valuable in non-English-speaking locales.

CAUTION: The library functions that gawk uses for regular expression matching currently recognize only POSIX character classes; they do not recognize collating symbols or equivalence classes.

Inside a bracket expression, an opening bracket (‘[’) that does not start a character class, collating element or equivalence class is taken literally. This is also true of ‘.’ and ‘*’.

Next: Using Dynamic Regexps, Previous: Using Bracket Expressions, Up: Regular Expressions   [Contents][Index]

3.5 How Much Text Matches? ¶

Consider the following:

echo aaaabcd | awk '{ sub(/a+/, "<A>"); print }'

This example uses the sub() function to make a change to the input record. (sub() replaces the first instance of any text matched by the first argument with the string provided as the second argument; see String-Manipulation Functions.) Here, the regexp /a+/ indicates “one or more ‘a’ characters,” and the replacement text is ‘<A>’.

The input contains four ‘a’ characters. awk (and POSIX) regular expressions always match the leftmost, longest sequence of input characters that can match. Thus, all four ‘a’ characters are replaced with ‘<A>’ in this example:

$ echo aaaabcd | awk '{ sub(/a+/, "<A>"); print }'
-| <A>bcd

For simple match/no-match tests, this is not so important. But when doing text matching and substitutions with the match(), sub(), gsub(), and gensub() functions, it is very important. Understanding this principle is also important for regexp-based record and field splitting (see How Input Is Split into Records, and also see Specifying How Fields Are Separated).

Next: gawk-Specific Regexp Operators, Previous: How Much Text Matches?, Up: Regular Expressions   [Contents][Index]

3.6 Using Dynamic Regexps ¶

The righthand side of a ‘~’ or ‘!~’ operator need not be a regexp constant (i.e., a string of characters between slashes). It may be any expression. The expression is evaluated and converted to a string if necessary; the contents of the string are then used as the regexp. A regexp computed in this way is called a dynamic regexp or a computed regexp:

BEGIN { digits_regexp = "[[:digit:]]+" }
$0 ~ digits_regexp    { print }

This sets digits_regexp to a regexp that describes one or more digits, and tests whether the input record matches this regexp.

NOTE: When using the ‘~’ and ‘!~’ operators, be aware that there is a difference between a regexp constant enclosed in slashes and a string constant enclosed in double quotes. If you are going to use a string constant, you have to understand that the string is, in essence, scanned twice: the first time when awk reads your program, and the second time when it goes to match the string on the lefthand side of the operator with the pattern on the right. This is true of any string-valued expression (such as digits_regexp, shown in the previous example), not just string constants.

What difference does it make if the string is scanned twice? The answer has to do with escape sequences, and particularly with backslashes. To get a backslash into a regular expression inside a string, you have to type two backslashes.

For example, /\*/ is a regexp constant for a literal ‘*’. Only one backslash is needed. To do the same thing with a string, you have to type "\\*". The first backslash escapes the second one so that the string actually contains the two characters ‘\’ and ‘*’.

Given that you can use both regexp and string constants to describe regular expressions, which should you use? The answer is “regexp constants,” for several reasons:

• String constants are more complicated to write and more difficult to read. Using regexp constants makes your programs less error-prone. Not understanding the difference between the two kinds of constants is a common source of errors.
• It is more efficient to use regexp constants. awk can note that you have supplied a regexp and store it internally in a form that makes pattern matching more efficient. When using a string constant, awk must first convert the string into this internal form and then perform the pattern matching.
• Using regexp constants is better form; it shows clearly that you intend a regexp match.

$ awk '$0 ~ "[ \t\n]"'
error→ awk: newline in character class [
error→ ]...
error→  source line number 1
error→  context is
error→        $0 ~ "[ >>>  \t\n]" <<<

$ awk '$0 ~ /[ \t\n]/'
here is a sample line
-| here is a sample line
Ctrl-d

Next: Case Sensitivity in Matching, Previous: Using Dynamic Regexps, Up: Regular Expressions   [Contents][Index]

3.7 gawk-Specific Regexp Operators ¶

GNU software that deals with regular expressions provides a number of additional regexp operators. These operators are described in this section and are specific to gawk; they are not available in other awk implementations. Most of the additional operators deal with word matching. For our purposes, a word is a sequence of one or more letters, digits, or underscores (‘_’):

\s

Matches any space character as defined by the current locale. Think of it as shorthand for ‘[[:space:]]’.

\S

Matches any character that is not a space, as defined by the current locale. Think of it as shorthand for ‘[^[:space:]]’.

\w

Matches any word-constituent character—that is, it matches any letter, digit, or underscore. Think of it as shorthand for ‘[[:alnum:]_]’.

\W

Matches any character that is not word-constituent. Think of it as shorthand for ‘[^[:alnum:]_]’.

\<

Matches the empty string at the beginning of a word. For example, /\<away/ matches ‘away’ but not ‘stowaway’.

\>

Matches the empty string at the end of a word. For example, /stow\>/ matches ‘stow’ but not ‘stowaway’.

\y ¶

Matches the empty string at either the beginning or the end of a word (i.e., the word boundary). For example, ‘\yballs?\y’ matches either ‘ball’ or ‘balls’, as a separate word.

\B

Matches the empty string that occurs between two word-constituent characters. For example, /\Brat\B/ matches ‘crate’, but it does not match ‘dirty rat’. ‘\B’ is essentially the opposite of ‘\y’. Another way to think of this is that ‘\B’ matches the empty string provided it’s not at the edge of a word.

There are two other operators that work on buffers. In Emacs, a buffer is, naturally, an Emacs buffer. Other GNU programs, including gawk, consider the entire string to match as the buffer. The operators are:

\`

Matches the empty string at the beginning of a buffer (string)

\'

Matches the empty string at the end of a buffer (string)

Because ‘^’ and ‘$’ always work in terms of the beginning and end of strings, these operators don’t add any new capabilities for awk. They are provided for compatibility with other GNU software.

In other GNU software, the word-boundary operator is ‘\b’. However, that conflicts with the awk language’s definition of ‘\b’ as backspace, so gawk uses a different letter. An alternative method would have been to require two backslashes in the GNU operators, but this was deemed too confusing. The current method of using ‘\y’ for the GNU ‘\b’ appears to be the lesser of two evils.

The various command-line options (see Command-Line Options) control how gawk interprets characters in regexps:

No options

In the default case, gawk provides all the facilities of POSIX regexps and the previously described GNU regexp operators. GNU regexp operators described in Regular Expression Operators.

--posix

Match only POSIX regexps; the GNU operators are not special (e.g., ‘\w’ matches a literal ‘w’). Interval expressions are allowed.

--traditional ¶

Match traditional Unix awk regexps. The GNU operators are not special. Because BWK awk supports them, the POSIX character classes (‘[[:alnum:]]’, etc.) are available. So too, interval expressions are allowed. Characters described by octal and hexadecimal escape sequences are treated literally, even if they represent regexp metacharacters.

--re-interval

This option remains for backwards compatibility but no longer has any real effect.

Next: Selecting the Regexp Matching Engine, Previous: gawk-Specific Regexp Operators, Up: Regular Expressions   [Contents][Index]

3.8 Case Sensitivity in Matching ¶

Case is normally significant in regular expressions, both when matching ordinary characters (i.e., not metacharacters) and inside bracket expressions. Thus, a ‘w’ in a regular expression matches only a lowercase ‘w’ and not an uppercase ‘W’.

The simplest way to do a case-independent match is to use a bracket expression—for example, ‘[Ww]’. However, this can be cumbersome if you need to use it often, and it can make the regular expressions harder to read. There are two alternatives that you might prefer.

One way to perform a case-insensitive match at a particular point in the program is to convert the data to a single case, using the tolower() or toupper() built-in string functions (which we haven’t discussed yet; see String-Manipulation Functions). For example:

tolower($1) ~ /foo/  { ... }

converts the first field to lowercase before matching against it. This works in any POSIX-compliant awk.

Another method, specific to gawk, is to set the variable IGNORECASE to a nonzero value (see Predefined Variables). When IGNORECASE is not zero, all regexp and string operations ignore case.

Changing the value of IGNORECASE dynamically controls the case sensitivity of the program as it runs. Case is significant by default because IGNORECASE (like most variables) is initialized to zero:

x = "aB"
if (x ~ /ab/) ...   # this test will fail
IGNORECASE = 1
if (x ~ /ab/) ...   # now it will succeed

In general, you cannot use IGNORECASE to make certain rules case insensitive and other rules case sensitive, as there is no straightforward way to set IGNORECASE just for the pattern of a particular rule.²⁰ To do this, use either bracket expressions or tolower(). However, one thing you can do with IGNORECASE only is dynamically turn case sensitivity on or off for all the rules at once.

IGNORECASE can be set on the command line or in a BEGIN rule (see Other Command-Line Arguments; also see Startup and Cleanup Actions). Setting IGNORECASE from the command line is a way to make a program case insensitive without having to edit it.

In multibyte locales, the equivalences between upper- and lowercase characters are tested based on the wide-character values of the locale’s character set. Prior to version 5.0, single-byte characters were tested based on the ISO-8859-1 (ISO Latin-1) character set. However, as of version 5.0, single-byte characters are also tested based on the values of the locale’s character set.²¹

The value of IGNORECASE has no effect if gawk is in compatibility mode (see Command-Line Options). Case is always significant in compatibility mode.

Next: Summary, Previous: Case Sensitivity in Matching, Up: Regular Expressions   [Contents][Index]

3.9 Selecting the Regexp Matching Engine ¶

Release 5.4.0 of gawk introduced a new regular expression matching engine, named MinRX.

MinRX is fully compliant with the POSIX standard for Extended Regular Expressions (EREs), including the additional features needed by awk and gawk. It is also a little stricter that the original matchers are in terms of accepting valid regular expression syntax when specifying a regexp. (These restrictions apply to corner cases that should not come up in day-to-day use.)

Previously, gawk used GNU regex and dfa from GNULIB. These matchers are fast and generally robust, albeit not fully POSIX compliant. MinRX replaces both of them.

Because regular expression matching is such a fundamental part of what awk programs do, introducing a new regular expression engine has some risk associated with it. To alleviate the risk, for the term of one major release, gawk continues to provide access to the original regexp matchers should that be needed.

If the environment variable GAWK_GNU_MATCHERS exists, then gawk switches to using GNU regex and dfa, as previously. Otherwise, the MinRX matcher is the default and that is what it uses.

Should you find a need to switch from MinRX to the original matchers, please submit a bug report describing what did not work (see Reporting Problems and Bugs). Doing so is very important, as it will help the maintainers and the MinRX author fix any issues that are found.

After one major release, the old matchers, and the use of the GAWK_GNU_MATCHERS environment variable, will be removed from gawk.

Previous: Selecting the Regexp Matching Engine, Up: Regular Expressions   [Contents][Index]

3.10 Summary ¶

• Regular expressions describe sets of strings to be matched. In awk, regular expression constants are written enclosed between slashes: /…/.
• Regexp constants may be used standalone in patterns and in conditional expressions, or as part of matching expressions using the ‘~’ and ‘!~’ operators.
• Escape sequences let you represent nonprintable characters and also let you represent regexp metacharacters as literal characters to be matched.
• Regexp operators provide grouping, alternation, and repetition.
• Bracket expressions give you a shorthand for specifying sets of characters that can match at a particular point in a regexp. Within bracket expressions, POSIX character classes let you specify certain groups of characters in a locale-independent fashion.
• Regular expressions match the leftmost longest text in the string being matched. This matters for cases where you need to know the extent of the match, such as for text substitution and when the record separator is a regexp.
• Matching expressions may use dynamic regexps (i.e., string values treated as regular expressions).
• gawk’s IGNORECASE variable lets you control the case sensitivity of regexp matching. In other awk versions, use tolower() or toupper().

Next: Examining Fields, Up: Reading Input Files   [Contents][Index]

4.1 How Input Is Split into Records ¶

awk divides the input for your program into records and fields. It keeps track of the number of records that have been read so far from the current input file. This value is stored in a predefined variable called FNR, which is reset to zero every time a new file is started. Another predefined variable, NR, records the total number of input records read so far from all data files. It starts at zero, but is never automatically reset to zero.

Normally, records are separated by newline characters. You can control how records are separated by assigning values to the built-in variable RS. If RS is any single character, that character separates records. Otherwise (in gawk), RS is treated as a regular expression. This mechanism is explained in greater detail shortly.

NOTE: When gawk is invoked with the --csv option, nothing in this section applies. See Working With Comma Separated Value Files, for the details.

• Record Splitting with Standard awk
• Record Splitting with gawk

awk 'BEGIN { RS = "u" }
     { print $0 }' mail-list

$ awk 'BEGIN { RS = "u" }
>      { print $0 }' mail-list

-| Amelia       555-5553     amelia.zodiac
-| sq
-| e@gmail.com    F
-| Anthony      555-3412     anthony.assert
-| ro@hotmail.com   A
-| Becky        555-7685     becky.algebrar
-| m@gmail.com      A
-| Bill         555-1675     bill.drowning@hotmail.com       A
-| Broderick    555-0542     broderick.aliq
-| otiens@yahoo.com R
-| Camilla      555-2912     camilla.inf
-| sar
-| m@skynet.be     R
-| Fabi
-| s       555-1234     fabi
-| s.
-| ndevicesim
-| s@
-| cb.ed
-|     F
-| J
-| lie        555-6699     j
-| lie.perscr
-| tabor@skeeve.com   F
-| Martin       555-6480     martin.codicib
-| s@hotmail.com    A
-| Sam
-| el       555-3430     sam
-| el.lanceolis@sh
-| .ed
-|         A
-| Jean-Pa
-| l    555-2127     jeanpa
-| l.campanor
-| m@ny
-| .ed
-|      R
-|

Bill         555-1675     bill.drowning@hotmail.com       A

awk '{ print $0 }' RS="u" mail-list

$ echo record 1 AAAA record 2 BBBB record 3 |
> gawk 'BEGIN { RS = "\n|( *[[:upper:]]+ *)" }
>             { print "Record =", $0,"and RT = [" RT "]" }'

-| Record = record 1 and RT = [ AAAA ]
-| Record = record 2 and RT = [ BBBB ]
-| Record = record 3 and RT = [
-| ]

BEGIN { RS = "\0" }  # whole file becomes one record?

Next: Nonconstant Field Numbers, Previous: How Input Is Split into Records, Up: Reading Input Files   [Contents][Index]

4.2 Examining Fields ¶

When awk reads an input record, the record is automatically parsed or separated by the awk utility into chunks called fields. By default, fields are separated by whitespace, like words in a line. Whitespace in awk means any string of one or more spaces, TABs, or newlines; other characters that are considered whitespace by other languages (such as formfeed, vertical tab, etc.) are not considered whitespace by awk.

The purpose of fields is to make it more convenient for you to refer to these pieces of the record. You don’t have to use them—you can operate on the whole record if you want—but fields are what make simple awk programs so powerful.

You use a dollar sign (‘$’) to refer to a field in an awk program, followed by the number of the field you want. Thus, $1 refers to the first field, $2 to the second, and so on. (Unlike in the Unix shells, the field numbers are not limited to single digits. $127 is the 127th field in the record.) For example, suppose the following is a line of input:

This seems like a pretty nice example.

Here the first field, or $1, is ‘This’, the second field, or $2, is ‘seems’, and so on. Note that the last field, $7, is ‘example.’. Because there is no space between the ‘e’ and the ‘.’, the period is considered part of the seventh field.

NF is a predefined variable whose value is the number of fields in the current record. awk automatically updates the value of NF each time it reads a record. No matter how many fields there are, the last field in a record can be represented by $NF. So, $NF is the same as $7, which is ‘example.’. If you try to reference a field beyond the last one (such as $8 when the record has only seven fields), you get the empty string. If used in a numeric operation, you get zero.²³

The use of $0, which looks like a reference to the “zeroth” field, is a special case: it represents the whole input record. Use it when you are not interested in specific fields. Here are some more examples:

$ awk '$1 ~ /li/ { print $0 }' mail-list
-| Amelia       555-5553     amelia.zodiacusque@gmail.com    F
-| Julie        555-6699     julie.perscrutabor@skeeve.com   F

This example prints each record in the file mail-list whose first field contains the string ‘li’.

By contrast, the following example looks for ‘li’ in the entire record and prints the first and last fields for each matching input record:

$ awk '/li/ { print $1, $NF }' mail-list
-| Amelia F
-| Broderick R
-| Julie F
-| Samuel A

Next: Changing the Contents of a Field, Previous: Examining Fields, Up: Reading Input Files   [Contents][Index]

4.3 Nonconstant Field Numbers ¶

A field number need not be a constant. Any expression in the awk language can be used after a ‘$’ to refer to a field. The value of the expression specifies the field number. If the value is a string, rather than a number, it is converted to a number. Consider this example:

awk '{ print $NR }'

Recall that NR is the number of records read so far: one in the first record, two in the second, and so on. So this example prints the first field of the first record, the second field of the second record, and so on. For the twentieth record, field number 20 is printed; most likely, the record has fewer than 20 fields, so this prints a blank line. Here is another example of using expressions as field numbers:

awk '{ print $(2*2) }' mail-list

awk evaluates the expression ‘(2*2)’ and uses its value as the number of the field to print. The ‘*’ represents multiplication, so the expression ‘2*2’ evaluates to four. The parentheses are used so that the multiplication is done before the ‘$’ operation; they are necessary whenever there is a binary operator²⁴ in the field-number expression. This example, then, prints the type of relationship (the fourth field) for every line of the file mail-list. (All of the awk operators are listed, in order of decreasing precedence, in Operator Precedence (How Operators Nest).)

If the field number you compute is zero, you get the entire record. Thus, ‘$(2-2)’ has the same value as $0. Similarly, string expressions that evaluate to zero also yield the entire record. For example, ‘$"answer"’, ‘$"0foo"’, or even ‘$("foo" "bar")’.

Negative field numbers are not allowed; trying to reference one usually terminates the program. (The POSIX standard does not define what happens when you reference a negative field number. gawk notices this and terminates your program. Other awk implementations may behave differently.)

As mentioned in Examining Fields, awk stores the current record’s number of fields in the built-in variable NF (also see Predefined Variables). Thus, the expression $NF is not a special feature—it is the direct consequence of evaluating NF and using its value as a field number.

Next: Specifying How Fields Are Separated, Previous: Nonconstant Field Numbers, Up: Reading Input Files   [Contents][Index]

4.4 Changing the Contents of a Field ¶

The contents of a field, as seen by awk, can be changed within an awk program; this changes what awk perceives as the current input record. (The actual input is untouched; awk never modifies the input file.) Consider the following example and its output:

$ awk '{ nboxes = $3 ; $3 = $3 - 10
>        print nboxes, $3 }' inventory-shipped
-| 25 15
-| 32 22
-| 24 14
...

The program first saves the original value of field three in the variable nboxes. The ‘-’ sign represents subtraction, so this program reassigns field three, $3, as the original value of field three minus ten: ‘$3 - 10’. (See Arithmetic Operators.) Then it prints the original and new values for field three. (Someone in the warehouse made a consistent mistake while inventorying the red boxes.)

For this to work, the text in $3 must make sense as a number; the string of characters must be converted to a number for the computer to do arithmetic on it. The number resulting from the subtraction is converted back to a string of characters that then becomes field three. See Conversion of Strings and Numbers.

When the value of a field is changed (as perceived by awk), the text of the input record is recalculated to contain the new field where the old one was. In other words, $0 changes to reflect the altered field. Thus, this program prints a copy of the input file, with 10 subtracted from the second field of each line:

$ awk '{ $2 = $2 - 10; print $0 }' inventory-shipped
-| Jan 3 25 15 115
-| Feb 5 32 24 226
-| Mar 5 24 34 228
...

It is also possible to assign contents to fields that are out of range. For example:

$ awk '{ $6 = ($5 + $4 + $3 + $2)
>        print $6 }' inventory-shipped
-| 168
-| 297
-| 301
...

We’ve just created $6, whose value is the sum of fields $2, $3, $4, and $5. The ‘+’ sign represents addition. For the file inventory-shipped, $6 represents the total number of parcels shipped for a particular month.

Creating a new field changes awk’s internal copy of the current input record, which is the value of $0. Thus, if you do ‘print $0’ after adding a field, the record printed includes the new field, with the appropriate number of field separators between it and the previously existing fields.

This recomputation affects and is affected by NF (the number of fields; see Examining Fields). For example, the value of NF is set to the number of the highest field you create. The exact format of $0 is also affected by a feature that has not been discussed yet: the output field separator, OFS, used to separate the fields (see Output Separators).

Note, however, that merely referencing an out-of-range field does not change the value of either $0 or NF. Referencing an out-of-range field only produces an empty string. For example:

if ($(NF+1) != "")
    print "can't happen"
else
    print "everything is normal"

should print ‘everything is normal’, because NF+1 is certain to be out of range. (See The if-else Statement for more information about awk’s if-else statements. See Variable Typing and Comparison Expressions for more information about the ‘!=’ operator.)

It is important to note that making an assignment to an existing field changes the value of $0 but does not change the value of NF, even when you assign the empty string to a field. For example:

$ echo a b c d | awk '{ OFS = ":"; $2 = ""
>                       print $0; print NF }'
-| a::c:d
-| 4

The field is still there; it just has an empty value, delimited by the two colons between ‘a’ and ‘c’. This example shows what happens if you create a new field:

$ echo a b c d | awk '{ OFS = ":"; $2 = ""; $6 = "new"
>                       print $0; print NF }'
-| a::c:d::new
-| 6

The intervening field, $5, is created with an empty value (indicated by the second pair of adjacent colons), and NF is updated with the value six.

Decrementing NF throws away the values of the fields after the new value of NF and recomputes $0. (d.c.) Here is an example:

$ echo a b c d e f | awk '{ print "NF =", NF;
>                           NF = 3; print $0 }'
-| NF = 6
-| a b c

CAUTION: Some versions of awk don’t rebuild $0 when NF is decremented. Until August, 2018, this included BWK awk; fortunately his version now handles this correctly.

Finally, there are times when it is convenient to force awk to rebuild the entire record, using the current values of the fields and OFS. To do this, use the seemingly innocuous assignment:

$1 = $1   # force record to be reconstituted
print $0  # or whatever else with $0

This forces awk to rebuild the record. It does help to add a comment, as we’ve shown here.

There is a flip side to the relationship between $0 and the fields. Any assignment to $0 causes the record to be reparsed into fields using the current value of FS. This also applies to any built-in function that updates $0, such as sub() and gsub() (see String-Manipulation Functions).

Next: Reading Fixed-Width Data, Previous: Changing the Contents of a Field, Up: Reading Input Files   [Contents][Index]

4.5 Specifying How Fields Are Separated ¶

The field separator, which is either a single character or a regular expression, controls the way awk splits an input record into fields. awk scans the input record for character sequences that match the separator; the fields themselves are the text between the matches.

In the examples that follow, we use the bullet symbol (•) to represent spaces in the output. If the field separator is ‘oo’, then the following line:

moo goo gai pan

is split into three fields: ‘m’, ‘•g’, and ‘•gai•pan’. Note the leading spaces in the values of the second and third fields.

The field separator is represented by the predefined variable FS. Shell programmers take note: awk does not use the name IFS that is used by the POSIX-compliant shells (such as the Unix Bourne shell, sh, or Bash).

The value of FS can be changed in the awk program with the assignment operator, ‘=’ (see Assignment Expressions). Often, the right time to do this is at the beginning of execution before any input has been processed, so that the very first record is read with the proper separator. To do this, use the special BEGIN pattern (see The BEGIN and END Special Patterns). For example, here we set the value of FS to the string ":":

awk 'BEGIN { FS = ":" } ; { print $2 }'

Given the input line:

John Q. Smith: 29 Oak St.: Walamazoo: MI 42139

this awk program extracts and prints the string ‘•29•Oak•St.’.

Sometimes the input data contains separator characters that don’t separate fields the way you thought they would. For instance, the person’s name in the example we just used might have a title or suffix attached, such as:

John Q. Smith: LXIX: 29 Oak St.: Walamazoo: MI 42139

The same program would extract ‘•LXIX’ instead of ‘•29•Oak•St.’. If you were expecting the program to print the address, you would be surprised. The moral is to choose your data layout and separator characters carefully to prevent such problems. (If the data is not in a form that is easy to process, perhaps you can massage it first with a separate awk program.)

• Whitespace Normally Separates Fields
• Using Regular Expressions to Separate Fields
• Making Each Character a Separate Field
• Working With Comma Separated Value Files
• Setting FS from the Command Line
• Making the Full Line Be a Single Field
• Field-Splitting Summary