by Tim Parker

IN THIS CHAPTER

• What Is the awk Language?

• Files, Records, and Fields

• Pattern-Action Pairs

• Calling gawk Programs

• Control Structures

The awk programming language was created by the three people who gave their last-name initials to the language: Alfred Aho, Peter Weinberger, and Brian Kernighan. The gawk program included with Linux is the GNU implementation of that programming language.

The awk language is more than just a programming language; it is an almost indispensable tool for many system administrators and UNIX programmers. The language itself is easy to learn, easy to master, and amazingly flexible. Once you get the hang of using awk, you'll be surprised how often you can use it for routine tasks on your system.

To help you understand gawk, I will follow a simple order of introducing the elements of the programming language, as well as showing good examples. You are encouraged, or course, to experiment as the chapter progresses.

I can't cover all the different aspects and features of gawk in this chapter, but we will look at the basics of the language and show you enough, hopefully, to get your curiosity working.

awk is designed to be an easy-to-use programming language that lets you work with information either stored in files or piped to it. The main strengths of awk are its capabilities to do the following:

• Display some or all the contents of a file, selecting rows, columns, or fields as necessary.

• Analyze text for frequency of words, occurrences, and so on.

• Prepare formatted output reports based on information in a file.

• Filter text in a very powerful manner.

• Perform calculations with numeric information from a file.

awk isn't difficult to learn. In many ways, awk is the ideal first programming language because of its simple rules, basic formatting, and standard usage. Experienced programmers will find awk refreshingly easy to use.

Usually, gawk works with data stored in files. Often this is numeric data, but gawk can work with character information, too. If data is not stored in a file, it is supplied to gawk through a pipe or other form of redirection. Only ASCII files (text files) can be properly handled with gawk. Although it does have the ability to work with binary files, the results are often unpredictable. Since most information on a Linux system is stored in ASCII, this isn't a problem.

As a simple example of a file that gawk works with, consider a telephone directory. It is composed of many entries, all with the same format: last name, first name, address, telephone number. The entire telephone directory is a database of sorts, although without a sophisticated search routine. Indeed, the telephone directory relies on a pure alphabetical order to enable users to search for the data they need.

Each line in the telephone directory is a complete set of data on its own and is called a record. For example, the entry in the telephone directory for "Smith, John," which includes his address and telephone number, is a record.

Each piece of information in the record--the last name, the first name, the address, and the telephone number--is called a field. For the gawk language, the field is a single piece of information. A record, then, is a number of fields that pertain to a single item. A set of records makes up a file.

In most cases, fields are separated by a character that is used only to separate fields, such as a space, a tab, a colon, or some other special symbol. This character is called a field separator. A good example is the file /etc/passwd, which looks like this:

tparker:t36s62hsh:501:101:Tim Parker:/home/tparker:/bin/bash

etreijs:2ys639dj3h:502:101:Ed Treijs:/home/etreijs:/bin/tcsh

ychow:1h27sj:503:101:Yvonne Chow:/home/ychow:/bin/bash

If you look carefully at the file, you will see that it uses a colon as the field separator. Each line in the /etc/passwd file has seven fields: the user name, the password, the user ID, the group ID, a comment field, the home directory, and the startup shell. Each field is separated by a colon. Colons exist only to separate fields. A program looking for the sixth field in any line needs only count five colons across (because the first field doesn't have a colon before it).

That's where we find a problem with the gawk definition of fields as they pertain to the telephone directory example. Consider the following lines from a telephone directory:

Smith, John    13 Wilson St.              555-1283

Smith, John    2736 Artside Dr, Apt 123   555-2736

Smith, John    125 Westmount Cr           555-1726

We "know" there are four fields here: the last name, the first name, the address, and the telephone number. But gawk doesn't see it that way. The telephone book uses the space character as a field separator, so on the first line it sees "Smith" as the first field, "John" as the second, "13" as the third, "Wilson" as the fourth, and so on. As far as gawk is concerned, the first line when using a space character as a field separator has six fields. The second line has eight fields.

To make sense of the telephone directory the way we want to handle it, we have to find another way of structuring the data so that there is a field separator between the sections. For example, the following uses the slash character as the field separator:

Smith/John/13 Wilson St./555-1283

Smith/John/2736 Artside Dr, Apt 123/555-2736

Smith/John/125 Westmount Cr/555-1726

By default, gawk uses blank characters (spaces or tabs) as field separators unless instructed to use another character. If gawk is using spaces, it doesn't matter how many are in a row; they are treated as a single block for purposes of finding fields. Naturally, there is a way to override this behavior, too.

The gawk language has a particular format for almost all instructions. Each command is composed of two parts: a pattern and a corresponding action. Whenever the pattern is matched, gawk executes the action that matches that pattern.

Pattern-action pairs can be thought of in more common terms to show how they work. Consider instructing someone how to get to the post office. You might say, "Go to the end of the street and turn right. At the stop sign, turn left. At the end of the street, go right." You have created three pattern-action pairs with these instructions:

end of street: turn right

stop sign: turn left

end of street: turn right

When these patterns are met, the corresponding action is taken. You wouldn't turn right before you reached the end of the street, and you don't turn right until you get to the end of the street, so the pattern must be matched precisely for the action to be performed. This is a bit simplistic, but it gives you the basic idea.

With gawk, the patterns to be matched are enclosed in a pair of slashes, and the actions are in a pair of curly braces:

/pattern1/{action1}

/pattern2/{action2}

/pattern3/{action3}

This format makes it quite easy to tell where the pattern starts and ends, and when the action starts and ends. All gawk programs are sets of these pattern-action pairs, one after the other. Remember these pattern-action pairs are working on text files, so a typical set of patterns might be matching a set of strings, and the actions might be to print out parts of the line that matched.

Suppose there isn't a pattern? In that case, the pattern matches every time and the action is executed every time. If there is no action, gawk copies the entire line that matched without change.

Here are some simple examples. The gawk command

gawk `/tparker/' /etc/passwd

will look for each line in the /etc/passwd file that contains the pattern tparker and display it (there is no action, only a pattern). The output from the command will be the one line in the /etc/passwd file that contains the string tparker. If there is more than one line in the file with that pattern, they all will be displayed. In this case, gawk is acting exactly like the grep utility!

This example shows you two important things about gawk: It can be invoked from the command line by giving it the pattern-action pair to work with and a filename, and it likes to have single quotes around the pattern-action pair in order to differentiate them from the filename.

The gawk language is literal in its matching. The string cat will match any lines with cat in them, whether the word "cat" by itself or part of another word such as "concatenate." To be exact, put spaces on either side of the word. Also, case is important. We'll see how to expand the matching in the section "Metacharacters" a little later in the chapter.

Jumping ahead slightly, we can introduce a gawk command. The command

gawk `{print $3}' file2.data

has only one action, so it performs that action on every line in the file file2.data. The action is print $3, which tells gawk to print the third field of every line. The default field separator, a space, is used to tell where fields begin and end. If we had tried the same command on the /etc/passwd file, nothing would have been displayed because the field separator used in that file is the colon.

We can combine the two commands to show a complete pattern-action pair:

gawk `/UNIX/{print $2}' file2.data

This command will search file2.data line by line, looking for the string UNIX. If it finds UNIX, it prints the second field of that line (record).

You can combine more than one pattern-action pair in a command. For example,

gawk `/scandal/{print $1} /rumor/{print $2}' gossip_file

scans each line of gossip_file for the patterns "scandal" and "rumor." When a match is found, gawk prints the first or second field, respectively.

As you might have figured out, gawk numbers all of the fields in a record. The first field is $1, the second is $2, and so on. The entire record is called $0. As a short form, gawk allows you to ignore the $0 in simple commands, so the instructions

gawk `/tparker/{print $0}' /etc/passwd

gawk `/tparker/{print}' /etc/passwd

gawk `/tparker/' /etc/passwd

result in the same output (the latter one because no action causes the entire line to be printed).

Sometimes you want to do more than match a simple character string. The gawk language has many powerful features, but I'll just introduce a few at the moment. We can, for example, make a comparison of a field with a value. The command

gawk `$2 == "foo" {print $3}' testfile

instructs gawk to compare the second field ($2) of each record in testfile and check to see whether it is equal to the string foo. If it is, gawk prints the third field ($3).

This command demonstrates a few important points. First, there are no slashes around the pattern because we are not matching a pattern but are evaluating something. Slashes are used only for character matches. Second, the == sign means "is equal to." We must use two equal signs, because the single equal sign is used for assignment of values, as you will see shortly. Finally, we put double quotations around foo because we want gawk to interpret it literally. Only strings of characters that are to be literally interpreted must be quoted in this manner.

An essential component of any programming language is the ability to compare two strings or numbers and evaluate whether they are equal or different. The gawk program has several comparisons, including ==, which you just saw in an example. Table 26.1 shows the important comparisons.

These are probably familiar to you from arithmetic and other programming languages you may have seen. From this, you can surmise that the command

gawk `$4 > 100' testfile

will display every line in testfile in which the value in the fourth field is greater than 100.

All of the normal arithmetic commands are available, including add, subtract, multiply, and divide. There are also more advanced functions such as exponentials and remainders (also called modulus). Table 26.2 shows the basic arithmetic operations that gawk supports.

You can combine fields and math, too. For example, the action

{print $3/2}

divides the number in the third field by 2.

There is also a set of arithmetic functions for trigonometry and generating random numbers. See Table 26.3.

The order of operations is important to gawk, as it is to regular arithmetic. The rules gawk follows are the same as with arithmetic: all multiplications, divisions, and remainders are performed before additions and subtractions. For example, the command

{print $1+$2*$3}

multiplies field two by field three and then adds the result to field one. If you wanted to force the addition first, you would have to use parentheses:

{print ($1+$2)*$3}

Because these are the same rules you used in algebra, they shouldn't cause you any confusion. Remember, if in doubt, put parentheses in the proper places to force the operations.

If you've used any other programming language, these concepts will be familiar to you. If you are new to programming, you will probably find them obvious, but you'd be surprised how many people get things hopelessly muddled by using strings when they should have used numbers.

A string is a set of characters to be interpreted literally by gawk. Strings are surrounded by quotation marks. Numbers are not surrounded by quotation marks and are treated as real values.

For example, the command

gawk `$1 != "Tim" {print}' testfile

will print any line in testfile that doesn't have the word Tim in the first field. If we had left out the quotation marks around Tim, gawk wouldn't have processed the command properly. The command

gawk `$1 == "50" {print}' testfile

will display any line that has the string 50 in it. It does not attempt to see if the value stored in the first field is different than 50; it just does a character check. The string 50 is not equal to the number 50 as far as gawk is concerned.

We've seen how to do simple actions in the commands we've already discussed, but you can do several things in an action. For example, the command

gawk `$1 != "Tim" {print $1, $5, $6, $2}' testfile

will print the first, fifth, sixth, and second field of testfile for every line that doesn't have the first field equal to "Tim". You can place as many of these fields as you want in a print command.

Indeed, you can place strings in a print command, too, such as in the command

gawk `$1 != "Tim" {print "The entry for ", $1, "is not Tim. ", $2}' testfile

which will print the strings and the fields as shown. Each section of the print command is separated by a comma. There are also spaces at the end of the strings to ensure there is a space between the string and the value of the field that is printed.

You can use additional formatting instructions to make gawk format the output properly. These instructions are borrowed from the C language, and they use the command printf (print formatted) instead of print.

The printf command uses a placeholder scheme, but the gawk language knows how to format the entry because of the placeholder and looks later in the command line to find out what to put there. An example will help make this obvious:

{printf "%5s likes this language\n", $2}

The %5s part of the line instructs gawk how to format the string, in this case using five string characters. The value to place in this position is given at the end of the line as the second column. The \n at the end of the quoted section is a newline character. If the second field of a four-line file held names, printf would format the output like this:

  Tim likes this language

Geoff likes this language

 Mike likes this language

  Joe likes this language

You will notice that the %5s format means to right-justify the column entry. This prevents awkward spacing.

The gawk language supports several format placeholders. They are shown in Table 26.4.

Whenever you use one of the format characters, you can place a number before the character to show how many digits or characters are to be used. Therefore, the format 6d would have six digits of an integer. Many formats can be on a line, but each must have a value at the end of the line, as in this example:

{printf "%5s works for %5s and earns %2d an hour", $1, $2, $3}

Here, the first string is the first field, the second string is the second field, and the third set of digits is from the third field in a file. The output would be something like this:

Joe works for Mike and earns 12 an hour

A few little tricks are useful. As you saw in an earlier example, strings are right-justified, so the command

{printf "%5s likes this language\n", $2}

results in the output

  Tim likes this language

Geoff likes this language

 Mike likes this language

  Joe likes this language

To left-justify the names, place a minus sign in the format statement:

{printf "%-5s likes this language\n", $2}

This will result in the output

Tim   likes this language

Geoff likes this language

Mike  likes this language

Joe   likes this language

Notice that the name is justified on the left instead of on the right.

When dealing with numbers, you can specify the precision to be used, so that the command

{printf "%5s earns $%.2f an hour", $3, $6}

will use the third field and put five characters from it in the first placeholder, and then take the value in the sixth field and place it in the second placeholder with two digits after the decimal point. The output of the command would be like this:

Joe earns $12.17 an hour

The dollar sign was inside the quotation marks in the printf command, and was not generated by the system. It has no special meaning inside the quotation marks. If you want to limit the number of digits to the right of the period, you can do that too. The command

{printf "%5s earns $%6.2f an hour", $3, $6}

will put six digits before the period and two after.

Finally, we can impose some formatting on the output lines themselves. In an earlier example, you saw the use of \n to add a newline character. These are called escape codes, because the backslash is interpreted by gawk to mean something different than a backslash. Table 26.5 shows the important escape codes that gawk supports.

You can, for example, escape a quotation mark by using the sequence \", which will place a quotation mark in the string without interpreting it to mean something special. For example:

{printf "I said \"Hello\" and he said "\Hello\"."

Awkward-looking, perhaps, but necessary to avoid problems. You'll see lots more escape characters used in examples later in this chapter. To use a literal backslash, use \\ in your program.

As I mentioned earlier, the default field separator is always a whitespace character (spaces or tabs). This is not often convenient, as we found with the /etc/passwd file. You can change the field separator on the gawk command line by using the -F option followed by the separator you want to use:

gawk -F":" `/tparker/{print}' /etc/passwd

This command changes the field separator to a colon and searches the /etc/passwd file for the lines containing the string tparker. The new field separator is put in quotation marks to avoid any confusion. Also, the -F option (it must be a capital F) is before the first quote character enclosing the pattern-action pair. If it came after, it wouldn't be applied.

Earlier I mentioned that gawk is particular about its pattern-matching habits. The string cat will match anything with the three letters on the line. Sometimes you want to be more exact in the matching. If you only want to match the word "cat" but not "concatenate," you should put spaces on either side of the pattern:

/ cat / {print}

What about matching different cases? That's where the or instruction, represented by a vertical bar, comes in. For example,

/ cat | CAT / {print}

will match "cat" or "CAT" on a line. However, what about "Cat?" That's where we also need to specify options within a pattern. With gawk, we use square brackets for this. To match any combination of "cat" in upper- or lowercase, we must write the pattern like this:

/ [Cc][Aa][Tt] / {print}

This can get pretty awkward, but it's seldom necessary. To match just "Cat" and "cat," for example, we would use the pattern

/ [Cc]at / {print}

A useful matching operator is the tilde (~). This is used when you want to look for a match in a particular field in a record. For example, the pattern

$5 ~ /tparker/

will match any records where the fifth field is tparker. It is similar to the == operator. The matching operator can be negated, so

$5 !~ /tparker/

will find any record where the fifth field is not equal to tparker.

A few characters (called metacharacters) have special meaning to gawk. Many of these metacharacters will be familiar to shell users, because they are carried over from UNIX shells. The metacharacters shown in Table 26.6 can be used in gawk patterns.

Some of these metacharacters are used frequently. You will see some examples later in this chapter.

Running pattern-action pairs one or two at a time from the command line would be pretty difficult (and time consuming), so gawk allows you to store pattern-action pairs in a file. A gawk program (called a script) is a set of pattern-action pairs stored in an ASCII file. For example, this could be the contents of a valid gawk script:

/tparker/{print $6}

$2 != "foo" {print}

The first line would look for tparker and print the sixth field, and the second line would look for second fields that don't match the string "foo", then display the entire line. When you are writing a script, you don't need to worry about the quotation marks around the pattern-action pairs as you did on the command line, because the new command to execute this script makes it obvious where the pattern-action pairs start and end. After you have saved all of the pattern-action pairs in a program, they are called by gawk with the -f option on the command line:

gawk -f script filename

This command causes gawk to read all of the pattern-action pairs from the file script and process them against the file called filename. This is how most gawk programs are written. Don't confuse the -f and -F options!

If you want to specify a different field separator on the command line (they can be specified in the script, but use a special format you'll see later), the -F option must follow the -f option:

gawk -f script -F":" filename

If you want to process more than one file using the script, just append the names of the files:

gawk -f script filename1 filename2 filename3 ...

By default, all output from the gawk command is displayed on the screen. You could redirect it to a file with the usual UNIX redirection commands:

gawk -f script filename > save_file

There is another way of specifying the output file from within the script, but we'll come back to that in a moment.

Two special patterns supported by gawk are useful when writing scripts. The BEGIN pattern is used to indicate any actions that should take place before gawk starts processing a file. This is usually used to initialize values, set parameters such as field separators, and so on. The END pattern is used to execute any instructions after the file has been completely processed. Typically, this can be for summaries or completion notices.

Any instructions following the BEGIN and END patterns are enclosed in curly braces to identify which instructions are part of both patterns. Both BEGIN and END must appear in capitals. Here's a simple example of a gawk script that uses BEGIN and END, albeit only for sending a message to the terminal:

BEGIN { print "Starting to process the file" }

$1 == "UNIX" {print}

$2 > 10 {printf "This line has a value of %d", $2}

END { print "Finished processing the file.  Bye!"}

In this script, a message is initially printed, and each line that has the word UNIX in the first field is echoed to the screen. Next, any line with the second field greater than 10 is found, and the message is generated with its current value. Finally, the END pattern prints a message that the program is finished.

If you have used any programming language before, you know that a variable is a storage location for a value. Each variable has a name and an associated value, which may change.

With gawk, you assign a variable a value by using =, the assignment operator:

var1 = 10

This assigns the value 10 (numeric, not string) to the variable var1. With gawk, you don't have to declare variable types before you use them as you must with most other languages. This makes it easy to work with variables in gawk.

The gawk language lets you use variables within actions, so the pattern-action pair

$1 == "Plastic" { count = count + 1 }

checks to see if the first field is equal to the string "Plastic", and if it is, increments the value of count by one. Somewhere above this line we should set a preliminary value for the variable count (usually in the BEGIN section), or we will be adding one to an unknown value.

Here's a more complete example:

BEGIN { count = 0 }

$5 == "UNIX" { count = count + 1 }

END { printf "%d occurrences of UNIX were found", count }

In the BEGIN section, the variable count is set to zero. Then, the gawk pattern-action pair is processed, with every occurrence of "UNIX" adding one to the value of count. After the entire file has been processed, the END statement displays the total number.

Variables can be used in combination with fields and values, so all of the following statements are legal:

count = count + $6

count = $5 - 8

count = $5 + var1

Variables can also be part of a pattern. The following are all valid as pattern-action pairs:

$2 > max_value {print "Max value exceeded by ", $2 - max_value}

$4 - var1 < min_value {print "Illegal value of ", $4}

Two special operators are used with variables to increment and decrement by one, because these are common operations. Both of these special operators are borrowed from the C language:

count++	Increments count by one
count--	Decrements count by one

The gawk language has a few built-in variables that are used to represent things such as the total number of records processed. These are useful when you want to get totals. Table 26.7 shows the important built-in variables.

The NR and FNR values are the same if you are processing only one file, but if you are doing more than one file, NR is a running total of all files, while FNR is the total for the current file only.

The FS variable is useful, because it controls the input file's field separator. To use the colon for the /etc/passwd file, for example, you would use the command

FS=":"

in the script, usually as part of the BEGIN pattern.

You can use these built-in variables as you would any other. For example, the command

NF <= 5 {print "Not enough fields in the record"}

gives you a way to check the number of fields in the file you are processing and generate an error message if the values are incorrect.

Enough of the details have been covered to allow us to start doing some real gawk programming. Although we have not covered all of gawk's pattern and action considerations, we have seen all the important material. Now we can look at writing control structures.

If you have any programming experience at all, or have tried some shell script writing, many of these control structures will appear familiar. Follow the examples and try a few test programs of your own.

Incidentally, gawk enables you to place comments anywhere in your scripts, as long as the comment starts with a # sign. You should use comments to indicate what is going on in your scripts if it is not immediately obvious.