awk

USAGE

Unix: awk ‘/pattern/ {print “$1”}’ # standard Unix shells
DOS/Win: awk ‘/pattern/ {print “$1”}’ # okay for DJGPP compiled
awk “/pattern/ {print \”$1\”}” # required for Mingw32
Most of my experience comes from version of GNU awk (gawk) compiled for Win32. Note in particular that DJGPP compilations permit the awk script to follow Unix quoting syntax ‘/like/ {“this”}’. However, the user must know that single quotes under DOS/Windows do not protect the redirection arrows (<, >) nor do they protect pipes (|). Both are special symbols for the DOS/CMD command shell and their special meaning is ignored only if they are placed within “double quotes.” Likewise, DOS/Win users must remember that the percent sign (%) is used to mark DOS/Win environment variables, so it must be doubled (%%) to yield a single percent sign visible to awk.

If I am sure that a script will NOT need to be quoted in Unix, DOS, or CMD, then I normally omit the quote marks. If an example is peculiar to GNU awk, the command ‘gawk’ will be used. Please notify me if you find errors or new commands to add to this list (total length under 65 characters). I usually try to put the shortest script first.

File Spacing

Double space a file

awk ‘1;{print “”}’
awk ‘BEGIN{ORS=”\n\n”};1’
Double space a file which already has blank lines in it. Output file should contain no more than one blank line between lines of text. NOTE: On Unix systems, DOS lines which have only CRLF (\r\n) are often treated as non-blank, and thus ‘NF’ alone will return TRUE.

awk ‘NF{print $0 “\n”}’
Triple space a file

awk ‘1;{print “\n”}’
Numbering and Calculations

Precede each line by its line number FOR THAT FILE (left alignment). Using a tab (\t) instead of space will preserve margins.

awk ‘{print FNR “\t” $0}’ files*
Precede each line by its line number FOR ALL FILES TOGETHER, with tab.

awk ‘{print NR “\t” $0}’ files*
Number each line of a file (number on left, right-aligned) Double the percent signs if typing from the DOS command prompt.

awk ‘{printf(“%5d : %s\n”, NR,$0)}’
Number each line of file, but only print numbers if line is not blank Remember caveats about Unix treatment of \r (mentioned above)

awk ‘NF{$0=++a ” :” $0};{print}’
awk ‘{print (NF? ++a ” :” :””) $0}’
Count lines (emulates “wc -l”)

awk ‘END{print NR}’
Print the sums of the fields of every line

awk ‘{s=0; for (i=1; i<=NF; i++) s=s+$i; print s}' Add all fields in all lines and print the sum awk '{for (i=1; i<=NF; i++) s=s+$i}; END{print s}' Print every line after replacing each field with its absolute value awk '{for (i=1; i<=NF; i++) if ($i < 0) $i = -$i; print }' awk '{for (i=1; i<=NF; i++) $i = ($i < 0) ? -$i : $i; print }' Print the total number of fields ("words") in all lines awk '{ total = total + NF }; END {print total}' file Print the total number of lines that contain "Beth" awk '/Beth/{n++}; END {print n+0}' file Print the largest first field and the line that contains it Intended for finding the longest string in field #1 awk '$1 > max {max=$1; maxline=$0}; END{ print max, maxline}’
Print the number of fields in each line, followed by the line

awk ‘{ print NF “:” $0 } ‘
Print the last field of each line

awk ‘{ print $NF }’
Print the last field of the last line

awk ‘{ field = $NF }; END{ print field }’
Print every line with more than 4 fields

awk ‘NF > 4’
Print every line where the value of the last field is > 4

awk ‘$NF > 4’
Text Conversion and Substitution

IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format

awk ‘{sub(/\r$/,””);print}’ # assumes EACH line ends with Ctrl-M
IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format

awk ‘{sub(/$/,”\r”);print}
IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format

awk 1
IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format Cannot be done with DOS versions of awk, other than gawk:

gawk -v BINMODE=”w” ‘1’ infile >outfile
Use “tr” instead.

tr -d \r outfile # GNU tr version 1.22 or higher
Delete leading whitespace (spaces, tabs) from front of each line aligns all text flush left

awk ‘{sub(/^[ \t]+/, “”); print}’
Delete trailing whitespace (spaces, tabs) from end of each line

awk ‘{sub(/[ \t]+$/, “”);print}’
Delete BOTH leading and trailing whitespace from each line

awk ‘{gsub(/^[ \t]+|[ \t]+$/,””);print}’
awk ‘{$1=$1;print}’ # also removes extra space between fields
Insert 5 blank spaces at beginning of each line (make page offset)

awk ‘{sub(/^/, ” “);print}’
Align all text flush right on a 79-column width

awk ‘{printf “%79s\n”, $0}’ file*
Center all text on a 79-character width

awk ‘{l=length();s=int((79-l)/2); printf “%”(s+l)”s\n”,$0}’ file*
Substitute (find and replace) “foo” with “bar” on each line

awk ‘{sub(/foo/,”bar”);print}’ # replaces only 1st instance
gawk ‘{$0=gensub(/foo/,”bar”,4);print}’ # replaces only 4th instance
awk ‘{gsub(/foo/,”bar”);print}’ # replaces ALL instances in a line
Substitute “foo” with “bar” ONLY for lines which contain “baz”

awk ‘/baz/{gsub(/foo/, “bar”)};{print}’
Substitute “foo” with “bar” EXCEPT for lines which contain “baz”

awk ‘!/baz/{gsub(/foo/, “bar”)};{print}’
Change “scarlet” or “ruby” or “puce” to “red”

awk ‘{gsub(/scarlet|ruby|puce/, “red”); print}’
Reverse order of lines (emulates “tac”)

awk ‘{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j–] }’ file*
If a line ends with a backslash, append the next line to it (fails if there are multiple lines ending with backslash…)

awk ‘/\\$/ {sub(/\\$/,””); getline t; print $0 t; next}; 1’ file*
Print and sort the login names of all users

awk -F “:” ‘{ print $1 | “sort” }’ /etc/passwd
Print the first 2 fields, in opposite order, of every line

awk ‘{print $2, $1}’ file
Switch the first 2 fields of every line

awk ‘{temp = $1; $1 = $2; $2 = temp}’ file
Print every line, deleting the second field of that line

awk ‘{ $2 = “”; print }’
Print in reverse order the fields of every line

awk ‘{for (i=NF; i>0; i–) printf(“%s “,i);printf (“\n”)}’ file
Remove duplicate, consecutive lines (emulates “uniq”)

awk ‘a !~ $0; {a=$0}’
Remove duplicate, nonconsecutive lines

awk ‘! a[$0]++’ # most concise script
awk ‘!($0 in a) {a[$0];print}’ # most efficient script
Concatenate every 5 lines of input, using a comma separator between fields

awk ‘ORS=%NR%5?”,”:”\n”‘ file
Selective Printing of Certain Lines

Print first 10 lines of file (emulates behavior of “head”)

awk ‘NR < 11' Print first line of file (emulates "head -1") awk 'NR>1{exit};1′
Print the last 2 lines of a file (emulates “tail -2”)

awk ‘{y=x “\n” $0; x=$0};END{print y}’
Print the last line of a file (emulates “tail -1”)

awk ‘END{print}’
Print only lines which match regular expression (emulates “grep”)

awk ‘/regex/’
Print only lines which do NOT match regex (emulates “grep -v”)

awk ‘!/regex/’
Print the line immediately before a regex, but not the line containing the regex

awk ‘/regex/{print x};{x=$0}’
awk ‘/regex/{print (x==”” ? “match on line 1” : x)};{x=$0}’
Print the line immediately after a regex, but not the line containing the regex

awk ‘/regex/{getline;print}’
Grep for AAA and BBB and CCC (in any order)

awk ‘/AAA/; /BBB/; /CCC/’
Grep for AAA and BBB and CCC (in that order)

awk ‘/AAA.*BBB.*CCC/’
Print only lines of 65 characters or longer

awk ‘length > 64’
Print only lines of less than 65 characters

awk ‘length < 64' Print section of file from regular expression to end of file awk '/regex/,0' awk '/regex/,EOF' Print section of file based on line numbers (lines 8-12, inclusive) awk 'NR==8,NR==12' Print line number 52 awk 'NR==52' awk 'NR==52 {print;exit}' # more efficient on large files Print section of file between two regular expressions (inclusive) awk '/Iowa/,/Montana/' # case sensitive Selective Deletion of Certain Lines: Delete ALL blank lines from a file (same as "grep '.' ") awk NF awk '/./' awk one-liner Tips Print column1, column5 and column7 of a data file or output of any columns list $awk ‘{print $1, $5, $7}’ data_file $cat file_name |awk ‘{print $1 $5 $7}’ $ls –al |awk ‘{print $1, $5, $7}’ -- Prints file_permissions,size and date Syntax of running an awk program Awk ‘program’ input file(s) List all files names whose file size greater than zero. $ls –al |awk ‘$5 > 0 {print $9}’

List all files whose file size equal to 512bytes.

$ls –al |awk ‘$5 == 0 {print $9}’

print all lines

$awk ‘{print }’ file_name

$awk ‘{print 0}’ file_name

Number of lines in a file

$awk ‘ END {print NR}’ file_name

Number of columns in each row of a file

$awk ‘ {print NF’} file_name

Sort the output of file and eliminate duplicate rows

$awk ‘{print $1, $5, $7}’ |sort –u

List all file names whose file size is greater than 512bytes and owner is “oracle”

$ls –al |awk ‘$3 == “oracle” && $5 > 512 {print $9}’

List all file names whose owner could be either “oracle” or “root”

$ls –al |awk ‘$3 == “oracle” || $3 == “root” {print $9}’

list all the files whose owner is not “oracle

$ls –al |awk ‘$3 != “oracle” {print $9}’

List all lines which has atlease one or more characters

$awk ‘NF > 0 {print }’ file_name

List all lines longer that 50 characters

$awk ‘length($0) > 50 ‘{print }’ file_name

List first two columns

$awk ‘{print $1, $2}’ file_name

Swap first two columns of a file and print

$awk ‘{temp = $1; $1 = $2; $2 = temp; print }’ file_name

Replace first column as “ORACLE” in a data file

$awk ‘{$1 = “ORACLE”; print }’ data_file

Remove first column values in a data file

$awk ‘{$1 =””; print }’ data_file

Calculate total size of a directory in Mb

$ls –al |awk ‘{total +=$5};END {print “Total size: ” total/1024/1024 ” Mb”}’

Calculate total size of a directory including sub directories in Mb

$ls –lR |awk ‘{total +=$5};END {print “Total size: ” total/1024/1024 ” Mb”}’

Find largest file in a directory including sub directories

$ls –lR |awk ‘{print $5 “\t” $9}’ |sort –n |tail -1

awk

awk [ -f progfile ] [ -Fc ] [ ‘prog’ ] [ parameters ] [ filename ] …
An awk program is a sequence of pattern-action commands, each of the form
pattern{action}
awk scans each input filename (multiple ones can be given and a hyphen, -, can be used as the “filename” for standard input) for lines which match patterns specified in a program given either by the prog string (which is enclosed in single quotes) or in the file progfile.
If no filename present, the program operates on standard input.
For each pattern in the program, the corresponding action is done on all lines of the input which the pattern matches.
-f progfile
Take the pattern-action statments/commands from progfile rather than from prog.
-Fc
Use character c, rather than space/blank as the field separator on each line of the input.
awk Commands

Examples

% cat in
Alice 20 4
Bob 12 3
Carol 18 3
Dave 24 4
Ed 30 5
%
% awk ‘/0/{print}’ in
Alice 20 4
Ed 30 5
%
% awk ‘{print NR”:”$0″ =”$3″-“$2”-“$1}’ < in 1:Alice 20 4 =4-20-Alice 2:Bob 12 3 =3-12-Bob 3:Carol 18 3 =3-18-Carol 4:Dave 24 4 =4-24-Dave 5:Ed 30 5 =5-30-Ed % % awk -F2 '{print "="$2"="$1"="}' in =0 4=Alice = = 3=Bob 1= ==Carol 18 3= =4 4=Dave = ==Ed 30 5= % % awk 'BEGIN{print "Hi"}{s+=$2}END{print "Sum2=",s}' in Hi Sum2= 104 % % awk '$3==4{s+=$2}END{print s}' in 44 % ls -l | awk '/^d/{d++}/^-/{f++}END{print d" dirs + "f" files"}' 3 dirs + 6 files % % awk '$2~/[3-8]/{print;s+=$3}END{print s}' in Carol 18 3 Dave 24 4 Ed 30 5 12 % % awk '{printf "%s = %4.1f\n",$0,$2/$3}' in Alice 20 4 = 5.0 Bob 12 3 = 4.0 Carol 18 3 = 6.0 Dave 24 4 = 6.0 Ed 30 5 = 6.0 % % awk '{s+=$2;t+=$3}END{printf "%4d/%d=%4.2f\n",s,t,s/t}' in 104/19=5.47 % % awk 'BEGIN{step=2}{n++; if(n>=step){print NR”: “$0;n=0}}’ in

or

% awk ‘{n++; if(n>=step){print NR”: “$0;n=0}}’ step=2 in
2: Bob 12 3
4: Dave 24 4
%
Explanations

awk ‘/0/{print}’ in
Print all lines from the file in which contain a “0”
awk ‘{print $0 ” = ” $3 “-” $2 “-” $1}’ < in Print each line preceded by its number (and a colon) and followed by the 3 fields of the line in reverse order, using "=" and "-" as separators. Note: NR = "Number of this Record," a built-in variable. awk -F2 '{print "="$2"="$1"="}' in Using "2" as a field separator, print fields 2 and 1 in that order, using "=" to delimit them. awk 'BEGIN{print "Hi"}{s+=$2}END{print "Sum2=",s}' in Print "Hi" at the start, sum up field 2 of each line, and print the sum at the end. Note the use of BEGIN and END as special patterns. Note also that "+=" means that s is to be incremented by the value of the 2nd field and that there is no "$" before the s. awk '$3==4{s+=$2}END{print s}' in Print the sum of the 2nd field of each line whose 3rd field is 4. Note the use of "==" to test for equality. ls -l|awk '/^d/{d++}/^-/{f++}END{print d" dirs + "f" files"}' Count separately the lines beginning "d" and those beginning "-" Note that ++ means that the variable's value is incremented by 1. awk '$2~/[3-8]/{print;s+=$3}END{print s}' in Print each line whose 2nd field matches the pattern "[3-8]" and then print the sum of the 3rd field of each of these lines. awk '{printf "%s = %4.1f\n",$0,$2/$3}' in Append to each line field 2 divided by field 3. awk '{s+=$2;t+=$3}END{printf "%4d/%d=%4.2f\n",s,t,s/t}' in Total columns 3 and 4 and print the average of the totals. % awk 'BEGIN{step=2}{n++; if(n>=step){print NR”: “$0;n=0}}’ in
% awk ‘{n++; if(n>=step){print NR”: “$0;n=0}}’ step=2 in
Print, preceded by its line number, every line whose number is a multiple of step.
One can also do this as follows:
awk ‘{if(step<=++n){print NR": "$0;n=0}}' step=2 in or even awk '0==NR%step{print NR": "$0}' step=2 in awk programs A awk program consists of one or more commands of the form: pattern{action} The action is performed on all lines of the input which match the pattern. Every input line matches an empty pattern. awk patterns empty action is done for each line. BEGIN action is done before first line. END action is done after last line. /RegularExpression/ action is done if the line matches RegularExpression. PatternMatchingExpression action is done if the PatternMatchingExpression is true for this line. Such expressions are composed of the ~ (match) and !~ (no match) operators RelationalExpression action is done if the RelationalExpression is true for this line. The relational operations are: < <= == != >= >
BooleanExpression
action is done if the BooleanExpression is true for this line.
Such expressions are the combination of pattern matching and relational expressions using boolean operations (&&, ||, and !) and (if/as needed) parentheses.
awk Built-in Variables

FILENAME
Name of the current input file
FS
input Field Separator regular expression (default blank and tab)
NF
Number of Fields in the current record
NR
Number of the current Record
OFMT
Output ForMaT for numbers (default %.6g)
OFS
Output Field Separator (default blank)
ORS
Output Record Separator (default newline)
RS
input Record Separator (default newline)
Special Characters and Regular Expressions in awk

\ Combines with following character to give it special meaning (see below) or, if it would have had a special meaning without the \, to make it revert to its literal meaning.
\a Alert/Bell (CTRL/G = ASCII 7)
\b Backspace (CTRL/H = ASCII 8)
\f Form feed (CTRL/L = ASCII 12)
\n Newline (CTRL/J = ASCII 10)
\r Carriage return (CTRL/M = ASCII 13)
\t Tab (CTRL/I = ASCII 9)
\/ Literal slash (in regular expressions)
\nnn Octal value nnn
. Match any character
^ Match start of line
$ Match end of line
[…] Match any character in brackets
Example: [abcA-Z7]
[^…] Match any character except those in brackets
Example: [^abcA-Z7]
* Match 0 or more repetitions of previous item
+ Match 1 or more repetitions of previous item
? Match 0 or 1 repetitions of previous item
(…) Treat enclosed text as a group/item
| Separator for items which are considered alternatives.
Example: (NY|LA|SF)
Examples

% cat in
Alice 20 4
Bob 12 3
Carol 18 3
Dave 24 4
Ed 30 5
% awk ‘/^[^A-CE]/{print}’ in
Dave 24 4
% awk ‘($1~/e/)||(NR>4){print}’ in
Alice 20 4
Dave 24 4
Ed 30 5
% awk ‘($1~/e/)&&($2\!~/0/){print}’ in
Dave 24 4
% sh
$ awk ‘($1~/e/)&&($2!~/0/){print}’ in
Dave 24 4
$ echo $0
sh
$ exit
% echo $0
tcsh
% awk ‘{s+=$2*$3}END{print s}’ in
416
% ls -la ~ | awk ‘/^-/{s+=$5;n++}END{print n” files, Avg=”s/n” bytes”}’
32 files, Avg=1376.21 bytes
awk Arithmetic

+ – * / % ^
Addition, subtraction, multiplication, division, modulus(remainder), exponentiation.
++ —
Increment and Decrement the value of a variable: prefix (++x) before the value is used or postfix (x++) after it is used.
= += -= *= /= %= ^=
Assignment: x ?= y is the same as x = x ? y
String concatenation is indicated by a blank or simple juxtaposition when token boundaries are clear.
awk Statements

An awk action is a sequence of statments, each terminated by a semicolon, newline, or right brace.
if ( expression ) statement [else statement ]
while ( expression ) statement
do statement while ( expression ) # In nawk
for ( expression ;expression ; expression) statement
for ( var in array ) statement
break
continue
{ [ statement ] . . . }
expression # commonly variable = expression
print [ expression-list ] [ > expression ]
printf format [ ,expression-list ] [ > expression ]
next # skip remaining patterns on this input line
exit [expr] # skip the rest of the input; exit status is expr
Examples

% cat in
Alice 20 4
Bob 12 3
Carol 18 3
Dave 24 4
Ed 30 5
%
% awk ‘$2<20{print $1" is small"}' in Bob is small Carol is small % % awk '{if($2<20)print $1" is small"}' in Bob is small Carol is small % % awk '{if($2<20)print $1" is small " else print $1 " is big"}' in awk: syntax error near line 1 awk: illegal statement near line 1 % % awk '{if($2<20){print $1" is small "} else print $1 " is big"}' in Alice is big Bob is small Carol is small Dave is big Ed is big % % awk '{if ($2<20) print $1" is small " else {print $1 " is big"}}' in awk: syntax error near line 1 awk: illegal statement near line 1 awk: syntax error near line 1 awk: bailing out near line 1 % % awk 'END{while(i++<3)print i}' < /dev/null 1 2 3 % % awk 'END{while(++i<3)print i}' < /dev/null 1 2 % % awk 'END{do print i}while (++i<3)}' < /dev/null awk: syntax error near line 1 awk: illegal statement near line 1 awk: bailing out near line 1 % % nawk 'END{do{print i}while (++i<3)}' < /dev/null 1 2 % % nawk 'END{while(++i<3)print i}' < /dev/null 1 2 % % awk 'END{for(i=0;i<3;i++)print i}' < /dev/null 0 1 2 % % awk 'END{for(;i<3;i++)print i}' < /dev/null 1 2 % % % cat in2 Alice:20:4 Bob:12:3 Carol:18:3 Dave:24:4 Ed:30:5 % % cat 2.awk /^[A-C]/{print "A-C: "$0} /e/{print "--e: "$0} % % awk -F':' -f 2.awk in2 A-C: Alice:20:4 --e: Alice:20:4 A-C: Bob:12:3 A-C: Carol:18:3 --e: Dave:24:4 % % cat 2n.awk /^[A-C]/{print "A-C: "$0; next} /e/{print "--e: "$0} % % awk -F: -f 2n.awk in2 A-C: Alice:20:4 A-C: Bob:12:3 A-C: Carol:18:3 --e: Dave:24:4 % % awk -F: '/o/{exit}END{print "Exit at\n"NR"="$0}' in2 Exit at 2=Bob:12:3 % awk Arrays awk's array variables are associative arrays. Each array variable is, in fact, a collection of variables written in the form avar[index] where avar is a name they share and index (called a "subscript" or "index") can be an integer or string value. Like all other, user-defined variables, avar[index] is automatically created when it is first used. Thus the array variable avar is created and extended simply by being used. Examples % cat in Alice 20 4 Bob 12 3 Carol 18 3 Dave 24 4 Ed 30 5 % % % awk '{myline[$1]=NR}END{for(n in myline)print n"="myline[n]}' in Ed=5 Bob=2 Alice=1 Dave=4 Carol=3 % % awk '{name[NR]=$1}END{for(n in name)print n"="name[n]}' in 2=Bob 3=Carol 4=Dave 5=Ed 1=Alice % % awk '{name[NR]=$1}END{for(n=1;n<=NR;n++)print n"="name[n]}' in 1=Alice 2=Bob 3=Carol 4=Dave 5=Ed % % cat in3 Alice 20 Bob 12 Carol 18 Dave 24 Ed 30 Alice 16 Bob 12 Carol 12 Dave 14 Ed 20 % % awk '{sum[$1]+=$2}END{for(n in sum)print n"="sum[n]}' in3 Ed=50 Bob=24 Alice=36 Dave=38 Carol=30 % awk functions awk has a generous set of built-in functions. The on-line form of Chapter 11: The awk Programming Language of UNIX in a Nutshell: System V Edition, 3rd Edition by Arnold Robbins, has an excellent Group Listing and Alphabetic Summary of awk Functions and Commands. Standard/minimal awk doesn't include user-defined functions. nawk, gawk, and other extended versions of awk do provide user-defined functions. Examples % cat in.val Alice 1 2 3 4 Bob 2 5 4 1 Carol 3 2 4 2 Dave 4 3 2 1 Ed 5 3 4 2 % % awk -f valsum.awk in.val awk: syntax error near line 4 awk: bailing out near line 4 % % cat valsum.awk # valsum.awk -- sum values in each record # input: name followed by a series of values function sum(s) { for (i=2; i<=NF; ++i) s+=$i return s } {print($0" = " sum())} % % nawk -f valsum.awk in.val Alice 1 2 3 4 = 10 Bob 2 5 4 1 = 12 Carol 3 2 4 2 = 11 Dave 4 3 2 1 = 10 Ed 5 3 4 2 = 14 % % nawk -f valsort.awk in.val Alice: 1 2 3 4 Bob: 1 2 4 5 Carol: 2 2 3 4 Dave: 1 2 3 4 Ed: 2 3 4 5 % % cat valsort.awk # valsort.awk -- sort values in each record # input: name followed by a series of values # based on grade.sort.awk script from chapter 9 of # "sed & awk, 2nd ed," Dougherty & Robbins, O'Reilly, 1997 # sort function -- sort numbers in ascending order function sort(A,n) { for (i=2; i<=n; ++i) for (j=i; (j>1) in A && A[j-1]>A[j]; –j) {
tmp=A[j]; A[j]=A[j-1]; A[j-1]=tmp
}
return
}

# main routine
{
for (i=2; i<=NF; ++i) val[i-1]=$i sort(val, NF-1) printf("%7s: ", $1) for (j=1; j<=NF-1; ++j) printf("%d ", val[j]) printf("\n") }