Structure¶
BEGIN { <run once before input> }
pattern { action }
pattern { action }
END { <run once after input> }
awk -F: '{print $1, $NF}' /etc/passwd # -F: sets the field separator
awk '
BEGIN { print "start" }
!/(login|shutdown)/ { print NR, $0 }
END { print "done" }
' /etc/passwd
| Item | Meaning |
|---|---|
$0 |
whole line |
$1, $2, ... |
fields |
$NF |
last field; $(NF-1) second to last |
NR |
line number (all input); FNR per file |
NF |
number of fields in this line |
FS, OFS |
input / output field separator |
Quick references: QuickRef.ME awk and the GNU awk manual (info gawk).
Everyday one-liners¶
awk '{ print $1 }' file # first column
awk '{ print $6 " " $11 }' file | sort -k1,1 -r
awk '$3 > 100 { print $1, $3 }' file # filter on a numeric field
awk '{ sum += $2 } END { print sum }' file # total a column
awk '{ sum += $2 } END { print sum/NR }' file # average
# Lines that are not comments and not blank
awk '!/^ *#/ && NF' file
# Remove duplicate lines, keep the first (order preserved)
awk '!seen[$0]++' file1 > file2
# Insert a blank line after every 2nd line
awk '{ print } NR % 2 == 0 { print "" }' in.txt > out.txt
# Print substring: 3 characters starting at position 7 / drop the first 36 characters
awk '{ print substr($0, 7, 3) }' file
awk '{ print substr($0, 37) }' file
# Last three characters of each line
awk '{ print substr($0, length($0)-2) }' file
Package names without version¶
rpm -qa output such as plymouth-core-libs-0.9.3-16.el8.x86_64 Wed Dec 2 15:54:35 2020:
rpm -qa --qf '%{NAME}\n' | grep plymouth # best: ask rpm for just the name
rpm -qa | grep plymouth | awk -F'-[0-9]' '{ print $1 }' # split at the first "-<digit>"
Splitting on a fixed string like -0 (as older notes did) breaks on versions that start with another digit. Splitting on -[0-9] (a dash followed by any digit) is safer.
Extract fields from log lines¶
grep 'Relay access denied' /var/log/maillog | awk '{print $10}'
# -> host.example.net[203.0.113.99]
# then keep only what is inside the brackets
grep 'Relay access denied' /var/log/maillog | awk '{print $10}' | awk -F'[][]' '{print $2}'
# -> 203.0.113.99
Feeding extracted addresses into a firewall automatically needs care: parse strictly and rate-limit, or attackers who control log content can get arbitrary addresses blocked. Use fail2ban for this.