So, let’s consider the following log file:
www.three.com 10.15.101.11 1353280801 TEST 345
www.one.com 10.14.101.11 1353280801 TEST 343
www.three.com 1.10.11.71 1353280801 TEST 323
www.one.com 10.15.11.61 1353280801 TEST 365
www.two.com 10.10.11.51 1353290801 TEST 55
www.two.com 10.20.13.11 1353290801 REST 435
www.one.com 10.20.14.41 1353290801 REST 65
www.two.com 10.10.11.14 1353290801 REST 345
www.three.com 10.10.11.31 1354280801 REST 34
www.one.com 10.10.13.144 1354280801 JSON 65
www.two.com 10.50.11.141 1354280801 JSON 665
www.three.com 120.10.11.11 1354280801 JSON 555
www.two.com 10.144.11.11 1383280801 RAW 33
www.one.com 10.103.141.141 1383280801 RAW 315
Now, here are some things you can do to this log file:
How many files are in a directory: ls | wc -l
Print the file: cat sample.log
Print lines that match a particular word: grep “RAW” sample.log
Print those lines to a file called test.log: grep “RAW” sample.log > test.log
Print particular columns and sort: cat sample.log | awk ‘{ print $1,$2}’ | sort -k 1
Find and Replace using SED and Regex: cat sample.log | sed ‘s/TEST/JSON/g’
Split a log file into multiple files using a column as name with AWK: awk ‘{ print >>($4?.log”); close($4?.log”) }’ sample.log
Use substr (removes last character) in AWK to manipulate a string per line: cat sample.log | awk ‘{ print $1,$2,$3,substr($4,1,length($4)-1),$5}’
Print first line of file with SED: sed q test.log
Print last line of file with SED: sed ‘$!d’ sample.log
Perform a regular expression on last character of entire file using SED: cat sample.log | sed ‘$ s/5$//’
Add some text to beginning and end of a file with AWK: cat sample.log | awk ‘BEGIN{print “START” } { print } END{print “END”}’
Count and print how many unique fields are in all rows using AWK: cat sample.log | awk ‘{ if (a[$1]++ == 0) print $1 }’ | wc -l
Make everything lowercase with AWK: cat sample.log | awk ‘{print tolower($0)}’
Multiple SED regular expressions: sed ’1s/^/START/;$ s/5$/END/’ sample.log
Regex with SED on multiple files: for file in *; do sed ’1s/^/START/’ $file > $file’.json’; done
n a command on each file in directory: for i in `ls`; do $i; done
Rename the extension of all files in a folder: for old in *.txt; do mv $old `basename $old .txt`.json; done
Merge all files in a directory with a comma seperator: find . -type f -not -name output.txt -exec cat {} ; -exec echo `,` ; > output.txt
Recent Comments