grep, cat, zgrep and zcat<\/p>\n
More on log parsing, I\u2019m taking notes on how to read log files and get the information that I need. On Linux environment, these tools are perfect: grep, cat, zgrep and zcat.<\/p>\n
Extracting patterns with grep<\/p>\n
We can extract information from a text file using grep. Example, we can extract lines of log file containing patterns like GET \/checkout\/* where status code is 500.<\/p>\n
1
\ngrep -E -e ‘GET \/checkout\/.* HTTP\/1\\.(0|1)” 500’ some-log-file.log
\nDepending on the Apache log format, above will extract lines whose request is \/checkout\/* and status code is 500 where it may support HTTP\/1.0 or HTTP\/1.1. However, that would extract the whole line. To only extract the matching pattern, use the -o option.<\/p>\n
1
\ngrep -o -E -e ‘GET \/checkout\/.* HTTP\/1\\.(0|1)” 500’ some-log-file.log
\nAnd to save the matching patterns to a file, simply redirect the output to file.<\/p>\n
1
\ngrep -E -e ‘GET \/checkout\/.* HTTP\/1\\.(0|1)” 500’ some-log-file.log > checkout-errors.txt
\nUsing cat<\/p>\n
cat is usually used to output contents of a file. This is a small but very useful Linux utility. For example, we can combine multiple log files (uncompressed) into a single log file.<\/p>\n
1
\ncat \/path\/to\/log-files\/*.log > \/combined\/log-file.log
\nCompressed counterpart<\/p>\n
grep and cat have their compressed file counterpart. For grep, there\u2019s zgrep.<\/p>\n
1
\nzgrep -E -e ‘GET \/checkout\/.* HTTP\/1\\.(0|1)” 500’ some-log-file.gz > checkout-errors.txt
\nFor cat, there\u2019s zcat.<\/p>\n
1
\nzcat \/path\/to\/log-files\/*.gz > \/combined\/log-file.log
\nI\u2019ve done so many combination last week that I don\u2019t remember them all and not able to include in this post. Happy log parsing.<\/p>\n
# List out successful ssh login attempts
\ncat secure | grep ‘Accepted’ | awk ‘{print $1 ” ” $2 ” ” $3 ” User: ” $9 ” ” }’
\ncat secure* | sort | grep ‘Accepted’ | awk ‘{print $1 ” ” $2 ” ” $3 ” User: ” $9 ” IP:” $11 }’<\/p>\n
# List out successful ssh login attempts from sudo users
\ncat \/var\/log\/secure | grep ‘session opened for user root’ | awk ‘{print $1 ” ” $2 ” ” $3 ” Sudo User: ” $13 ” ” }’<\/p>\n
# List out ssh login attempts from non-existing and unauthorized user accounts
\ncat \/var\/log\/secure | grep ‘Invalid user’<\/p>\n
# List out ssh login attempts by authorized ssh accounts with failed password
\ncat \/var\/log\/secure | grep -v invalid | grep ‘Failed password’<\/p>\n
Indeed, and even grep | awk can be shortened to awk \/\u2026\/. So you could save a bit of space in the final script. For a typical log file (~200 kb), you might save 1 ms processing it. Or to be exact, 1.8 ms removing the cat and grep, and 0.3 ms using only awk instead of grep | awk.<\/p>\n
time for i in `seq 1000`; do cat secure | grep Accepted | awk ‘{print $1 ” ” $2 ” ” $3 ” User: ” $9 ” ” }’; done > \/tmp\/a
\ntime for i in `seq 1000`; do grep Accepted secure | awk ‘{print $1 ” ” $2 ” ” $3 ” User: ” $9 ” ” }’; done > \/tmp\/b
\ntime for i in `seq 1000`; do awk ‘\/Accepted\/ {print $1 ” ” $2 ” ” $3 ” User: ” $9 ” ” }’ secure; done > \/tmp\/c<\/p>\n
However, more interestingly, when the size of the log file is increased to 200 MB, it turns out that the cat | grep | awk chain is significantly faster, at 1.096 s over 100 runs. The single awk command will not max out the CPUs, while the pipe chain does.<\/p>\n
for i in `seq 1000`; do cat secure >> s; done
\ntime for i in `seq 100`; do cat s | grep Accepted | awk ‘{print $1 ” ” $2 ” ” $3 ” User: ” $9 ” ” }’; done > \/tmp\/a1
\ntime for i in `seq 100`; do awk ‘\/Accepted\/ {print $1 ” ” $2 ” ” $3 ” User: ” $9 ” ” }’ s; done > \/tmp\/c1<\/p>\n
My perl script awk:
\nCode:
\nmy $OldTime= ‘Sep 10, 2012 5:20:41′;
\nmy $NewTime=’Sep 10, 2012 5:49:40’;<\/p>\n
my $test2 = qx{ssh -o stricthostkeychecking=no $WLS “awk ‘\/$OldTime\/,\/$NewTime\/’ $WLSP\/logs\/CDSServer.* “}; my $startTime = $ARGV[0]; # make sure the log file exists # convert date\/time strings to seconds since epoch print “Start time: $startTime ($start_sec)\\n”; open(LOG,’<',$log) or die \"can't read '$log': $!\\n\";\nwhile( # convert date\/time string in log entro to epoch seconds # print line if it falls into the range my $startTime = $ARGV[0]; # make sure the log file exists # convert date\/time strings to seconds since epoch # make sure we got nothing but digits in the variables print “Start time: $startTime ($start_sec)\\n”; my @lines; open(LOG,’<',$log) or die \"can't read '$log': $!\\n\";\nwhile( # convert date\/time string in log entro to epoch seconds # save line w\/time string to array if it falls into the range # print the saved lines Did small change: my $startTime = $ARGV[0]; # make sure the log file exists # convert date\/time strings to seconds since epoch # make sure we got nothing but digits in the variables print “Start time: $startTime ($start_sec)\\n”; my @lines; open(LOG,’<',$log) or die \"can't read '$log': $!\\n\";\nwhile( # convert date\/time string in log entro to epoch seconds # save line w\/time string to array if it falls into the range # print the saved lines grep, cat, zgrep and zcat<\/p>\n More on log parsing, I\u2019m taking notes on how to read log files and get the information that I need. On Linux environment, these tools are perfect: grep, cat, zgrep and zcat.<\/p>\n Extracting patterns with grep<\/p>\n We can extract information from a text file using grep. Example, we can extract […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[47],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/3415"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3415"}],"version-history":[{"count":5,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/3415\/revisions"}],"predecessor-version":[{"id":3432,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/3415\/revisions\/3432"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3415"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3415"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3415"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nMy log file format:
\nCode:
\n####
\n####
\n####
\n E.g.: $0 ‘Sep 10, 2012 5:20:41 PM’ ‘Sep 10, 2012 5:49:40 PM’ CDSServer.log\\n”
\n unless($#ARGV == 2);<\/p>\n
\nmy $stopTime = $ARGV[1];
\nmy $log = $ARGV[2];<\/p>\n
\ndie “$log: No such file\\n” unless(-f$log);<\/p>\n
\nmy $start_sec = str2time($startTime);
\nmy $stop_sec = str2time($stopTime);<\/p>\n
\nprint “Stop time: $stopTime ($stop_sec)\\n”;<\/p>\n
\n chomp;
\n my $line = $_; # save original line
\n s\/[ \\t]+\/ \/; # replace contiguous white spaces w\/single space
\n if(\/^####<([a-zA-Z]{3} [0-9]{1,2}, [0-9]{4} [0-9]{1,2}:[0-9]{2}:[0-9]{2} [AP]M) [A-Z]{3}>\/){
\n my $timedate = $1;<\/p>\n
\n my $seconds = str2time($timedate);<\/p>\n
\n print $line,”\\n” if(($seconds >= $start_sec)&&($seconds <= $stop_sec));\n }\n}\nclose(LOG);\nReply With Quote Reply With Quote\n09-19-2012 #3\ncharith charith is offline\nJust Joined!\nJoin Date\nNov 2010\nPosts\n26\nHi atreyu,\n\nThank you very much for your great clean reply.\n\nYour script working fine and it gives lines those match given times but what i need is get all lines whatever between that time range. I'm sorry for my bad log file format i attached correct log file below.[Errors not begin with time] \n\nCode:\n####
\n###
\n E.g.: $0 ‘Sep 10, 2012 5:20:41 PM’ ‘Sep 10, 2012 5:49:40 PM’ CDSServer.log\\n”
\n unless($#ARGV == 2);<\/p>\n
\nmy $stopTime = $ARGV[1];
\nmy $log = $ARGV[2];<\/p>\n
\ndie “$log: No such file\\n” unless(-f$log);<\/p>\n
\nmy $start_sec = str2time($startTime);
\nmy $stop_sec = str2time($stopTime);<\/p>\n
\ndie “Failed to convert $startTime to seconds\\n” unless($start_sec =~ \/^[0-9]*$\/);
\ndie “Failed to convert $stopTime to seconds\\n” unless($stop_sec =~ \/^[0-9]*$\/);<\/p>\n
\nprint “Stop time: $stopTime ($stop_sec)\\n”;<\/p>\n
\nmy $stop;<\/p>\n
\n chomp;
\n my $line = $_; # save original line
\n s\/[ \\t]+\/ \/; # replace contiguous white spaces w\/single space
\n if(\/^####<([a-zA-Z]{3} [0-9]{1,2}, [0-9]{4} [0-9]{1,2}:[0-9]{2}:[0-9]{2} [AP]M) [A-Z]{3}>\/){
\n my $timedate = $1;<\/p>\n
\n my $seconds = str2time($timedate);<\/p>\n
\n if($seconds >= $start_sec){
\n push(@lines,$line) unless($stop);
\n }elsif($seconds >= $stop_sec){
\n push(@lines,$line) unless($stop);
\n $stop = 1;
\n }
\n }else{
\n # save line w\/o time string to array if it falls into the range
\n push(@lines,$line) if(($#lines>=0)&&!($stop));
\n }
\n}
\nclose(LOG);<\/p>\n
\nprint “$_\\n” for(@lines);
\nReply With Quote Reply With Quote
\n09-24-2012 #5
\ncharith charith is offline
\nJust Joined!
\nJoin Date
\nNov 2010
\nPosts
\n26
\nHi atreyu,
\nIt’s working fine thank you very much.<\/p>\n
\nCode:
\n# save line w\/time string to array if it falls into the range
\n if($seconds >= $start_sec){
\nas
\nCode:
\nif (($seconds >= $start_sec)&&($seconds <= $stop_sec)){ }\n}\nclose(LOG);\n\n#!\/usr\/bin\/perl\nuse strict;\nuse warnings;\nuse Date::Parse;\n\n# get command line arguments (3)\ndie \"\n Usage: $0 '
\n E.g.: $0 ‘Sep 10, 2012 5:20:41 PM’ ‘Sep 10, 2012 5:49:40 PM’ CDSServer.log\\n”
\n unless($#ARGV == 2);<\/p>\n
\nmy $stopTime = $ARGV[1];
\nmy $log = $ARGV[2];<\/p>\n
\ndie “$log: No such file\\n” unless(-f$log);<\/p>\n
\nmy $start_sec = str2time($startTime);
\nmy $stop_sec = str2time($stopTime);<\/p>\n
\ndie “Failed to convert $startTime to seconds\\n” unless($start_sec =~ \/^[0-9]*$\/);
\ndie “Failed to convert $stopTime to seconds\\n” unless($stop_sec =~ \/^[0-9]*$\/);<\/p>\n
\nprint “Stop time: $stopTime ($stop_sec)\\n”;<\/p>\n
\nmy $stop;<\/p>\n
\n chomp;
\n my $line = $_; # save original line
\n s\/[ \\t]+\/ \/; # replace contiguous white spaces w\/single space
\n if(\/^####<([a-zA-Z]{3} [0-9]{1,2}, [0-9]{4} [0-9]{1,2}:[0-9]{2}:[0-9]{2} [AP]M) [A-Z]{3}>\/){
\n my $timedate = $1;<\/p>\n
\n my $seconds = str2time($timedate);<\/p>\n
\n if($seconds >= $start_sec){
\n push(@lines,$line) unless($stop);
\n }elsif($seconds >= $stop_sec){
\n push(@lines,$line) unless($stop);
\n $stop = 1;
\n }
\n }else{
\n # save line w\/o time string to array if it falls into the range
\n push(@lines,$line) if(($#lines>=0)&&!($stop));
\n }
\n}
\nclose(LOG);<\/p>\n
\nprint “$_\\n” for(@lines);<\/p>\n","protected":false},"excerpt":{"rendered":"