{"id":2786,"date":"2014-02-10T23:26:26","date_gmt":"2014-02-10T15:26:26","guid":{"rendered":"http:\/\/rmohan.com\/?p=2786"},"modified":"2014-07-23T23:34:22","modified_gmt":"2014-07-23T15:34:22","slug":"apache-log-counting-using-awk-and-seed","status":"publish","type":"post","link":"https:\/\/mohan.sg\/?p=2786","title":{"rendered":"Apache Log Counting using Awk and Sed"},"content":{"rendered":"

Since I (and you as a visitor) don\u2019t want your IP-address to be spread around the internet, I\u2019ve anonymized the log data. It\u2019s a fairly easy process that is done in 2 steps:<\/span><\/h2>\n
    \n
  1. IP\u2019s are translated into random values.<\/li>\n
  2. Admin url\u2019s are removed.<\/li>\n<\/ol>\n

    Step 1: Translating IP\u2019s<\/h3>\n

    All the IP\u2019s are translated into random IP\u2019s, but every IP has it\u2019s own random counterpart. This means that you can still identify users who are browsing through the site. The actual command I have used for this is:<\/p>\n

    cat apache-anon-noadmin.log | awk 'function ri(n) {  return int(n*rand()); }\u00a0 \\\r\nBEGIN { srand(); }\u00a0 { if (! ($1 in randip)) {  \\\r\nrandip[$1] = sprintf(\"%d.%d.%d.%d\", ri(255), ri(255), ri(255), ri(255)); } \\\r\n$1 = randip[$1]; print $0\u00a0 }'<\/pre>\n
    If you read a bit further we will find out what this will actually do, but most of it you should be able to understand (a least the global format).<\/p>\n
    Step 2: Removing admin url\u2019s<\/h3>\n
    I don\u2019t like that everybody can view all the admin-requests I\u2019ve done on the site. Luckely this is a very simple process. We only have to remove the requests that start with \u201c\/wp-admin\u201d. This can be done by an inverse grep-command:<\/p>\n
    cat apache-anon.log | grep -v '\/wp-admin' > apache-anon-noadmin.log<\/pre>\n
     <\/p>\n
    Example 1: count http status codes<\/h2>\n
    For now we want to deal with the status-codes. This is found in field $9. The following code will print every field 9 for every record from our log:<\/p>\n
    cat apache-anon-noadmin.log | awk ' { print $9 } '<\/pre>\n
    That\u2019s nice, but let\u2019s aggregate this data. We want to know how many times we outputted each status code. By using the \u201cuniq\u201d command, we can count (and display) the number of times we encounter data, but before we can use uniq we have to sort the data since uniq will stop counting as soon as another piece of data is encountered. (try the following line with and without the \u201csort\u201d to see what I mean).<\/p>\n
    cat apache-anon-noadmin.log | awk ' { print $9 } ' | sort | uniq -c<\/pre>\n
    And the output should be:<\/p>\n
    72951 200\r\n  235 206\r\n 1400 301\r\n   38 302\r\n 2911 304\r\n 2133 404\r\n 1474 500<\/pre>\n
    As you see, the 200 (which stands for OK), is returned 72951 times, while we returned 2133 times a 404 (page not found). Cool\u2026<\/p>\n
     <\/p>\n
    Example 2: top 10 of visiting ip\u2019s<\/h2>\n
    Let\u2019s try to create some top-10?s. The first one about the IP\u2019s that did the most pageviews (my fans, but most probably it would be me :p)<\/p>\n
    cat apache-anon-noadmin.log | awk '{ print $1 ; }' | \\\r\nsort | uniq -c | sort -n -r | head -n 10<\/pre>\n
    We use awk to print the first field \u2013 the IP, we sort and count them. THEN we sort again, but this time in a reversed order and with a natural sort so 10 will be sorted after 9, instead of after 1. (again, remove the sort to find out what I mean). After this, we filter out the first 10 lines with the head command, which only prints the first 10 lines.<\/p>\n
    As you can see, I use (a lot of) different unix commands to achieve what I need to do. It MIGHT be possible to this all with awk itself as well, but by using other commands we get the job done quick and easy.<\/p>\n
     <\/p>\n
    Example 3: traffic in kilobytes per status code<\/h2>\n
    Let\u2019s introduce arrays. Field $10 holds the number of bytes we have send out, and field $9 the status code. In the null-pattern (the block without any pattern, that will be executed on every line) we add the number of bytes to the array in the $9 index. It will NOT print out any information yet. At the end of the program, we will iterate over the \u201ctotal\u201d-array and print each status code, and the total sum of bytes \/ 1024, so we get kilobytes. Still, pretty easy to understand.<\/p>\n
    cat apache-anon-noadmin.log\u00a0 | awk ' { total[$9] += $10 } \\\r\nEND {\u00a0 for (x in total) { printf \"Status code %3d : %9.2f Kb\\n\", x, total[x]\/1024 } } '\r\nStatus code 200 : 329836.22 Kb\r\nStatus code 206 :\u00a0  4649.29 Kb\r\nStatus code 301 :\u00a0\u00a0  535.72 Kb\r\nStatus code 302 :\u00a0\u00a0 \u00a0 20.26 Kb\r\nStatus code 304 :\u00a0\u00a0  572.77 Kb\r\nStatus code 404 :\u00a0  5106.29 Kb\r\nStatus code 500 :\u00a0  2336.42 Kb<\/pre>\n
    Not a lot of redirections, but still: 5 megabyte wasted by serving pages that are not found \ud83d\ude41<\/p>\n
    Let\u2019s expand this example so we get a total sum:<\/p>\n
    cat apache-anon-noadmin.log\u00a0 | awk ' { totalkb += $10; total[$9] += $10 } \\\r\nEND {\u00a0 for (x in total) { printf \"Status code %3d : %9.2f Kb\\n\", x, total[x]\/1024 } \\\r\nprintf (\"\\nTotal send\u00a0\u00a0\u00a0\u00a0\u00a0 : %9.2f Kb\\n\", totalkb\/1024); } '\r\nStatus code 200 : 329836.22 Kb\r\nStatus code 206 :\u00a0\u00a0 4649.29 Kb\r\nStatus code 301 :\u00a0\u00a0\u00a0 535.72 Kb\r\nStatus code 302 :\u00a0\u00a0\u00a0\u00a0 20.26 Kb\r\nStatus code 304 :\u00a0\u00a0\u00a0 572.77 Kb\r\nStatus code 404 :\u00a0\u00a0 5106.29 Kb\r\nStatus code 500 :\u00a0\u00a0 2336.42 Kb\r\n\r\nTotal send\u00a0\u00a0\u00a0\u00a0\u00a0 : 343056.96 Kb<\/pre>\n
     <\/p>\n
    Example 4: top 10 referrers<\/h2>\n
    We use the \u201d as separator here. We need this because the referrer is inside those quotes. This is how we can deal with request-url\u2019s, the referrers and user-agents without problems. This time we don\u2019t use a BEGIN block to change the FS-variable, but we change it through a command line parameter. Now, most of the referrers are either from our own blog, or a \u2018-\u2019, when no referrer is given. We add additional grep commands to remove those referrers. Again, sorting, doing a unique count, reverse nat sorting and limiting with head gives us a nice result:<\/p>\n
    cat apache-anon-noadmin.log | awk -F\\\" ' { print $4 } ' | \\\r\ngrep -v '-' | grep -v 'http:\/\/www.adayinthelife' | sort | \\\r\nuniq -c | sort -rn | head -n 10\r\n 343 http:\/\/www.phpdeveloper.org\/news\/15544\r\n 175 http:\/\/www.dzone.com\/links\/rss\/top5_certifications_for_every_php_programmer.html\r\n 71 http:\/\/www.dzone.com\/links\/index.html\r\n 64 http:\/\/www.google.com\/reader\/view\/\r\n 54 http:\/\/www.phpdeveloper.org\/\r\n 50 http:\/\/phpdeveloper.org\/\r\n 49 http:\/\/www.dzone.com\/links\/r\/top5_certifications_for_every_php_programmer.html\r\n 45 http:\/\/www.phpdeveloper.org\/news\/15544?utm_source=twitterfeed&utm_medium=twitter\r\n 22 http:\/\/abcphp.com\/41578\/\r\n 21 http:\/\/twitter.com<\/pre>\n
    At least I can see quickly to which sites I need to send some christmas cards to.<\/p>\n
     <\/p>\n
    Example 5: top 10 user-agents<\/h2>\n
    How simple is this? The user-agent is in column 6 instead of 4 and we don\u2019t need the grep\u2019s, so this one needs no explanation:<\/p>\n
    cat apache-anon-noadmin.log | awk -F\\\" ' { print $6 } ' | \\\r\nsort | uniq -c | sort -rn | head -n 10\r\n 5891 Mozilla\/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit\/534.10 (KHTML, like Gecko) Chrome\/8.0.552.215 Safari\/534.10\r\n 4145 Mozilla\/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.12) Gecko\/20101026 Firefox\/3.6.12\r\n 3440 Mozilla\/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit\/534.10 (KHTML, like Gecko) Chrome\/8.0.552.215 Safari\/534.10\r\n 2338 Mozilla\/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.12) Gecko\/20101026 Firefox\/3.6.12\r\n 2314 Mozilla\/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.6) Gecko\/2009011912 Firefox\/3.0.6\r\n 2001 Mozilla\/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko\/20101026 Firefox\/3.6.12\r\n 1959 Mozilla\/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; en-US) AppleWebKit\/534.10 (KHTML, like Gecko) Chrome\/8.0.552.215 Safari\/534.10\r\n 1241 Mozilla\/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; en-us) AppleWebKit\/533.19.4 (KHTML, like Gecko) Version\/5.0.3 Safari\/533.19.4\r\n 1122 Mozilla\/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit\/534.10 (KHTML, like Gecko) Chrome\/8.0.552.215 Safari\/534.10\r\n 1010 Mozilla\/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.12) Gecko\/20101026 Firefox\/3.6.12<\/pre>\n
    There are many different packages that allow you to generate reports on who’s visiting your site and what they’re doing. The most popular at this time appear to be “Analog”, “The Webalizer” and “AWStats” which are installed by default on many shared servers.<\/p>\n
    While such programs generate attractive reports, they only scratch the surface of what the log files can tell you. In this section we look at ways you can delve more deeply – focussing on the use of simple command line tools, particularly grep, awk and sed.<\/p>\n
    1. Combined log format<\/p>\n
    The following assumes an Apache HTTP Server combined log format where each entry in the log file contains the following information:<\/p>\n
    %h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-agent}i”
    \nwhere:<\/p>\n
    %h = IP address of the client (remote host) which made the request
    \n%l = RFC 1413 identity of the client
    \n%u = userid of the person requesting the document
    \n%t = Time that the server finished processing the request
    \n%r = Request line from the client in double quotes
    \n%>s = Status code that the server sends back to the client
    \n%b = Size of the object returned to the client
    \nThe final two items: Referer and User-agent give details on where the request originated and what type of agent made the request.<\/p>\n
    Sample log entries:<\/p>\n
    66.249.64.13 – – [18\/Sep\/2004:11:07:48 +1000] “GET \/robots.txt HTTP\/1.0” 200 468 “-” “Googlebot\/2.1”
    \n66.249.64.13 – – [18\/Sep\/2004:11:07:48 +1000] “GET \/ HTTP\/1.0” 200 6433 “-” “Googlebot\/2.1″
    \nNote: The robots.txt file gives instructions to robots as to which parts of your site they are allowed to index. A request for \/ is a request for the default index page, normally index.html.<\/p>\n
    2. Using awk<\/p>\n
    The principal use of awk is to break up each line of a file into ‘fields’ or ‘columns’ using a pre-defined separator. Because each line of the log file is based on the standard format we can do many things quite easily.<\/p>\n
    Using the default separator which is any white-space (spaces or tabs) we get the following:<\/p>\n
    awk ‘{print $1}’ combined_log # ip address (%h)
    \nawk ‘{print $2}’ combined_log # RFC 1413 identity (%l)
    \nawk ‘{print $3}’ combined_log # userid (%u)
    \nawk ‘{print $4,5}’ combined_log # date\/time (%t)
    \nawk ‘{print $9}’ combined_log # status code (%>s)
    \nawk ‘{print $10}’ combined_log # size (%b)
    \nYou might notice that we’ve missed out some items. To get to them we need to set the delimiter to the ” character which changes the way the lines are ‘exploded’ and allows the following:<\/p>\n
    awk -F\\” ‘{print $2}’ combined_log # request line (%r)
    \nawk -F\\” ‘{print $4}’ combined_log # referer
    \nawk -F\\” ‘{print $6}’ combined_log # user agent
    \nNow that you understand the basics of breaking up the log file and identifying different elements, we can move on to more practical examples.<\/p>\n
    3. Examples<\/p>\n
    You want to list all user agents ordered by the number of times they appear (descending order):<\/p>\n
    awk -F\\” ‘{print $6}’ combined_log | sort | uniq -c | sort -fr
    \nAll we’re doing here is extracing the user agent field from the log file and ‘piping’ it through some other commands. The first sort is to enable uniq to properly identify and count unique user agents. The final sort orders the result by number and name (both descending).<\/p>\n
    The result will look similar to a user agents report generated by one of the above-mentioned packages. The difference is that you can generate this ANY time from ANY log file or files.<\/p>\n
    If you’re not particulary interested in which operating system the visitor is using, or what browser extensions they have, then you can use something like the following:<\/p>\n
    awk -F\\” ‘{print $6}’ combined_log \\
    \n| sed ‘s\/(\\([^;]\\+; [^;]\\+\\)[^)]*)\/(\\1)\/’ \\
    \n| sort | uniq -c | sort -fr
    \nNote: The \\ at the end of a line simply indicates that the command will continue on the next line.<\/p>\n
    This will strip out the third and subsequent values in the ‘bracketed’ component of the user agent string. For example:<\/p>\n
    Mozilla\/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR)
    \nbecomes:<\/p>\n
    Mozilla\/4.0 (compatible; MSIE 6.0)
    \nThe next step is to start filtering the output so you can narrow down on a certain page or referer. Would you like to know which pages Google has been requesting from your site?<\/p>\n
    awk -F\\” ‘($6 ~ \/Googlebot\/){print $2}’ combined_log | awk ‘{print $2}’
    \nOr who’s been looking at your guestbook?<\/p>\n
    awk -F\\” ‘($2 ~ \/guestbook\\.html\/){print $6}’ combined_log
    \nIt’s just too easy isn’t it!<\/p>\n
    Using just the examples above you can already generate your own reports to back up any kind of automated reporting your ISP provides. You could even write your own log analysis program.<\/p>\n
    4. Using log files to identify problems with your site<\/p>\n
    The steps outlined below will let you identify problems with your site by identifying the different server responses and the requests that caused them:<\/p>\n
    awk ‘{print $9}’ combined_log | sort | uniq -c | sort
    \nThe output shows how many of each type of request your site is getting. A ‘normal’ request results in a 200 code which means a page or file has been requested and delivered but there are many other possibilities.<\/p>\n
    The most common responses are:<\/p>\n
    200 – OK
    \n206 – Partial Content
    \n301 – Moved Permanently
    \n302 – Found
    \n304 – Not Modified
    \n401 – Unauthorised (password required)
    \n403 – Forbidden
    \n404 – Not Found
    \nNote: For more on Status Codes you can read the article HTTP Server Status Codes.<\/p>\n
    A 301 or 302 code means that the request has been re-directed. What you’d like to see, if you’re concerned about bandwidth usage, is a lot of 304 responses – meaning that the file didn’t have to be delivered because they already had a cached version.<\/p>\n
    A 404 code may indicate that you have a problem – a broken internal link or someone linking to a page that no longer exists. You might need to fix the link, contact the site with the broken link, or set up a PURL so that the link can work again.<\/p>\n
    The next step is to identify which pages\/files are generating the different codes. The following command will summarise the 404 (“Not Found”) requests:<\/p>\n
    # list all 404 requests
    \nawk ‘($9 ~ \/404\/)’ combined_log<\/p>\n
    # summarise 404 requests
    \nawk ‘($9 ~ \/404\/)’ combined_log | awk ‘{print $9,$7}’ | sort
    \nOr, you can use an inverted regular expression to summarise the requests that didn’t return 200 (“OK”):<\/p>\n
    awk ‘($9 !~ \/200\/)’ combined_log | awk ‘{print $9,$7}’ | sort | uniq
    \nOr, you can include (or exclude in this case) a range of responses, in this case requests that returned 200 (“OK”) or 304 (“Not Modified”):<\/p>\n
    awk ‘($9 !~ \/200|304\/)’ combined_log | awk ‘{print $9,$7}’ | sort | uniq
    \nSuppose you’ve identifed a link that’s generating a lot of 404 errors. Let’s see where the requests are coming from:<\/p>\n
    awk -F\\” ‘($2 ~ “^GET \/path\/to\/brokenlink\\.html”){print $4,$6}’ combined_log
    \nNow you can see not just the referer, but the user-agent making the request. You should be able to identify whether there is a broken link within your site, on an external site, or if a search engine or similar agent has an invalid address.<\/p>\n
    If you can’t fix the link, you should look at using Apache mod_rewrite or a similar scheme to redirect (301) the requests to the most appropriate page on your site. By using a 301 instead of a normal (302) redirect you are indicating to search engines and other intelligent agents that they need to update their link as the content has ‘Moved Permanently’.<\/p>\n
    5. Who’s ‘hotlinking’ my images?<\/p>\n
    Something that really annoys some people is when their bandwidth is being used by their images being linked directly on other websites.<\/p>\n
    Here’s how you can see who’s doing this to your site. Just change www.example.net to your domain, and combined_log to your combined log file.<\/p>\n
    awk -F\\” ‘($2 ~ \/\\.(jpg|gif)\/ && $4 !~ \/^http:\\\/\\\/www\\.example\\.net\/){print $4}’ combined_log \\
    \n| sort | uniq -c | sort
    \nTranslation:<\/p>\n
    explode each row using “;
    \nthe request line (%r) must contain “.jpg” or “.gif”;
    \nthe referer must not start with your website address (www.example.net in this example);
    \ndisplay the referer and summarise.
    \nYou can block hot-linking using mod_rewrite but that can also result in blocking various search engine result pages, caches and online translation software. To see if this is happening, we look for 403 (“Forbidden”) errors in the image requests:<\/p>\n
    # list image requests that returned 403 Forbidden
    \nawk ‘($9 ~ \/403\/)’ combined_log \\
    \n| awk -F\\” ‘($2 ~ \/\\.(jpg|gif)\/){print $4}’ \\
    \n| sort | uniq -c | sort
    \nTranslation:<\/p>\n
    the status code (%>s) is 403 Forbidden;
    \nthe request line (%r) contains “.jpg” or “.gif”;
    \ndisplay the referer and summarise.
    \nYou might notice that the above command is simply a combination of the previous, and one presented earlier. It is necessary to call awk more than once because the ‘referer’ field is only available after the separator is set to \\”, wheras the ‘status code’ is available directly.<\/p>\n
    6. Blank User Agents<\/p>\n
    A ‘blank’ user agent is typically an indication that the request is from an automated script or someone who really values their privacy. The following command will give you a list of ip addresses for those user agents so you can decide if any need to be blocked:<\/p>\n
    awk -F\\” ‘($6 ~ \/^-?$\/)’ combined_log | awk ‘{print $1}’ | sort | uniq
    \nA further pipe through logresolve will give you the hostnames of those addresses.<\/p>\n
     <\/p>\n
    View Apache requests per day<\/p>\n
    Run the following command to see requests per day:
    \nawk ‘{print $4}’ rmohan.com | cut -d: -f1 | uniq -c
    \nView Apache requests per hour<\/p>\n
    Run the following command to see requests per hour:
    \ngrep “23\/Jan” rmohan.com | cut -d[ -f2 | cut -d] -f1 | awk -F: ‘{print $2″:00″}’ | sort -n | uniq -c
    \nView Apache requests per minute<\/p>\n
    grep “23\/Jan\/2013:06″ rmohan.com | cut -d[ -f2 | cut -d] -f1 | awk -F: ‘{print $2″:”$3}’ | sort -nk1 -nk2 | uniq -c | awk ‘{ if ($1 > 10) print $0}’
    \n1- Most Common 404s (Page Not Found)
    \ncut -d'”‘ -f2,3 \/var\/log\/apache\/access.log | awk ‘$4=404{print $4” “$2}’ | sort | uniq -c | sort -rg<\/p>\n
    2 – Count requests by HTTP code<\/p>\n
    cut -d'”‘ -f3 \/var\/log\/apache\/access.log | cut -d’ ‘ -f2 | sort | uniq -c | sort -rg<\/p>\n
    3 – Largest Images
    \ncut -d'”‘ -f2,3 \/var\/log\/apache\/access.log | grep -E ‘\\.jpg|\\.png|\\.gif’ | awk ‘{print $5” “$2}’ | sort | uniq | sort -rg<\/p>\n
    4 – Filter Your IPs Requests
    \ntail -f \/var\/log\/apache\/access.log | grep <your IP><\/p>\n
    5 – Top Referring URLS
    \ncut -d'”‘ -f4 \/var\/log\/apache\/access.log | grep -v ‘^-$’ | grep -v ‘^http:\/\/www.rmohan.com’ | sort | uniq -c | sort -rg<\/p>\n
    6 – Watch Crawlers Live
    \nFor this we need an extra file which we’ll call bots.txt. Here’s the contents:<\/p>\n
    Bot
    \nCrawl
    \nai_archiver
    \nlibwww-perl
    \nspider
    \nMediapartners-Google
    \nslurp
    \nwget
    \nhttrack<\/p>\n
    This just helps is to filter out common user agents used by crawlers.
    \nHere’s the command:
    \ntail -f \/var\/log\/apache\/access.log | grep -f bots.txt<\/p>\n
    7 – Top Crawlers
    \nThis command will show you all the spiders that crawled your site with a count of the number of requests.
    \ncut -d'”‘ -f6 \/var\/log\/apache\/access.log | grep -f bots.txt | sort | uniq -c | sort -rg
    \nHow To Get A Top Ten
    \nYou can easily turn the commands above that aggregate (the ones using uniq) into a top ten by adding this to the end:
    \n| head<\/p>\n
    That is pipe the output to the head command.
    \nSimple as that.<\/p>\n
    Zipped Log Files
    \nIf you want to run the above commands on a logrotated file, you can adjust easily by starting with a zcat on the file then piping to the first command (the one with the filename).<\/p>\n
    So this:
    \ncut -d'”‘ -f3 \/var\/log\/apache\/access.log | cut -d’ ‘ -f2 | sort | uniq -c | sort -rg
    \nWould become this:
    \nzcat \/var\/log\/apache\/access.log.1.gz | cut -d'”‘ -f3 | cut -d’ ‘ -f2 | sort | uniq -c | sort -rg<\/p>\n
     <\/p>\n
    Analyse an Apache access log for the most common IP addresses
    \nTerminal – Analyse an Apache access log for the most common IP addresses
    \ntail -10000 access_log | awk ‘{print $1}’ | sort | uniq -c | sort -n | tail
    \nTerminal – Alternatives
    \nzcat access_log.*.gz | awk ‘{print $7}’ | sort | uniq -c | sort -n | tail -n 20
    \nawk ‘NR<=10000{a[$1]++}END{for (i in a) printf “%-6d %s\\n”,a[i], i|”sort -n”}’ access.log<\/p>\n
     <\/p>\n
     <\/p>\n
    Print lines within a particular time range<\/p>\n
    awk ‘\/01:05:\/,\/01:20:\/’ access.log
    \nSort access log by response size (increasing)<\/p>\n
    awk –re-interval ‘{ match($0, \/(([^[:space:]]+|\\[[^\\]]+\\]|”[^”]+”)[[:space:]]+){7}\/, m); print m[2], $0 }’ access.log|sort -nk 1<\/p>\n
    View TCP connection status
    \nnetstat-nat | awk ‘{print $ 6}’ | sort | uniq-c | sort-rn
    \nnetstat-n | awk ‘\/ ^ tcp \/ {+ + S [$ NF]}; END {for (a in S) print a, S [a]}’
    \nnetstat-n | awk ‘\/ ^ tcp \/ {+ + state [$ NF]}; END {for (key in state) print key, “\/ t”, state [key]}’
    \nnetstat-n | awk ‘\/ ^ tcp \/ {+ + arr [$ NF]}; END {for (k in arr) print k, “\/ t”, arr [k]}’
    \nnetstat-n | awk ‘\/ ^ tcp \/ {print $ NF}’ | sort | uniq-c | sort-rn
    \nnetstat-ant | awk ‘{print $ NF}’ | grep-v ‘[az]’ | sort | uniq-c
    \nnetstat-ant | awk ‘\/ ip: 80 \/ {split ($ 5, ip, “:”); + + S [ip [1]]} END {for (a in S) print S [a], a}’ | sort-n
    \nnetstat-ant | awk ‘\/: 80 \/ {split ($ 5, ip, “:”); + + S [ip [1]]} END {for (a in S) print S [a], a}’ | sort-rn | head-n 10
    \nawk ‘BEGIN {printf (“http_code \/ tcount_num \/ n”)} {COUNT [$ 10] + +} END {for (a in COUNT) printf a “\/ t \/ t” COUNT [a] “\/ n”}’
    \nFind the number of requests please 20 IP (commonly used to find the source of attack):
    \nnetstat-anlp | grep 80 | grep tcp | awk ‘{print $ 5}’ | awk-F: ‘{print $ 1}’ | sort | uniq-c | sort-nr | head-n20
    \nnetstat-ant | awk ‘\/: 80 \/ {split ($ 5, ip, “:”); + + A [ip [1]]} END {for (i in A) print A [i], i}’ | sort-rn | head-n20
    \nWith tcpdump the sniffer port 80 access to see who the highest
    \ntcpdump-i eth0-tnn dst port 80-c 1000 | awk-F “.” ‘{print $ 1 “.” $ 2 “.” $ 3 “.” $ 4}’ | sort | uniq-c | sort-nr | head – 20
    \n4. Find more time_wait to connect
    \nnetstat-n | grep TIME_WAIT | awk ‘{print $ 5}’ | sort | uniq-c | sort-rn | head-n20
    \nFind check more SYN connection.
    \nnetstat-an | grep SYN | awk ‘{print $ 5}’ | awk-F: ‘{print $ 1}’ | sort | uniq-c | sort-nr | more
    \n6 according to the port column process
    \nnetstat-ntlp | grep 80 | awk ‘{print $ 7}’ | cut-d \/-f1
    \nWeb logs (Apache):
    \n1 ip address to gain access to the top 10
    \ncat access.log | awk ‘{print $ 1}’ | sort | uniq-c | sort-nr | head -10
    \ncat access.log | awk ‘{counts [$ (11)] + = 1}; END {for (url in counts) print counts [url], url}’
    \n2 the Most Visited file or page, take the top 20 and statistics all IP
    \ncat access.log | awk ‘{print $ 11}’ | sort | uniq-c | sort-nr | head -20
    \nawk ‘{print $ 1}’ access.log | sort-n-r | uniq-c | wc-l
    \n3 List the transmission exe file (analysis commonly used when the download station)
    \ncat access.log | awk ‘($ 7 ~ \/ \/. exe \/) {print $ 10 “” $ 1 “” $ 4 “” $ 7}’ | sort-nr | head -20
    \nExe file and the corresponding file occurrences List output is greater than 200000byte (about 200kb)
    \ncat access.log | awk ‘($ 10> 200000 && $ 7 ~ \/ \/. exe \/) {print $ 7}’ | sort-n | uniq-c | sort-nr | head -100
    \nIf the log of a page file transfer time, the page lists the most time-consuming to client
    \ncat access.log | awk ‘($ 7 ~ \/ \/. php \/) {print $ NF “” $ 1 “” $ 4 “” $ 7}’ | sort-nr | head -100
    \n6 page lists the most time-consuming (more than 60 seconds) as well as the frequency and the corresponding page
    \ncat access.log | awk ‘($ NF> 60 && $ 7 ~ \/ \/. php \/) {print $ 7}’ | sort-n | uniq-c | sort-nr | head -100
    \nList file transmission time over 30 seconds
    \ncat access.log | awk ‘($ NF> 30) {print $ 7}’ | sort-n | uniq-c | sort-nr | head -20
    \nStatistics website traffic (G)
    \ncat access.log | awk ‘{sum + = $ 10} END {print sum\/1024\/1024\/1024}’
    \nStatistics 404 connections 9.
    \nawk ‘($ 9 ~ \/ 404 \/)’ access.log | awk ‘{print $ 9, $ 7}’ | sort
    \n10 Statistics http status.
    \ncat access.log | awk ‘{counts [$ (9)] + = 1}; END {for (code in counts) print code, counts [code]}’
    \ncat access.log | awk ‘{print $ 9}’ | sort | uniq-c | sort-rn
    \n11 concurrent per second:
    \nawk ‘{if ($ 9 ~ \/ 200 | 30 | 404 \/) COUNT [$ 4] + +} END {for (a in COUNT) print a, COUNT [a]}’ | sort-k 2-nr | head-n10
    \n12. Bandwidth statistics
    \ncat apache.log | awk ‘{if ($ 7 ~ \/ GET \/) count + +} END {print “client_request =” count}’
    \ncat apache.log | awk ‘{BYTE + = $ 11} END {print “client_kbyte_out =” BYTE\/1024 “KB”}’
    \nAverage size of 13. Statistical number of objects and object
    \ncat access.log | awk ‘{byte + = $ 10} END {print byte\/NR\/1024, NR}’
    \ncat access.log | awk ‘{if ($ 9 ~ \/ 200 | 30 \/) COUNT [$ NF] + +} END {for (a in COUNT) print a, COUNT
    \n[A], NR, COUNT [a] \/ NR * 100 “%”}
    \n14 to take a 5-minute log
    \nif [$ DATE_MINUTE! = $ DATE_END_MINUTE]; then # determine the start timestamp and end timestamps are equal START_LINE = `sed-n” \/ $ DATE_MINUTE \/ = “$ APACHE_LOG | head-n1` # if not equal, then remove the the line numbers start timestamp and the end timestamp line number
    \n# END_LINE = `sed-n” \/ $ DATE_END_MINUTE \/ = “$ APACHE_LOG | tail-n1`
    \nEND_LINE = `sed-n” \/ $ DATE_END_MINUTE \/ = “$ APACHE_LOG | head-n1` sed-n “$ {START_LINE}, $ {END_LINE} p” $ APACHE_LOG> $ MINUTE_LOG # # by line number, remove the 5 minutes the contents of the log is stored into a temporary file
    \nGET_START_TIME = `sed-n” $ {START_LINE} p “$ APACHE_LOG | awk-F ‘[‘ ‘{print $ 2}’ | awk ‘{print $ 1}’ |
    \nsed ‘s # \/ # # g’ | sed ‘s #: # #’ `# remove timestamp obtained by the line number
    \nGET_END_TIME = `sed-n” $ {END_LINE} p “$ APACHE_LOG | awk-F ‘[‘ ‘{print $ 2}’ | awk ‘{print $ 1}’ | sed
    \n‘S # \/ # # g’ | sed ‘s #: # #’ `# line number to get the end timestamp
    \n15 Spiders analysis
    \nSee which spiders crawl the content
    \n\/ Usr \/ sbin \/ tcpdump-i eth0-l-s 0-w – dst port 80 | strings | grep-i user-agent | grep-i-E ‘bot | crawler | slurp | spider’
    \nSite on the analysis 2 (Squid papers)
    \nA flow rate of 2 domain statistics
    \nzcat squid_access.log.tar.gz | awk ‘{print $ 10, $ 7}’ | awk ‘BEGIN {FS = “[\/]”} {trfc [$ 4] + = $ 1} END {for
    \n(Domain in trfc) {printf “% s \/ t% d \/ n”, domain, trfc [domain]}} ‘
    \nEfficient perl version, please download here:
    \nDatabase articles
    \n1 View sql database
    \n\/ Usr \/ sbin \/ tcpdump-i eth0-s 0-l-w – dst port 3306 | strings | egrep-i ‘SELECT | UPDATE | DELETE | INSERT | SET | COMMIT | ROLLBACK | CREATE | DROP | ALTER | CALL’
    \nSystem Debug analyze articles
    \n1 debug command
    \nstrace-p pid
    \nTracking the specified process PID
    \ngdb-p PID<\/p>\n
    CHECKING FOR HIGH VISITS FROM A LIMITED NUMBER OF IPS<\/p>\n
    First locate the log file for your site. The generic log is generally at \/var\/log\/httpd\/access_log or\/var\/log\/apache2\/access_log (depending on your distro). For virtualhost-specific logs, check the conf files or (if you have one active site and others in the background) run ls -alt \/var\/log\/httpd to see which file is most recently updated.<\/p>\n
    cat access.log | awk ‘{print $1}’ | sort | uniq -c | wc -l<\/p>\n
    2. Check out unique visitors today:-
    \ncat access.log | grep `date ‘+%e\/%b\/%G’` | awk ‘{print $1}’ | sort | uniq -c | wc -l<\/p>\n
    3. Check out unique visitors this month:-
    \ncat access.log | grep `date ‘+%b\/%G’` | awk ‘{print $1}’ | sort | uniq -c | wc -l<\/p>\n
    4. Check out unique visitors on arbitrary nomber:-
    \ncat access.log | grep 22\/Mar\/2013 | awk ‘{print $1}’ | sort | uniq -c | wc -l<\/p>\n
    5. Check out unique visitors Month of march:-
    \ncat access.log | grep Mar\/2013 | awk ‘{print $1}’ | sort | uniq -c | wc -l<\/p>\n
    6. Check out statistics of number of visit\/request “visitors IP”-
    \ncat access.log | awk ‘{print “requests from ” $1}’ | sort | uniq -c | sort<\/p>\n
    7. Check out statistics of number of visit\/request with date-
    \ncat access.log | grep 26\/Mar\/2013 | awk ‘{print “requests from ” $1}’ | sort | uniq -c | sort<\/p>\n
    8. Find out  targets the last 5,000 hits:<\/p>\n
    tail -5000 access.log| awk ‘{print $1}’ | sort | uniq -c |sort -n<\/p>\n
    9. Finally, if you have a ton of domains you may want to use this to aggregate them:<\/p>\n
    for k in `ls –color=none`; do echo “Top visitors by ip for: $k”;awk ‘{print $1}’ ~\/logs\/$k\/http\/access.log|sort|uniq -c|sort -n|tail;done<\/p>\n
    10. This command is great if you want to see what is being called the most (that can often show you that a specific script is being abused if it’s being called way more times than anything else in the site):<\/p>\n
    awk ‘{print $7}’ access.log|cut -d? -f1|sort|uniq -c|sort -nk1|tail -n10<\/p>\n
    11. If you have multiple domains on and on a PS (PS only!) run this command to get all traffic for all domains on the PS:<\/p>\n
    for k in `ls -S \/home\/*\/logs\/*\/http\/access.log`; do wc -l $k | sort -r -n; done<\/p>\n
    12. Here is an alternative to the above command which does the same thing, this is for VPS only using an admin user:
    \n sudo find \/home\/*\/logs -type f -name “access.log” -exec wc -l “{}” \\; | sort -r -n<\/p>\n
    13. If you’re on a shared server you can run this command which will do the same as the one above but just to the domains in your logs directory. You have to run this commands while your in your user’s logs directory:
    \nfor k in `ls -S *\/http\/access.log`; do wc -l $k | sort -r -n; done<\/p>\n
    14. grep apache access.log and list IP’s by hits and date:-
    \ngrep Mar\/2013 \/var\/log\/apache2\/access.log | awk ‘{ print $1 }’ | sort -n | uniq -c | sort -rn | head<\/p>\n
    15. Find out top referring URLs: <\/p>\n
     cut -d'”‘ -f4 \/var\/log\/apache\/access.log | grep -v ‘^-$’ | grep -v ‘^http:\/\/www.your-site.com’ | sort | uniq -c | sort -rg<\/p>\n
    16. Check out top of ‘Page Not Found’s (404):
    \ncut -d'”‘ -f2,3 \/var\/log\/apache\/access.log | awk ‘$4=404{print $4″ “$2}’ | sort | uniq -c | sort -rg<\/p>\n
    17. Check out top largest images:-
    \ncut -d'”‘ -f2,3 \/var\/log\/apache\/access.log | grep -E ‘\\.jpg|\\.png|\\.gif’ | awk ‘{print $5” “$2}’ | sort | uniq | sort -rg
    \n18. Check out server response code:-
    \ncut -d'”‘ -f3 \/var\/log\/apache\/access.log | cut -d’ ‘ -f2 | sort | uniq -c | sort -rg
    \n19. Check out Apache request per day
    \nawk ‘{print $4}’ access.log | cut -d: -f1 | uniq -c
    \n20. Check out Apache request per houre.
    \ngrep “6\/May” access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: ‘{print $2″:00″}’ | sort -n | uniq -c
    \n21. Check out Apache request per minute.
    \ngrep “6\/May\/2013:06″ access.log | cut -d[ -f2 | cut -d] -f1 | awk -F: ‘{print $2”:”$3}’ | sort -nk1 -nk2 | uniq -c | awk ‘{ if ($1 > 10) print $0}’
    \n22. All those commands can be easily run on a log-rotated file:
    \nzcat \/var\/log\/apache\/access.log.1.gz | cut -d'”‘ -f3 | cut -d’ ‘ -f2 | sort | uniq -c | sort -rg<\/p>\n
    Example:
    \nGrep log log analysis collate finishing
    \n1 . Analyze log files to access the page next 2012-05-04 The top 20 URL and sorting
    \ncat access.log | grep ’04 \/ May\/2012 ‘| awk’ {print $ 11} ‘| sort | uniq-c | sort-nr | head -20
    \nQuery the URL address to access the page URL contains the IP address of www.abc.com
    \ncat access_log | awk ‘($ 11 ~ \/ \\ www.abc.com\/) {print $ 1}’ | sort | uniq-c | sort-nr
    \n(2) to gain access to up to 10 IP addresses can also be queried by time
    \ncat linewow-access.log | awk ‘{print $ 1}’ | sort | uniq-c | sort-nr | head -10
    \n1 to gain access to the ip address before 10
    \ncat access.log | awk ‘{print $ 1}’ | sort | uniq-c | sort-nr | head -10
    \ncat access.log | awk ‘{counts [$ (11)] + = 1}; END {for (url in counts) print counts [url], url}’
    \n2 Most Visited file or page , take the top 20 and all access to IP Statistics
    \ncat access.log | awk ‘{print $ 11}’ | sort | uniq-c | sort-nr | head -20
    \nawk ‘{print $ 1}’ access.log | sort-n-r | uniq-c | wc-l
    \ncat wangsu.log | egrep ’06 \/ Sep\/2012: 14:35 | 06\/Sep\/2012: 15:05 ‘| awk’ {print $ 1} ‘| sort | uniq-c | sort-nr | head -10 query log period of time the situation
    \n3 lists some of the largest transfer exe file ( download station when analyzing common )
    \ncat access.log | awk ‘($ 7 ~ \/ \\. exe \/) {print $ 10 “” $ 1 “” $ 4 “” $ 7}’ | sort-nr | head -20
    \n4 lists the output is greater than 200000byte ( about 200kb) an exe file and the number of occurrences of the corresponding file
    \ncat access.log | awk ‘($ 10> 200000 && $ 7 ~ \/ \\. exe \/) {print $ 7}’ | sort-n | uniq-c | sort-nr | head -100
    \n5 If the log records the last one is the page file transfer time , there are lists to the client the most time-consuming page
    \ncat access.log | awk ‘($ 7 ~ \/ \\. php \/) {print $ NF “” $ 1 “” $ 4 “” $ 7}’ | sort-nr | head -100
    \n6 lists the most time-consuming page ( more than 60 seconds ) as well as the corresponding page number of occurrences
    \ncat access.log | awk ‘($ NF> 60 && $ 7 ~ \/ \\. php \/) {print $ 7}’ | sort-n | uniq-c | sort-nr | head -100
    \n7 lists the transmission of documents longer than 30 seconds
    \ncat access.log | awk ‘($ NF> 30) {print $ 7}’ | sort-n | uniq-c | sort-nr | head -20
    \n8 Statistics website traffic (G)
    \ncat access.log | awk ‘{sum + = $ 10} END {print sum\/1024\/1024\/1024}’
    \n9 Statistics 404 connection
    \nawk ‘($ 9 ~ \/ 404 \/)’ access.log | awk ‘{print $ 9, $ 7}’ | sort
    \n10 Statistical http status.
    \ncat access.log | awk ‘{counts [$ (9)] + = 1}; END {for (code in counts) print code, counts [code]}’
    \ncat access.log | awk ‘{print $ 9}’ | sort | uniq-c | sort-rn
    \n11 sec Concurrency :
    \nawk ‘{if ($ 9 ~ \/ 200 | 30 | 404 \/) COUNT [$ 4] + +} END {for (a in COUNT) print a, COUNT [a]}’ | sort-k 2-nr | head-n10
    \n12 . Bandwidth statistics
    \ncat apache.log | awk ‘{if ($ 7 ~ \/ GET \/) count + +} END {print “client_request =” count}’
    \ncat apache.log | awk ‘{BYTE + = $ 11} END {print “client_kbyte_out =” BYTE\/1024 “KB”}’
    \nOne day out of the 10 most visited IP
    \ncat \/ tmp \/ access.log | grep “20\/Mar\/2011” | awk ‘{print $ 3}’ | sort | uniq-c | sort-nr | head
    \nMaximum number of connections that day ip ip are doing :
    \ncat access.log | grep “10.0.21.17” | awk ‘{print $ 8}’ | sort | uniq-c | sort-nr | head-n 10
    \nFind out the most visited several minutes
    \nawk ‘{print $ 1}’ access.log | grep “20\/Mar\/2011” | cut-c 14-18 | sort | uniq-c | sort-nr | head
    \nAttachment: View tcp connection status
    \nnetstat-nat | awk ‘{print $ 6}’ | sort | uniq-c | sort-rn
    \nnetstat-n | awk ‘\/ ^ tcp \/ {+ + S [$ NF]}; END {for (a in S) print a, S [a]}’
    \nnetstat-n | awk ‘\/ ^ tcp \/ {+ + state [$ NF]}; END {for (key in state) print key, “\\ t”, state [key]}’
    \nnetstat-n | awk ‘\/ ^ tcp \/ {+ + arr [$ NF]}; END {for (k in arr) print k, “\\ t”, arr [k]}’
    \nnetstat-n | awk ‘\/ ^ tcp \/ {print $ NF}’ | sort | uniq-c | sort-rn
    \nnetstat-ant | awk ‘{print $ NF}’ | grep-v ‘[az]’ | sort | uniq-c
    \nnetstat-ant | awk ‘\/ ip: 80 \/ {split ($ 5, ip, “:”); + + S [ip [1]]} END {for (a in S) print S [a], a}’ | sort-n
    \nnetstat-ant | awk ‘\/: 80 \/ {split ($ 5, ip, “:”); + + S [ip [1]]} END {for (a in S) print S [a], a}’ | sort-rn | head-n 10
    \nawk ‘BEGIN {printf (“http_code \\ tcount_num \\ n”)} {COUNT [$ 10] + +} END {for (a in COUNT) printf a “\\ t \\ t” COUNT [a] “\\ n”}’
    \n(2) Find requests please 20 IP ( commonly used in the attack source lookup ) :
    \nnetstat-anlp | grep 80 | grep tcp | awk ‘{print $ 5}’ | awk-F: ‘{print $ 1}’ | sort | uniq-c | sort-nr | head-n20
    \nnetstat-ant | awk ‘\/: 80 \/ {split ($ 5, ip, “:”); + + A [ip [1]]} END {for (i in A) print A [i], i}’ | sort-rn | head-n20
    \n3 with a sniffer tcpdump port 80 access to see who the highest
    \ntcpdump-i eth0-tnn dst port 80-c 1000 | awk-F “.” ‘{print $ 1 “.” $ 2 “.” $ 3 “.” $ 4}’ | sort | uniq-c | sort-nr | head – 20
    \n4 Find more time_wait connection
    \nnetstat-n | grep TIME_WAIT | awk ‘{print $ 5}’ | sort | uniq-c | sort-rn | head-n20
    \n5 more investigation to find SYN connections
    \nnetstat-an | grep SYN | awk ‘{print $ 5}’ | awk-F: ‘{print $ 1}’ | sort | uniq-c | sort-nr | more
    \n6 According to port out process
    \nnetstat-ntlp | grep 80 | awk ‘{print $ 7}’ | cut-d \/-f1<\/p>\n","protected":false},"excerpt":{"rendered":"
    Since I (and you as a visitor) don\u2019t want your IP-address to be spread around the internet, I\u2019ve anonymized the log data. It\u2019s a fairly easy process that is done in 2 steps: IP\u2019s are translated into random values. Admin url\u2019s are removed. Step 1: Translating IP\u2019s <\/p>\n
    All the IP\u2019s are translated into random IP\u2019s, […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/2786"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2786"}],"version-history":[{"count":7,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/2786\/revisions"}],"predecessor-version":[{"id":3379,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/2786\/revisions\/3379"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2786"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2786"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2786"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}