Sar… Networking

echo "Rcvd Bytes - Time";\
(unset LANG; sar -n DEV) | grep eth0 | awk '{print ($5" - "$1)}' | sort -nr | head -n5;\
echo "Average:    `sar -n DEV |grep eth0| tail -n1 | awk '{print ($5)}'`"

Example:

$ echo "Rcvd Bytes - Time";\
(unset LANG; sar -n DEV) | grep eth0 | awk '{print ($5" - "$1)}' | sort -nr | head -n5;\
echo "Average:    `sar -n DEV |grep eth0| tail -n1 | awk '{print ($5)}'`"
Rcvd Bytes - Time
1.72 - 09:00:01
1.58 - 01:30:01
1.26 - 01:20:01
1.10 - 01:40:01
1.09 - 03:30:01
Average:    0.83 

xample:

 echo "Rcvd Bytes - Time";\
(unset LANG; sar -n DEV) | grep eth0 | awk '{print ($5" - "$1)}' | sort -nr | head -n5;\
echo "Average:    `sar -n DEV |grep eth0| tail -n1 | awk '{print ($5)}'`"
Rcvd Bytes - Time
1.72 - 09:00:01
1.58 - 01:30:01
1.26 - 01:20:01
1.10 - 01:40:01
1.09 - 03:30:01
Average:    0.83

Also just the usual ‘top n tailed’ format to show the last 20 records for ‘received bytes’:

(unset LANG; sar -n DEV) | grep tx |head -n1| awk '{print ("          "$6)}';\
(unset LANG; sar -n DEV) | grep eth0 | tail -20 | awk '{print ($1,$2" - "$6)}';

Example:

  (unset LANG; sar -n DEV) | grep tx |head -n1| awk '{print ("          "$6)}';\
(unset LANG; sar -n DEV) | grep eth0 | tail -20 | awk '{print ($1,$2" - "$6)}';
txkB/s
11:50:01 eth0 - 1.69
12:00:01 eth0 - 0.40
12:10:01 eth0 - 0.01
12:20:01 eth0 - 0.01
12:30:01 eth0 - 0.01
12:40:01 eth0 - 0.01
12:50:01 eth0 - 0.01
13:00:01 eth0 - 0.01
13:10:01 eth0 - 0.01
13:20:01 eth0 - 0.01
13:30:01 eth0 - 0.01
13:40:01 eth0 - 0.01
13:50:01 eth0 - 0.25
14:00:01 eth0 - 0.10
14:10:01 eth0 - 0.02
14:20:01 eth0 - 0.01
14:30:01 eth0 - 0.03
14:40:01 eth0 - 0.86
14:50:01 eth0 - 0.22
Average: eth0 - 1.56

You’ll notice that I use ‘unset LANG’ in a lot of the examples. This removes AM/PM from the output and tailors the results instead in 24hr clock format. It makes the output a lot more predictable when on different machines!

Sar… Memory

o one of the big problems with Sar is that it’s pretty ‘dumb’ with its ‘memory free’ reporting. Although the information from Sar is great, a colleague of mine gave me the basis for a few more one liners that will perform a calculation to ensure that buffers/cache are taken into account as their usage is relinquished under system load.

Again, all of my commands are pretty hacky, and probably not examples of good scripting – but they get the job done!

Show the ‘real’ memory usage, the last column gives the true value:

(unset LANG ;sar  -r) | awk ‘$3~/[0-9]/{total=$3+$2; usedbc=$3-($5+$6);\
pc_used=(100*usedbc)/total;print $0,pc_used} $3!~/[0-9]/{print $0}’

Example:

(unset LANG ;sar  -r) | awk ‘$3~/[0-9]/{total=$3+$2; usedbc=$3-($5+$6);\
pc_used=(100*usedbc)/total;print $0,pc_used} $3!~/[0-9]/{print $0}’

Linux 2.6.xx (influencd.co.uk)     05/31/10
00:00:01    kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
00:10:01       109904    295708     72.90     43432     73392    764192     22232      2.83      3592 44.1022
00:20:01       107480    298132     73.50     43556     73404    764192     22232      2.83      3592 44.6663
00:30:01       107328    298284     73.54     43668     73404    764192     22232      2.83      3592 44.6762

Showing the ‘real’ memory usage in a presentable format, top and tailing the results:

(unset LANG ;sar  -r) | awk ‘$3~/[0-9]/{total=$3+$2; usedbc=$3-($5+$6); \
pc_used=(100*usedbc)/total;print $0,pc_used} $3!~/[0-9]/{print $0}’ | awk ‘{print ($1″    –  “$4” – “$11)}’| head -1;\
echo “Time     – Real memory used %”; (unset LANG ;sar  -r) | awk ‘$3~/[0-9]/{total=$3+$2; usedbc=$3-($5+$6);\
pc_used=(100*usedbc)/total;print $0,pc_used} $3!~/[0-9]/{print $0}’ | awk ‘{print ($1″ – “$11)}’| tail -10

Example:

(unset LANG ;sar  -r) | awk ‘$3~/[0-9]/{total=$3+$2; usedbc=$3-($5+$6); \
pc_used=(100*usedbc)/total;print $0,pc_used} $3!~/[0-9]/{print $0}’ | awk ‘{print ($1”    –  “$4” – “$11)}’| head -1;\
echo “Time     – Real memory used %”; (unset LANG ;sar  -r) | awk ‘$3~/[0-9]/{total=$3+$2; usedbc=$3-($5+$6);\
pc_used=(100*usedbc)/total;print $0,pc_used} $3!~/[0-9]/{print $0}’ | awk ‘{print ($1″ – “$11)}’| tail -10

Linux    –  05/31/10 –
Time     – Real memory used %
19:50:01 – 86.6651
20:00:01 – 87.8406
20:10:01 – 88.1177
20:20:01 – 87.031
20:30:01 – 87.315
20:40:01 – 89.1699
20:50:01 – 89.4323
21:00:01 – 89.0723
21:10:01 – 90.3524
Average: – 45.4244

The below will give an hourly snapshot of the memory usage for every day that there is a sar log for, in reverse date order:

for i in $(ls /var/log/sa/ | grep -v sar);\
do (unset LANG ;sar  -f /var/log/sa/$i -r) | awk ‘$3~/[0-9]/{total=$3+$2;\
usedbc=$3-($5+$6); pc_used=(100*usedbc)/total;print $0,pc_used} $3!~/[0-9]/{print $0}’\
| awk ‘{print ($1” – “$4” – “$11)}’| head -1 | grep -v “memused”; echo “Time     – Real memory used %”;\
(unset LANG ;sar  -r  -f /var/log/sa/$i) | awk ‘$3~/[0-9]/{total=$3+$2; usedbc=$3-($5+$6);pc_used=(100*usedbc)/total;\
print $0,pc_used} $3!~/[0-9]/{print $0}’ | awk ‘{print ($1″ – “$11)}’| grep “00:01” | grep -v “memused”; echo; done

Example:

[root@influencd ~]$  for i in $(ls -t /var/log/sa/ | grep -v sar);\
do (unset LANG ;sar  -f /var/log/sa/$i -r) | awk ‘$3~/[0-9]/{total=$3+$2;\
usedbc=$3-($5+$6); pc_used=(100*usedbc)/total;print $0,pc_used} $3!~/[0-9]/{print $0}’\
| awk ‘{print ($1″ – “$4” – “$11)}’| head -1 | grep -v “memused”; echo “Time     – Real memory used %”;\
(unset LANG ;sar  -r  -f /var/log/sa/$i) | awk ‘$3~/[0-9]/{total=$3+$2; usedbc=$3-($5+$6);pc_used=(100*usedbc)/total;\
print $0,pc_used} $3!~/[0-9]/{print $0}’ | awk ‘{print ($1″ – “$11)}’| grep “00:01” | grep -v “memused”; echo; done

Linux – 05/24/10 –
Time     – Real memory used %
00:00:01 –
01:00:01 – 89.3317
02:00:01 – 88.9066
03:00:01 – 88.5102
[snip]
Linux – 05/23/10 –
Time     – Real memory used %
00:00:01 –
01:00:01 – 49.6731
02:00:01 – 49.1317
[etc etc]

The below uses the day’s average from the sar information, and presents it for each day that there’s a log:

echo “Real memory used %”;\
for i in $(ls /var/log/sa/ | grep -v sar); do (unset LANG ;sar  -f /var/log/sa/$i -r) | awk ‘$3~/[0-9]/{total=$3+$2; \
usedbc=$3-($5+$6); pc_used=(100*usedbc)/total;print $0,pc_used} $3!~/[0-9]/{print $0}’ | awk ‘{print ($4)}’\
| head -1 | grep -v “memused” ;\
(unset LANG ;sar  -r  -f /var/log/sa/$i) | awk ‘$3~/[0-9]/{total=$3+$2; usedbc=$3-($5+$6);\
pc_used=(100*usedbc)/total;print $0,pc_used} $3!~/[0-9]/{print $0}’ | awk ‘{print (”           “$11)}’| tail -1;done

Example:

echo “Real memory used %”;\
for i in $(ls /var/log/sa/ | grep -v sar); do (unset LANG ;sar  -f /var/log/sa/$i -r) | awk ‘$3~/[0-9]/{total=$3+$2; \
usedbc=$3-($5+$6); pc_used=(100*usedbc)/total;print $0,pc_used} $3!~/[0-9]/{print $0}’ | awk ‘{print ($4)}’\
| head -1 | grep -v “memused” ;\
(unset LANG ;sar  -r  -f /var/log/sa/$i) | awk ‘$3~/[0-9]/{total=$3+$2; usedbc=$3-($5+$6);\
pc_used=(100*usedbc)/total;print $0,pc_used} $3!~/[0-9]/{print $0}’ | awk ‘{print (”           “$11)}’| tail -1;done

Real memory used %
05/23/10
49.2444
05/24/10
40.8514
05/25/10
43.1346
05/26/10
50.288
05/27/10
60.299
05/28/10
65.9872
05/29/10
70.6873
05/30/10
40.3156
05/31/10
45.7713

In contrast if we compare the ‘dumb’ memory usage to the example above, you can see some pretty major differences in terms of the value:

echo ”     Memory used %”; for i in $(ls /var/log/sa/ | grep -v sar); do echo $i ;\
sar  -r -f /var/log/sa/$i | tail -1 | awk ‘{print ($1″ – “$4)}’; done

Example:

echo ”     Memory used %”; for i in $(ls /var/log/sa/ | grep -v sar); do echo $i ;\
sar  -r -f /var/log/sa/$i | tail -1 | awk ‘{print ($1″ – “$4)}’; done

Memory used %
sa23
Average: – 70.35
sa24
Average: – 78.06
sa25
Average: – 94.58
sa26
Average: – 92.61
sa27
Average: – 89.86
sa28
Average: – 93.69
sa29
Average: – 97.31
sa30
Average: – 65.48
sa31
Average: – 75.65

 Sar… CPU

 

Cpu information:

Display ‘Processor Queue Length’ information from the last 15 intervals in a presentable format:

sar -q | head -3; sar -q | tail -n15

Example:

sar -q | head -3; sar -q | tail -n15
Linux 2.6.xx (influencd.co.uk)     05/31/2010
12:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
04:30:01 PM         0       104      0.02      0.01      0.00
04:40:01 PM         0       105      0.00      0.00      0.00
04:50:01 PM         0        97      0.00      0.00      0.00
05:00:01 PM         0        97      0.00      0.00      0.00
05:10:01 PM         0        97      0.00      0.00      0.00
05:20:01 PM         0        97      0.00      0.00      0.00
05:30:01 PM         0        97      0.00      0.00      0.00
05:40:01 PM         0        97      0.00      0.00      0.00
05:50:01 PM         0        97      0.00      0.00      0.00
06:00:01 PM         0        97      0.00      0.00      0.00
06:10:01 PM         0        98      0.07      0.02      0.00
06:20:01 PM         0       109      0.00      0.00      0.00
06:30:01 PM         0       106      0.13      0.03      0.01
06:40:01 PM         0       110      0.07      0.02      0.00
Average:            0        98      0.00      0.00      0.00

Display general CPU stats from the last 15 intervals in a presentable format:

sar | head -3; sar | tail -n15

Example:

sar | head -3; sar | tail -n15
Linux 2.6.xx (influencd.co.uk)     05/31/2010

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
04:30:01 PM     all      0.00      0.00      0.00      0.01      0.00     99.98
04:40:01 PM     all      0.01      0.00      0.00      0.03      0.00     99.95
04:50:01 PM     all      0.02      0.00      0.01      0.02      0.00     99.95
05:00:01 PM     all      0.00      0.00      0.00      0.00      0.00     99.98
05:10:01 PM     all      0.00      0.00      0.00      0.01      0.00     99.98
05:20:01 PM     all      0.01      0.00      0.00      0.00      0.00     99.98
05:30:01 PM     all      0.00      0.00      0.00      0.01      0.00     99.99
05:40:01 PM     all      0.00      0.00      0.00      0.01      0.00     99.99
05:50:01 PM     all      0.00      0.00      0.00      0.00      0.00     99.99
06:00:01 PM     all      0.00      0.00      0.00      0.01      0.00     99.98
06:10:01 PM     all      0.03      0.00      0.01      0.02      0.00     99.94
06:20:01 PM     all      0.17      0.00      0.02      0.21      0.01     99.59
06:30:01 PM     all      1.09      0.00      0.06      0.34      0.03     98.48
06:40:01 PM     all      0.35      0.00      0.02      0.15      0.01     99.47
Average:        all      0.03      0.00      0.01      0.05      0.00     99.92

Presentable CPU idle percentage, for the last 15 intervals:

unset LANG;date;echo “CPU idle  %”; sar | tail -20| awk ‘{print ($1″ “$2” – “$8)}’

Example:

unset LANG;date;echo “CPU idle  %”; sar | tail -20| awk ‘{print ($1″ “$2” – “$8)}’
Mon May 31 19:09:53 GMT 2010
CPU idle  %
16:00:01 all – 99.49
16:10:01 all – 99.97
16:20:02 all – 99.98
16:30:01 all – 99.98
16:40:01 all – 99.95
16:50:01 all – 99.95
17:00:01 all – 99.98
17:10:01 all – 99.98
17:20:01 all – 99.98
17:30:01 all – 99.99
17:40:01 all – 99.99
17:50:01 all – 99.99
18:00:01 all – 99.98
18:10:01 all – 99.94
18:20:01 all – 99.59
18:30:01 all – 98.48
18:40:01 all – 99.47
18:50:01 all – 99.80
19:00:01 all – 99.86
Average: all – 99.92

CPU Idle average for each day available in the Sar logs:

echo “Average CPU idle  %”; for i in $(ls /var/log/sa/ | grep -v sar); \
do ll /var/log/sa/$i | awk ‘{print ($6,$7)}’ ;\
sar -f /var/log/sa/$i | tail -1 | awk ‘{print (”       “$7)}’; done

Example:

echo “Average CPU idle  %”; for i in $(ls /var/log/sa/ | grep -v sar); \
do ll /var/log/sa/$i | awk ‘{print ($6,$7)}’ ;\
sar -f /var/log/sa/$i | tail -1 | awk ‘{print (”       “$7)}’; done
Average CPU idle  %
2010-05-23 23:50
0.00
2010-05-24 23:50
0.00
2010-05-25 23:50
0.00
2010-05-26 23:50
0.00
2010-05-27 23:50
0.00
2010-05-28 23:50
0.01
2010-05-29 23:50
0.00
2010-05-30 23:50
0.00
2010-05-31 19:00
0.00

Average processor queue length for every day available in the sar logs:

echo “Average Processor Queue length”; for i in $(ls /var/log/sa/ | grep -v sar);\
do ll /var/log/sa/$i | awk ‘{print ($6,$7)}’ ; \
sar -q -f /var/log/sa/$i | tail -1 | awk ‘{print (”       “$6)}’; done

Example:
echo “Average Processor Queue length”; for i in $(ls /var/log/sa/ | grep -v sar);\
do ll /var/log/sa/$i | awk ‘{print ($6,$7)}’ ; \
sar -q -f /var/log/sa/$i | tail -1 | awk ‘{print (”       “$6)}’; done
Average Processor Queue length
2010-05-23 23:50
0.00
2010-05-24 23:50
0.00
2010-05-25 23:50
0.00
2010-05-26 23:50
0.00
2010-05-27 23:50
0.00
2010-05-28 23:50
0.00
2010-05-29 23:50
0.00
2010-05-30 23:50
0.00
2010-05-31 20:10
0.00

Show CPU processor queue length use by hourly snapshot, for every day in the last month:

date;echo;echo “Processor Queue Length (15m)”;echo; for i in $(ls /var/log/sa/ | grep -v sar);\
do ll /var/log/sa/$i | awk ‘{print ($6,$7)}’;\
sar -q -f /var/log/sa/$i | grep “:00:” | awk ‘{print ($1,$2″ – “$7)}’; echo; done

Example:

[root@influencd ~]# date;echo;echo “Processor Queue Length (15m)”;echo; for i in $(ls /var/log/sa/ | grep -v sar);\
do ll /var/log/sa/$i | awk ‘{print ($6,$7)}’;\
sar -q -f /var/log/sa/$i | grep “:00:” | awk ‘{print ($1,$2″ – “$7)}’; echo; done
Mon May 31 19:02:30 GMT 2010
Processor Queue Length (15m)

2010-05-23 23:5012:00:01 AM – ldavg-15
01:00:01 AM – 0.00
02:00:01 AM – 0.00
03:00:01 AM – 0.00
[Snip]
2010-05-24 23:50
12:00:01 AM – ldavg-15
01:00:01 AM – 0.00
02:00:01 AM – 0.00
03:00:01 AM – 0.00
[etc etc]

Hourly Processor queue length with 15/m averages:

sar -q | grep 00:01| awk ‘{print ($1” “$2” – “$7)}’|head -3;\
sar -q |grep 00:01| awk ‘{print ($1,$2” – “$7)}’| tail -n15

Example:

sar -q | grep 00:01| awk ‘{print ($1” “$2” – “$7)}’|head -3;\
sar -q |grep 00:01| awk ‘{print ($1,$2” – “$7)}’| tail -n15
12:00:01 AM – ldavg-15
01:00:01 AM – 0.00
02:00:01 AM – 0.00
04:00:01 AM – 0.00
05:00:01 AM – 0.00
06:00:01 AM – 0.00
07:00:01 AM – 0.00
08:00:01 AM – 0.00
09:00:01 AM – 0.00
10:00:01 AM – 0.00
11:00:01 AM – 0.00
12:00:01 PM – 0.00
02:00:01 PM – 0.00
03:00:01 PM – 0.00
04:00:01 PM – 0.00
05:00:01 PM – 0.00
06:00:01 PM – 0.00
07:00:01 PM – 0.00