Bash Shell Shortcuts
Special Shell Variables

Common environment variables

PATH – Sets the search path for any executable command. Similar to the PATH variable in MSDOS.

HOME – Home directory of the user.

MAIL – Contains the path to the location where mail addressed to the user is stored.

IFS – Contains a string of characters which are used as word seperators in the command line. The string normally consists of the space, tab and the newline characters. To see them you will have to do an octal dump as follows:

$ echo $IFS | od -bc

PS1 and PS2 – Primary and secondary prompts in bash. PS1 is set to $ by default and PS2 is set to > . To see the secondary prompt, just run the command :

$ ls |

… and press enter.

USER – User login name.

TERM – indicates the terminal type being used. This should be set correctly for editors like Vim to work correctly.

SHELL – Determines the type of shell that the user sees on logging in.

To see what are the values held by the above environment variables, just do an echo of the name of the variable preceded with a $.

For example, if I do the following:

$ echo $USER
ravi

… I get the value (My login name) which is stored in the environment variable USER.

Some bash shell scripting rules

The first line in your script must be #!/bin/bash
… that is a # (Hash) followed by a ! (bang) followed by the path of the shell. This line lets the environment know the file is a shell script and the location of the shell.
Before executing your script, you should make the script executable. You do it by using the following command:

$ chmod ugo+x your_shell_script.sh

The name of your shell script must end with a .sh . This lets the user know that the file is a shell script. This is not compulsary but is the norm.
Conditional statements
‘if’ Statement
The ‘if’ statement evaluates a condition which accompanies its command line.
syntax:

if condition_is_true
then
//execute commands
else
//execute commands
fi

‘if’ condition also permits multi-way branching. That is you can evaluate more conditions if the previous condition fails.

if condition_is_true
then
//execute commands
elif another_condition_is_true
then
//execute commands
else
//execute commands
fi

Example :

if grep “aboutlinux” thisfile.html
then
echo “Found the word in the file”
else
echo “Sorry no luck!”
fi

if’s companion – test

test is an internal feature of the shell. ‘test’ evaluates the condition placed on its right, and returns either a true or false exit status. For this purpose, ‘test’ uses certain operators to evaluate the condition. They are as follows:

Relational Operators

-eq – Equal to
-lt – Less than
-gt – Greater than
-ge – Greater than or Equal to
-le – Less than or Equal to

File related tests

-f file – True if file exists and is a regular file.
-r file – True if file exists and is readable.
-w file – True if file exists and is writable.
-x file – True if file exists and is executable.
-d file – True if file exists and is a directory.
-s file – True if file exists and has a size greater than zero.

String tests

-n str – True if string str is not a null string.
-z str – True if string str is a null string.
str1 == str2 – True if both strings are equal.
str – True if string str is assigned a value and is not null.
str1 != str2 – True if both strings are unequal.
-s file – True if file exists and has a size greater than zero.
Test also permits the checking of more than one expression in the same line.
-a – Performs the AND function
-o – Performs the OR function

A few Example snippets of using test

test $d -eq 25 && echo $d

… which means, if the value in the variable d is equal to 25, print the value. Otherwise don’t print anything.

test $s -lt 50 && do_something

if [ $d -eq 25 ]
then
echo $d
fi

In the above example, I have used square brackets instead of the keyword test – which is another way of doing the same thing.

if [ $str1 == $str2 ]
then
//do something
fi

if [ -n “$str1” -a -n “$str2” ]
then
echo ‘Both $str1 and $str2 are not null’
fi

… above, I have checked if both strings are not null then execute the echo command.

Things to remember while using test

If you are using square brackets [ ] instead of test, then care should be taken to insert a space after the [ and before the ].
test is confined to integer values only. Decimal values are simply truncated.
Do not use wildcards for testing string equality – they are expanded by the shell to match the files in your directory rather than the string.
Case statement
Case statement is the second conditional offered by the shell.
Syntax:

case expression in
pattern1) //execute commands ;;
pattern2) //execute commands ;;
…
esac

The keywords here are in, case and esac. The ‘;;’ is used as option terminators. The construct also uses ‘)’ to delimit the pattern from the action.

Example:

…
echo “Enter your option : ”
read i;

case $i in
1) ls -l ;;
2) ps -aux ;;
3) date ;;
4) who ;;
5) exit
esac

The last case option need not have ;; but you can provide them if you want.

Here is another example:

case `date |cut -d” ” -f1` in
Mon) commands ;;
Tue) commands ;;
Wed) commands ;;
…
esac

Case can also match more than one pattern with each option.You can also use shell wild-cards for matching patterns.

…
echo “Do you wish to continue? (y/n)”
read ans

case $ans in
Y|y) ;;
[Yy][Ee][Ss]) ;;
N|n) exit ;;
[Nn][Oo]) exit ;;
*) echo “Invalid command”
esac

In the above case, if you enter YeS, YES,yEs and any of its combinations, it will be matched.

This brings us to the end of conditional statements.

Looping Statements

while loop

while loop syntax –

while condition_is_true
do
//execute commands
done

Example:

while [ $num -gt 100 ]
do
sleep 5
done

while :
do
//execute some commands
done

The above code implements a infinite loop. You could also write ‘while true’ instead of ‘while :’ .
Here I would like to introduce two keywords with respect to looping conditionals. They are break and continue.

break – This keyword causes control to break out of the loop.

continue – This keyword will suspend the execution of all statements following it and switches control to the top of the loop for the next iteration.

until loop

until complements while construct in the sense that the loop body here is executed repeatedly as long as the condition remains false.

Syntax:

until false
do
//execute commands
done

Example:

…
until [ -r myfile ]
do
sleep 5
done

The above code is executed repeatedly until the file myfile can be read.

for loop

for loop syntax :

for variable in list
do
//execute commands
done

Example:

…
for x in 1 2 3 4 5
do
echo “The value of x is $x”;
done

Here the list contains 5 numbers 1 to 5. Here is another example:

for var in $PATH $MAIL $HOME
do
echo $var
done

Suppose you have a directory full of java files and you want to compile those. You can write a script like this:

…
for file in *.java
do
javac $file
done

You can use wildcard expressions in your scripts.
Read Regular Expressions Tutorial to know more.

Special symbols used in BASH scripting

$* – This denotes all the parameters passed to the script at the time of its execution. Which includes $1, $2 and so on.
$0 – Name of the shell script being executed.
$# – Number of arguments specified in the command line.
$? – Exit status of the last command.

The above symbols are known as positional parameters. Let me explain the positional parameters with the aid of an example.

Suppose I have a shell script called my_script.sh . Now I execute this script in the command line as follows :

$ ./my_script.sh linux is a robust OS

… as you can see above, I have passed 5 parameters to the script. In this scenario, the values of the positional parameters are as follows:
$* – will contain the values ‘linux’,’is’,’a’,’robust’,’OS’.
$0 – will contain the value my_script.sh – the name of the script being executed.
$# – contains the value 5 – the total number of parameters.
$$ – contains the process ID of the current shell. You can use this parameter while giving unique names to any temporary files that you create at the time of execution of the shell.
$1 – contains the value ‘linux’
$2 – contains the value ‘is’
… and so on.

The set and shift statements
set – Lets you associate values with these positional parameters .

For example, try this:

$ set `date`
$ echo $1
$ echo $*
$ echo $#
$ echo $2

shift – transfers the contents of a positional parameter to its immediate lower numbered one. This goes on as many times it is called.

Example :

$ set `date`
$ echo $1 $2 $3
$ shift
$ echo $1 $2 $3
$ shift
$ echo $1 $2 $3

To see the process Id of the current shell, try this:

$ echo $$
2667

Validate that it is the same value by executing the following command:

$ ps -f |grep bash

Make your BASH shell script interactive

read statement

Make your shell script interactive. read will let the user enter values while the script is being executed. When a program encounters the read statement, the program pauses at that point. Input entered through the keyboard id read into the variables following read, and the program execution continues.

An example –

#!/bin/sh
echo “Enter your name : ”
read name
echo “Hello $name , Have a nice day.”

Exit status of the last command

Every command returns a value after execution. This value is called the exit status or return value of the command. A command is said to be true if it executes successfully, and false if it fails. This can be checked in the script using the $? positional parameter.

So, let’s consider the following log file:

www.three.com 10.15.101.11 1353280801 TEST 345
www.one.com 10.14.101.11 1353280801 TEST 343
www.three.com 1.10.11.71 1353280801 TEST 323
www.one.com 10.15.11.61 1353280801 TEST 365
www.two.com 10.10.11.51 1353290801 TEST 55
www.two.com 10.20.13.11 1353290801 REST 435
www.one.com 10.20.14.41 1353290801 REST 65
www.two.com 10.10.11.14 1353290801 REST 345
www.three.com 10.10.11.31 1354280801 REST 34
www.one.com 10.10.13.144 1354280801 JSON 65
www.two.com 10.50.11.141 1354280801 JSON 665
www.three.com 120.10.11.11 1354280801 JSON 555
www.two.com 10.144.11.11 1383280801 RAW 33
www.one.com 10.103.141.141 1383280801 RAW 315

Now, here are some things you can do to this log file:
How many files are in a directory: ls | wc -l
Print the file: cat sample.log
Print lines that match a particular word: grep “RAW” sample.log
Print those lines to a file called test.log: grep “RAW” sample.log > test.log
Print particular columns and sort: cat sample.log | awk ‘{ print $1,$2}’ | sort -k 1
Find and Replace using SED and Regex: cat sample.log | sed ‘s/TEST/JSON/g’
Split a log file into multiple files using a column as name with AWK: awk ‘{ print >>($4?.log”); close($4?.log”) }’ sample.log
Use substr (removes last character) in AWK to manipulate a string per line: cat sample.log | awk ‘{ print $1,$2,$3,substr($4,1,length($4)-1),$5}’
Print first line of file with SED: sed q test.log
Print last line of file with SED: sed ‘$!d’ sample.log
Perform a regular expression on last character of entire file using SED: cat sample.log | sed ‘$ s/5$//’
Add some text to beginning and end of a file with AWK: cat sample.log | awk ‘BEGIN{print “START” } { print } END{print “END”}’
Count and print how many unique fields are in all rows using AWK: cat sample.log | awk ‘{ if (a[$1]++ == 0) print $1 }’ | wc -l
Make everything lowercase with AWK: cat sample.log | awk ‘{print tolower($0)}’
Multiple SED regular expressions: sed ’1s/^/START/;$ s/5$/END/’ sample.log
Regex with SED on multiple files: for file in *; do sed ’1s/^/START/’ $file > $file’.json’; done

n a command on each file in directory: for i in `ls`; do $i; done
Rename the extension of all files in a folder: for old in *.txt; do mv $old `basename $old .txt`.json; done
Merge all files in a directory with a comma seperator: find . -type f -not -name output.txt -exec cat {} ; -exec echo `,` ; > output.txt

grep, cat, zgrep and zcat

More on log parsing, I’m taking notes on how to read log files and get the information that I need. On Linux environment, these tools are perfect: grep, cat, zgrep and zcat.

Extracting patterns with grep

We can extract information from a text file using grep. Example, we can extract lines of log file containing patterns like GET /checkout/* where status code is 500.

1
grep -E -e ‘GET /checkout/.* HTTP/1\.(0|1)” 500’ some-log-file.log
Depending on the Apache log format, above will extract lines whose request is /checkout/* and status code is 500 where it may support HTTP/1.0 or HTTP/1.1. However, that would extract the whole line. To only extract the matching pattern, use the -o option.

1
grep -o -E -e ‘GET /checkout/.* HTTP/1\.(0|1)” 500’ some-log-file.log
And to save the matching patterns to a file, simply redirect the output to file.

1
grep -E -e ‘GET /checkout/.* HTTP/1\.(0|1)” 500’ some-log-file.log > checkout-errors.txt
Using cat

cat is usually used to output contents of a file. This is a small but very useful Linux utility. For example, we can combine multiple log files (uncompressed) into a single log file.

1
cat /path/to/log-files/*.log > /combined/log-file.log
Compressed counterpart

grep and cat have their compressed file counterpart. For grep, there’s zgrep.

1
zgrep -E -e ‘GET /checkout/.* HTTP/1\.(0|1)” 500’ some-log-file.gz > checkout-errors.txt
For cat, there’s zcat.

1
zcat /path/to/log-files/*.gz > /combined/log-file.log
I’ve done so many combination last week that I don’t remember them all and not able to include in this post. Happy log parsing.

# List out successful ssh login attempts
cat secure | grep ‘Accepted’ | awk ‘{print $1 ” ” $2 ” ” $3 ” User: ” $9 ” ” }’
cat secure* | sort | grep ‘Accepted’ | awk ‘{print $1 ” ” $2 ” ” $3 ” User: ” $9 ” IP:” $11 }’

# List out successful ssh login attempts from sudo users
cat /var/log/secure | grep ‘session opened for user root’ | awk ‘{print $1 ” ” $2 ” ” $3 ” Sudo User: ” $13 ” ” }’

# List out ssh login attempts from non-existing and unauthorized user accounts
cat /var/log/secure | grep ‘Invalid user’

# List out ssh login attempts by authorized ssh accounts with failed password
cat /var/log/secure | grep -v invalid | grep ‘Failed password’

Indeed, and even grep | awk can be shortened to awk /…/. So you could save a bit of space in the final script. For a typical log file (~200 kb), you might save 1 ms processing it. Or to be exact, 1.8 ms removing the cat and grep, and 0.3 ms using only awk instead of grep | awk.

time for i in `seq 1000`; do cat secure | grep Accepted | awk ‘{print $1 ” ” $2 ” ” $3 ” User: ” $9 ” ” }’; done > /tmp/a
time for i in `seq 1000`; do grep Accepted secure | awk ‘{print $1 ” ” $2 ” ” $3 ” User: ” $9 ” ” }’; done > /tmp/b
time for i in `seq 1000`; do awk ‘/Accepted/ {print $1 ” ” $2 ” ” $3 ” User: ” $9 ” ” }’ secure; done > /tmp/c

However, more interestingly, when the size of the log file is increased to 200 MB, it turns out that the cat | grep | awk chain is significantly faster, at 1.096 s over 100 runs. The single awk command will not max out the CPUs, while the pipe chain does.

for i in `seq 1000`; do cat secure >> s; done
time for i in `seq 100`; do cat s | grep Accepted | awk ‘{print $1 ” ” $2 ” ” $3 ” User: ” $9 ” ” }’; done > /tmp/a1
time for i in `seq 100`; do awk ‘/Accepted/ {print $1 ” ” $2 ” ” $3 ” User: ” $9 ” ” }’ s; done > /tmp/c1

My perl script awk:
Code:
my $OldTime= ‘Sep 10, 2012 5:20:41′;
my $NewTime=’Sep 10, 2012 5:49:40’;

my $test2 = qx{ssh -o stricthostkeychecking=no $WLS “awk ‘/$OldTime/,/$NewTime/’ $WLSP/logs/CDSServer.* “};
My log file format:
Code:
#### Kernel>> <>
#### <[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'>
#### <12d58f5205084394:4db5aec7:1393b6c3ce7:-8000-00000000 #### <[ACTIVE] ExecuteThread: ' Reply With Quote Reply With Quote 09-18-2012 #2 atreyu atreyu is offline Trusted Penguin Join Date May 2011 Posts 4,353 Hi, I'd make two suggestions. The first one is to use the Date::Parse Perl module. It is hopefully already packaged for your distro. This module will allow you to easily convert date/time strings to seconds since the epoch (which is an easy way to do date/time math). It will give you the equivalent output to this GNU date command: Code: date +%s -d "Sep 10, 2012 5:20:41 PM" The second suggestion would be to put the script on the server and pass to it 3 arguments: 1. the start date/time range 2. the end date/time range 3. the log file to parse then you'd do something like this to call it: Code: ssh server /tmp/parse-log.pl 'Sep 10, 2012 5:20:41 PM' 'Sep 10, 2012 5:44:42 PM' /path/to/CDSserver.log and here is the parse-log.pl script: Code: #!/usr/bin/perl use strict; use warnings; use Date::Parse; # get command line arguments (3) die " Usage: $0 '‘ ‘‘
E.g.: $0 ‘Sep 10, 2012 5:20:41 PM’ ‘Sep 10, 2012 5:49:40 PM’ CDSServer.log\n”
unless($#ARGV == 2);

my $startTime = $ARGV[0];
my $stopTime = $ARGV[1];
my $log = $ARGV[2];

# make sure the log file exists
die “$log: No such file\n” unless(-f$log);

# convert date/time strings to seconds since epoch
my $start_sec = str2time($startTime);
my $stop_sec = str2time($stopTime);

print “Start time: $startTime ($start_sec)\n”;
print “Stop time: $stopTime ($stop_sec)\n”;

open(LOG,’<',$log) or die "can't read '$log': $!\n"; while(){
chomp;
my $line = $_; # save original line
s/[ \t]+/ /; # replace contiguous white spaces w/single space
if(/^####<([a-zA-Z]{3} [0-9]{1,2}, [0-9]{4} [0-9]{1,2}:[0-9]{2}:[0-9]{2} [AP]M) [A-Z]{3}>/){
my $timedate = $1;

# convert date/time string in log entro to epoch seconds
my $seconds = str2time($timedate);

# print line if it falls into the range
print $line,”\n” if(($seconds >= $start_sec)&&($seconds <= $stop_sec)); } } close(LOG); Reply With Quote Reply With Quote 09-19-2012 #3 charith charith is offline Just Joined! Join Date Nov 2010 Posts 26 Hi atreyu, Thank you very much for your great clean reply. Your script working fine and it gives lines those match given times but what i need is get all lines whatever between that time range. I'm sorry for my bad log file format i attached correct log file below.[Errors not begin with time] Code: #### f-tuning)’> <> <> <> <1345916502340> <[ACTIVE] ExecuteThread: '20' for queue: ' java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:443) at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:670oracle.jdbc.driver.T4CConnection.(T4CConnection.java:230
### <[ACTIVE] ExecuteThrea #### <[ACT Above java.sql.SQLRecoverableException: IO Error: part should be retrieve. Reply With Quote Reply With Quote 09-20-2012 #4 atreyu atreyu is offline Trusted Penguin Join Date May 2011 Posts 4,353 ah, okay. yeah, that changes things, but not by too much. basically, you can just set a marker once the start time string is matched, then set a stop marker once the end time string is matched, and save everything in between to an array. then print the array once you're done looping thru the file. Code: #!/usr/bin/perl use strict; use warnings; use Date::Parse; # get command line arguments (3) die " Usage: $0 '‘ ‘‘
E.g.: $0 ‘Sep 10, 2012 5:20:41 PM’ ‘Sep 10, 2012 5:49:40 PM’ CDSServer.log\n”
unless($#ARGV == 2);

my $startTime = $ARGV[0];
my $stopTime = $ARGV[1];
my $log = $ARGV[2];

# make sure the log file exists
die “$log: No such file\n” unless(-f$log);

# convert date/time strings to seconds since epoch
my $start_sec = str2time($startTime);
my $stop_sec = str2time($stopTime);

# make sure we got nothing but digits in the variables
die “Failed to convert $startTime to seconds\n” unless($start_sec =~ /^[0-9]*$/);
die “Failed to convert $stopTime to seconds\n” unless($stop_sec =~ /^[0-9]*$/);

print “Start time: $startTime ($start_sec)\n”;
print “Stop time: $stopTime ($stop_sec)\n”;

my @lines;
my $stop;

open(LOG,’<',$log) or die "can't read '$log': $!\n"; while(){
chomp;
my $line = $_; # save original line
s/[ \t]+/ /; # replace contiguous white spaces w/single space
if(/^####<([a-zA-Z]{3} [0-9]{1,2}, [0-9]{4} [0-9]{1,2}:[0-9]{2}:[0-9]{2} [AP]M) [A-Z]{3}>/){
my $timedate = $1;

# convert date/time string in log entro to epoch seconds
my $seconds = str2time($timedate);

# save line w/time string to array if it falls into the range
if($seconds >= $start_sec){
push(@lines,$line) unless($stop);
}elsif($seconds >= $stop_sec){
push(@lines,$line) unless($stop);
$stop = 1;
}
}else{
# save line w/o time string to array if it falls into the range
push(@lines,$line) if(($#lines>=0)&&!($stop));
}
}
close(LOG);

# print the saved lines
print “$_\n” for(@lines);
Reply With Quote Reply With Quote
09-24-2012 #5
charith charith is offline
Just Joined!
Join Date
Nov 2010
Posts
26
Hi atreyu,
It’s working fine thank you very much.

Did small change:
Code:
# save line w/time string to array if it falls into the range
if($seconds >= $start_sec){
as
Code:
if (($seconds >= $start_sec)&&($seconds <= $stop_sec)){ } } close(LOG); #!/usr/bin/perl use strict; use warnings; use Date::Parse; # get command line arguments (3) die " Usage: $0 '‘ ‘‘
E.g.: $0 ‘Sep 10, 2012 5:20:41 PM’ ‘Sep 10, 2012 5:49:40 PM’ CDSServer.log\n”
unless($#ARGV == 2);

my $startTime = $ARGV[0];
my $stopTime = $ARGV[1];
my $log = $ARGV[2];

# make sure the log file exists
die “$log: No such file\n” unless(-f$log);

# convert date/time strings to seconds since epoch
my $start_sec = str2time($startTime);
my $stop_sec = str2time($stopTime);

# make sure we got nothing but digits in the variables
die “Failed to convert $startTime to seconds\n” unless($start_sec =~ /^[0-9]*$/);
die “Failed to convert $stopTime to seconds\n” unless($stop_sec =~ /^[0-9]*$/);

print “Start time: $startTime ($start_sec)\n”;
print “Stop time: $stopTime ($stop_sec)\n”;

my @lines;
my $stop;

open(LOG,’<',$log) or die "can't read '$log': $!\n"; while(){
chomp;
my $line = $_; # save original line
s/[ \t]+/ /; # replace contiguous white spaces w/single space
if(/^####<([a-zA-Z]{3} [0-9]{1,2}, [0-9]{4} [0-9]{1,2}:[0-9]{2}:[0-9]{2} [AP]M) [A-Z]{3}>/){
my $timedate = $1;

# convert date/time string in log entro to epoch seconds
my $seconds = str2time($timedate);

# save line w/time string to array if it falls into the range
if($seconds >= $start_sec){
push(@lines,$line) unless($stop);
}elsif($seconds >= $stop_sec){
push(@lines,$line) unless($stop);
$stop = 1;
}
}else{
# save line w/o time string to array if it falls into the range
push(@lines,$line) if(($#lines>=0)&&!($stop));
}
}
close(LOG);

# print the saved lines
print “$_\n” for(@lines);