grep
grep is a saviour command for unix users. grep can be used to search for patterns in a file or standard input. The pattern search includes finding the line number of the keyword, counting the number of occurrences of a keyword and many more. we will see in the following examples.
Example 1) grep in its simplest form,i:e without any arguments displays all the lines where the pattern occurs.
/home/kapoor $ cat AllPhones.lst
Samsung C5010 Squash : Rs. 3,245
Samsung C5130 : Rs. 3,500
LG GU285 : Rs. 3,750
Nokia C2 01 : Rs. 3,799
Nokia 2730 Classic : Rs. 3,960
INQ Mini 3G : Rs. 4,000
Spice G6500 : Rs. 4,295
Sony Ericsson Cedar : Rs. 5,100
Samsung Star Nano 3G S3370 : Rs. 5,100
Samsung L700 : Rs. 5,300
Spice QT95 : Rs. 5,500
nokia 7230 : Rs. 5,700
Sony Ericsson J105 Naite : Rs. 5,800
Samsung Metro 3G S5350 : Rs. 6,000
/home/kapoor $ grep Nokia AllPhones.lst
Nokia C2 01 : Rs. 3,799
Nokia 2730 Classic : Rs. 3,960
To make case insensitive search use grep with –i option
i:e grep –i Nokia AllPhones.lst
now the result would also include the line nokia 7230 : Rs. 5,700
Example 2)
It might be required for you to count the number of occurrences of a particular keyword, use -c option .
/home/kapoor $ grep -c -i Samsung AllPhones.lst
5
Remember always to use –i just before the search keyword.
Example 3)
You might be searching a particular thing excluding a particular word, then use -v option.
/home/kapoor $ grep –v Samsung AllPhones.lst
LG GU285 : Rs. 3,750
Nokia C2 01 : Rs. 3,799
Nokia 2730 Classic : Rs. 3,960
INQ Mini 3G : Rs. 4,000
Spice G6500 : Rs. 4,295
Sony Ericsson Cedar : Rs. 5,100
Spice QT95 : Rs. 5,500
nokia 7230 : Rs. 5,700
Sony Ericsson J105 Naite : Rs. 5,800
Example 4)
When you wish to search more than one keyword use –e option of grep before each keyword.
/home/kapoor $ grep -e LG –e –i spice AllPhones.lst
LG GU285 : Rs. 3,750
Spice G6500 : Rs. 4,295
Spice QT95 : Rs. 5,500
Similarly if you require to search for all other phones other than LG and spice use
grep -v -e LG –e –i spice AllPhones.lst
Example 5)
To accomplish a search within a searched line use pipes. suppose you are planning to buy
a nokia phone other than a classic one, use grep as shown
/home/kapoor $ grep –i nokia AllPhones.lst | grep -v –i Classic
Nokia C2 01 : Rs. 3,799
nokia 7230 : Rs. 5,700
why grep is such a great command in searching these results for you is that it does them amazingly fast.
Example 6)
You have seen in Example 4) how multiple keyword search was done using –e option. but it becomes very lengthy and untidy to use –e after each keyword if there are 100s of things to be searched.
In such cases you can use grep with -f option.the below lines show how.
Save the list of phones in a file.
/home/kapoor $cat >required_phones
Sony Ericsson
INQ
Nokia
^D
/home/kapoor $grep –f required_phones AllPhones.lst
Nokia C2 01 : Rs. 3,799
Nokia 2730 Classic : Rs. 3,960
INQ Mini 3G : Rs. 4,000
Sony Ericsson Cedar : Rs. 5,100
Sony Ericsson J105 Naite : Rs. 5,800
Similarly you can use –v –f to display all the lines except those in the
The situation is like this. You have a key word and you want to search it in a large number of
files in a path. The following examples (7-9)show you various techniques and scenarios.
Example 7)
To give only a list of files which contain your keyword use –l option of grep.
The command below lists all the scripts which uses awk .
/home/kapoor $grep -l “awk” *.sh
Pattern-gen.sh
Large-box.sh
Example 8)
If grep is used to search in a group of files for a pattern, it gives the following default output.
/home/kapoor $grep sed *.sh
pgfile-2.sh:sed -n 1,$p $h2_file.txt
pgfile-2.sh:sed ‘s/;//g’ $h3_file.txt
daily-b1.sh:sed ‘s/-/_/g’ prime_f.js
If you do not want grep to display the searched filenames and only want the patterns use grep -h
grep -h sed *.sh gives you the output.
sed -n 1,$p $h2_file.txt
sed ‘s/;//g’ $h3_file.txt
sed ‘s/-/_/g’ prime_f.js
Example 9)
To search for patterns recursively among files of directory and its sub directories use grep -r .
/home/kapoor $grep -l -r awk *.sh
Pattern-gen.sh
Large-box.sh
Shell1/getfiles_2.sh
Shell1/remainder.sh
Shell1/byte-logs.sh
Shell1/new-scripts/byte-logs.sh
Shell2/vault-value.sh
Example 10)
In shell scripts you may need to search for a pattern but do not want the command to write to
Standard output but you need to decide on whether the search succeeded, then use grep –q.
….. #lines of the script
….. ‘’
grep -q win victory-status.log
if [ $? -eq 0 ]
then
flag=1
else
flag=0
fi
..
..
Here grep does not give any output, but the exit condition is 0 if search is found and non zero if not found ( $? Stores the exit condition of the output of previous command.)
Example 11)
You can also know the line numbers of the searched patterns through grep by using –n command.
/home/kapoor $ grep –n error status-run.log
40: error cannot open file gt1.txt
160:error cannot open file uu6.txt
178:parse error
alias
alias
alias is one of those commands for people who want to be lazy. you can use alias
in situations where it is too time consuming to type the same commands again and again.
But avoid aliases to commands like rm, kill etc.
Example. 1)
To always use vim instead of vi, and to make sure that whenever
there is a system crash, network failure etc during editing ,
all the contents are recovered, use the alias as follows.
/home/viru$ alias vi = ‘vim –r’
To make this happen every time you work after logging in, save the above line in
your .profile
i:e in the file $HOME/.profile
after saving it in .profile do not forget to run it.
ie /home/viru$ . $HOME/.profile
|
|
a dot here is necessary.
Example2) after running .profile ,to view all the aliases that are set globally, just enter alias.
/home/viru$ alias
alias ls=’ls –lrt’
alias psu=’ps –fu $LOGNAME’
alias df =‘df –gt’
alias jbin=’cd /home/viru/utils/java/bin’
alias jlib=’cd /home/viru/utils/java/lib’
seems viru is so lazy..!!
Example3)
To prevent the effect of aliases defined for a word, token or a command, use unalias.
/home/viru$ unalias jbin
/home/viru$ jbin
Jbin:not found
There is another way to accomplish this,that is by using quotes (‘’) after a command.the following lines show how.
/home/viru$ alias same
alias same=’/opt/bin/samefile.exe’
/home/viru$ same ‘’
same:not found.
The effect of alias was nullified for the word same. If you use double quotes after an aliased command name ,(eg: alias df =‘df –gt’) then the actual command will be run.(ie df’’ would run /usr/bin/df instead of df -gt)
Aliases which are defined outside a script or in your .profile do not work inside scripts. so make sure not to use aliased words in a shell script to contain their actual values used outside.
df
df
df command gives you the information of the disk spaces. It is a very good command for
system administration and file system monitoring. You can also view various parameters
like total number of files in a particular file system and maximum allocated values.
Reducing the space occupied by a file system helps to boost system performance by making
The applications and utilities that use these file systems take lesser time for accessing data,
So constantly running df helps.
Example 1)
df comes various options like -g,-m,-k etc which helps you to view file system sizes in gigabytes, megabytes or kilobytes blocks resp , including t gives total allocated space.
Consider the output of df –gt.
/home/Prod$ df –gt.
Filesystem GB blocks Used Free Used Mounted on
/dev/hd4 0.75 0.35 0.40 48% /
/dev/hd2 5.50 3.68 1.82 67% /usr
/dev/hd9var 4.00 0.63 3.37 16% /var
/dev/hd3 4.62 0.71 3.91 16% /tmp
/dev/hd1 8.25 3.96 4.29 48% /home
/dev/hd10opt 4.50 1.30 3.20 29% /opt
/dev/lv00 0.12 0.00 0.12 4% /var/adm/csd
/dev/fslv00 5.00 0.03 4.97 1% /utilities
/dev/ora10g_lv 15.00 5.18 9.82 35% /ora10g
The output of the command might not look organised in some unix systems.ie: alignment does not seems to be right,
Use awk for help.
/home/Prod$ df –gt | awk ‘BEGIN{aa=”Filesystem”;bb=”GB blocks”;cc=”Used”;dd=”Free”;ee=”Used”;ff=”Mounted on”;printf(“%-40s %9s %6s %6s %6s %-11s\n”,aa,bb,cc,dd,ee,ff);} {if($1!=”Filesystem”){printf(“%-40s %9s %6s %6s %6s %-9s\n”,$1,$2,$3,$4,$5,$6) } }’
Filesystem GB blocks Used Free Used Mounted on
/dev/hd4 0.75 0.35 0.40 48% /
/dev/hd2 5.50 3.68 1.82 67% /usr
/dev/hd9var 4.00 0.63 3.37 16% /var
/dev/hd3 4.62 0.71 3.91 16% /tmp
/dev/hd10opt 4.50 1.30 3.20 29% /opt
/dev/lv00 0.12 0.00 0.12 4% /var/adm/csd
/dev/fslv00 5.00 0.03 4.97 1 % /utilitie/dev/hd1
/dev/hd1 8.25 3.96 4.29 48% /home
/dev/ora10g_lv 15.00 5.18 9.82 35% /ora10g
Now ,that seems to be a much better organised output. Use this in a script and keep an
alias for df command to this script named df.
Example 2)
To constantly monitor disk space you can make a script and run it constantly.
/home/Prod$ cat disk-monitor.sh
#!/usr/bin/ksh
for Pspace in `df -gt | awk ‘{ print $4}’ | grep –v -e used -e /ora | tr -d ‘%’ `
do
if [ $ Pspace -gt 80 ]
then
alarmfunc #call the alarm function.
fi
done
Example 3)
df is also helpful in knowing number of files present in a mount point.
It is also referred to as Inode. For all all the file system mount points there is a
Inode value assigned which gives the maximum number of files that can be placed in the file system. When the number of files exceed too a large value, you get ‘parameter list is too long’
error with commands that use wildcards as arguments to filename.
Every file has an inode value of 1 and a directory has an inode value of 2.
To see the Inode values along with other attributes,use df –i, and off course with awk filtering.(Use different values before %s for proper alignment)
/home/Prod$ df -i
Filesystem 512-blocks Free %Used Iused %Iused Mounted on
/dev/hd0 19368 9976 48% 4714 5% /
/dev/hd1 24212 4808 80% 5031 19% /usr
/dev/hd2 9744 9352 4% 1900 4% /site
/dev/hd3 3868 3856 0% 986 0% /tmp
/dev/hd1 56890 23409 41% 2345 40% /home
wc
wc
wc is a very useful utility which can count lines, characters, words, bytes etc in a plain text file
or standard input. In shell scripts, It helps to store a value for the total lines of a file or output of
a command into a variable for subsequent use. The examples below show how these things can be achieved.
Example 1)
To get the counts of lines, words and bytes of a file use wc as follows.
/home/mark$ wc Bulk-SMS-file.txt
45 140 990——————————> No of bytes
| |———-> No of Words
No of lines
The individual values can be retrieved as field through awk , but wc provides options.
wc -l : Total Lines
wc -w : Total words
wc -c : Total bytes
wc -k : Total characters
The wc command considers a word to be a string of characters of non-zero length which are delimited by a white space and lines are counted when newline characters occur.
Example 2)
Using wc on multiple files.
/home/mark$ ls file_list-201111*
file_list-20111105.txt
file_list-20111113.txt
file_list-20111120.txt
file_list-20111128.txt
/home/mark$ wc -l file_list-201111*
98164 file_list-20111105.txt
531665 file_list-20111113.txt
527303 file_list-20111120.txt
564207 file_list-20111128.txt
1721339 total
This gives you the no of lines in every file as well as the sum of all the individual line counts.
Same command can be run without the -l option to get the counts of various parameters and their totals.
Example 3)
wc can also count these values from standard output through pipes.
/home/mark$ echo “I Love You” | wc
1 3 17
Hmm! wc achieved something cool this time…
Similarly you can use wc -l as an alternative to grep -c.
/home/mark$ grep –c tremendous Director-speech.txt
Is same as
/home/mark$ grep tremendous Director-speech.txt | wc -l
Example 4)
wc -l may consume a lot of time on counting the line numbers when the size of the file is huge.
If we are sure that every line in a file contains equal number of characters, there is a faster method to achieve it.
Assume that you have a continuous file, i.e. a file which has same number of characters (or bytes ) in each line.
/home/mark$ ls -lrt All_Customers.lst
-rw-r–r– 1 mark Administ 1321655460912 Dec 16 14:37 All_Customers.lst
/home/mark$ wc -l All_Customers.lst
23456898
The following steps explain the method.
Step1)
save the first thousand lines of the file in a separate file.
/home/mark$ head -1000 All_Customers.lst > All_Customers_1000.lst`
Step2)
get the ratio of the total size of the file to the line count(1000) of the file.
/home/mark$ ls -lrt All_Customers_1000.lst
-rw-r–r– 1 mark Administ 56344000 Dec 16 14:37 All_Customers.lst
/home/mark$ fsize=` ls -lrt All_Customers_1000.lst | awk ‘{ print $5}’`
/home/mark$ fratio=`expr $fsize / 1000`
/home/mark$ echo $fratio
56344
Now ,fratio actually stores the number of bytes per file.
Step3)
Now divide the total size of All_Customers.lst with fratio.
/home/mark$tot_lines=` expr 1321655460912 / $fsize`
/home/mark$echo $tot_lines
23456898
Which is same as calculated by wc -l.
All These steps can be used in a shell script for any given file of this type.
If there are multiple files of such types (such as file_list-201111*) and all have the same fratio.(defined above),then you can write a script to get the similar output as that of
wc –l file_list-201111*
The script is as shown.
fratio=313
tot_val=0
for stream in `ls file_list-201111*`
do
str=`ls -lrt $stream|awk ‘{print $5 ” + “}’|tr -d “\n”|sed ‘s/$/0/’`
val=`echo “($str)/$fratio”|bc`
echo “$val $stream ”
tot_val=`expr $tot_val + $val`
done
echo ” $tot_val total ”
Here ,between the output of ls giving sizes of all the files ,a ‘ +’ symbol is placed and 0 at the end and is passed to bc command as an expression whose value (the sum) is divided by the
ratio just as explained in the 3 steps above.
tail
tail
tail command is one of my favourite commands because of its usefulness in viewing the last few lines of a required file and also what is being written into a file ( using –f option).it can also be used to see the progress of cp, mv, tar and other commands.
tail by default shows last ten lines of a file.it can be made to view any last ‘n’ lines where n is a number
Example 1)
to view the last 1000 lines of a huge script page wise ,
/home/k109$ tail -1000 valuefind.sh | pg
……………..
……………… #lines of scripts
………..
.
.
.
……………..
Standard input <--- #here pg waits and asks for you to enter a key before which it displays the next page. This is helpful in analysing a script or debugging it page wise.
Example 2)
To view the file contents while it is being written, use –f option. If an application is writing continuously into a file and to know what is being written, this option can be used.
One disadvantage of this command is however that it cannot be used inside scripts because once
The application stops writing to a file the command does not come out of tail directly and requires some signal like stop or kill to come out.
Suppose a file is being written by cp command, you can use tail –f to know
exactly what is being written.
Example 3)
To view the contents of the file starting from a particular line, use tail +
Suppose you have a file worldsport.txt
/home/k109$cat worldsport.txt
The game is played on a rectangular field of grass or green artificial turf, with a goal in the middle of each of the short ends. The object of the game is to score by driving the ball into the opposing goal. In general play, the goalkeepers are the only players allowed to touch the ball with their hands or arms, while the field players typically use their feet to kick the ball into position, occasionally using their torso or head to intercept a ball in midair. The team that scores the most goals by the end of the match wins. If the score is tied at the end of the game, either adraw is declared or the game goes into extra time and/or a penalty shootout, depending on the format of the competition.
If it is required to view the file starting from second line,use
/home/k109$tail +2 worldsport.txt
short ends. The object of the game is to score by driving the ball into the opposing goal. In general play, the goalkeepers are the only players allowed to touch the ball with their hands or arms, while the field players typically use their feet to kick the ball into position, occasionally using their torso or head to intercept a ball in midair. The team that scores the most goals by the end of the match wins. If the score is tied at the end of the game, either adraw is declared or the game goes into extra time and/or a penalty shootout, depending on the format of the competition.
sort
sort
Sort , as the name suggests sorts a file or output of a command. Various options determine the sort criteria which are called as sort keys. If multiple files are passed as parameters to sort the output of sort is concatenated.
Sorting is done in ascending lexicographic order , I e the way in which it appears in a dictionary. If a file contains numbers and alphabet both, by default sort places the sorted alphabets first and then the numbers.
Example1)
Consider a file containing values as shown.
/home/jones$ cat flowers.txt
rose
lily
tulip
marigold
hibiscus
Chrysanthemum
/home/jones$ sort flowers.txt
Chrysanthemum
hibiscus
lily
marigold
rose
tulip
If you want to sort it in reverse order ,use
/home/jones$ sort -r flowers.txt
tulip
rose
marigold
lily
hibiscus
Chrysanthemum
Example2)
If you want the sort to to be case insensitive sort –f must be used.
/home/jones $ cat flowers.txt
rose
lily
Chrysanthemum
tulip
marigold
Rose
hibiscus
chrysanthemum
To default sort(without any option) would give
/home/jones $ Sort flowers.txt
Chrysanthemum
Rose
Chrysanthemum
hibiscus
lily
marigold
rose
tulip
Here it sorted the letters starting with uppercase first and then sorted lower case ones, Now use sort -f
/home/jones $ sort –f flowers.txt
Chrysanthemum
chrysanthemum
hibiscus
lily
marigold
Rose
rose
tulip
Example3)
A file may contain duplicate lines, and if you want to remove duplicate files before sorting , use sort –u
/home/jones $ grep error cron.log
error code 45:invalid time
error code 35:Invalid name
error code 45:invalid time
error code 25:Invalid email-id
error code 25:Invalid email-id
error code 35 Invalid name
/home/jones $ grep error cron.log | sort -u
error code 25:Invalid email-id
error code 35:Invalid name
error code 45:invalid time
Example 4)
You require to sort based on a particular field when they are separated by a delimiter.
/home/jones $ cat detailed-list.csv
dolphin|mammal|12
giraffe|mammal|7
kingfisher|aves|3
moth|insecta|1
shark|fish|6
viper|reptile|2
Now, The output of default sort command is
dolphin|mammal|12
giraffe|mammal|7
kingfisher|aves|3
moth|insecta|1
shark|fish|6
viper|reptile|2
it sorted on the basis of first field (separated by ‘|’) in alphabetic order. For a change, you required to sort it based on another column.
/home/jones$ sort -t “|” +1 detailed-list.csv
kingfisher|aves|3
shark|fish|6
moth|insecta|1
dolphin|mammal|12
giraffe|mammal|7
viper|reptile|2
Here -t “|” tells the sort command do sorting on fields delimited by a “|” character. If you do not use -t option, sequence of space characters is considered as default delimiter . “+1” instructs sort to ignore the first field or sort from second field.
Similarly if you wanted to sort it based on the third column, just using +2 instead of +1 might not work in the given example. The output of that sort command would be as follows.
/home/jones$ sort -t “|” +2 detailed-list.csv
moth|insecta|1
dolphin|mammal|12
viper|reptile|2
kingfisher|aves|3
shark|fish|6
giraffe|mammal|7
It sorted the third column according to its alphabetic value and not arithmetic value. To do so you must use -n option
/home/jones$ sort -n -t “|” +2 detailed-list.csv
moth|insecta|1
viper|reptile|2
kingfisher|aves|3
shark|fish|6
giraffe|mammal|7
dolphin|mammal|12
Example5)
The sorting based on fields can be achieved using sort with –k option. consider a file containing numbers.
/home/jones$ cat num-luck
6758 987 456
2586 324 934
0437 235 417
2586 324 934
Suppose you wanted to sort this list such that sorting should start from 3rd column of 1st field and 4th column of 1st field. Here fields are separated by one or more spaces. Columns here refer to characters.
/home/jones$ sort -k1.3,1.4 num-luck
2812 624 208
0437 235 417
6758 987 456
2586 324 934
To sort from 2nd column of 1st field and 3rd column of second field in reverse order,
/home/jones$ sort -k1.2,2.3r num-luck
2812 624 208
6758 987 456
2586 324 934
0437 235 417
Similarly, to sort lines based on 1st and 3rd fields, use
Sort –k1 –k3
Example 6)
You may require to sort a particular file and rewrite the file with the sorted file. A simple command of the form
Sort filename >new_sorted_filename will not work and is dangerous as it rewrites it into an empty file. Use sort with –o option.
/home/jones$ sort -o flowers.txt flowers.txt
The syntax is sort –o
Example 7)
Sort uses a lot of temporary space while sorting huge files. and by default it uses /tmp directory. If sufficient space was not allocated to /tmp, then the command would abort abruptly. so sort provides an option -T by which you can use an alternative directory for storing temporary files.
The sort command sort -t “|” +1 detailed-list.csv in example 4 can be written as
sort -t “|” +1 -T /backup/jones detailed-list.csv.
this will use /backup/jones directory for storing temporary files.
sed
sed
sed or stream editor is an editing utility that modifies lines of a file as specified by an instruction string and writes it to standard output .sed can thus perform the same functions of the commands like tr, grep and awk ,even though sed has functionalities and options unique to itself.
Though the applications of sed are enormous, only some of the important uses of the command are discussed here with examples.
Example 1)
It is required to you to print the contents of a file starting from a particular line number and ending into another ,use sed as follows.
/home/UAT$ sed -n ‘35,46p’ general-dates.prt
This command prints the contents of the file “general-dates.prt “ starting from its 35th line to the 46th.
To print only the 35th line of a file use
/home/UAT$ sed -n ‘35p’ general-dates.prt
More examples
· To print all the lines of a file starting from its 1st line to the last line
sed -n ‘1,$p’ general-dates.prt
· To print all the lines of a file between line numbers whose values are stored in variables $start-line and $end-line .
sed –n “${start-line} , ${end-line}p” general-dates.prt
· To print all the lines of the file except the 5th line
sed ‘5d’ general-dates.prt
· To print all the lines of the file except those lines between the 8th and the 40th (i.e. 8th line and 40th lines are also not printed).
sed ‘8,40d’ general-dates.prt
Example2)
sed can also search for patterns in file or standard input just like grep , but much more sophisticated pattern searches can be done as shown in the following examples.
· A grep command can be implemented using sed as follows
Consider a grep command
grep industry general-dates.prt
alternatively sed can perform the same using..
sed –n ‘/industry/p’ general-dates.prt
· To print all the lines of a file between the first occurrences of two patterns
sed –n ‘/industry/,/station/p’ general-dates.prt
This prints all the lines starting from the line which has the first occurrence of ‘ industry’ and the first occurrence of ‘station’ .
· To print all the lines of a file starting from its first line and the line of first occurrence of ‘industry’,
sed –n ‘1,/ industry/p’ general-dates.prt
· To put all the lines starting from line with the first occurrence of the pattern ‘industry’ in the file to the last line of the file into standard output(i.e. print).
sed –n ‘/industry/,$p’ general-dates.prt
· To print all the lines starting from the line containing the pattern string whose value is stored in a variable $ptrn to the 15th line of the file.
sed –n ‘/’$ptrn‘/,15p’ general-dates.prt
The pattern space can contain more complex regular expressions .
Example3)
sed can replace the occurrences of patterns into a new one from a file and put it to standard output.
· To replace the first occurrence of ‘teacher’ with ‘tutor’ in the file groupsList.xml
sed ‘s/ teacher/ tutor/’ groupsList.xml
· To replace all the occurrences of ‘teacher’ with ‘tutor’ in the file groupsList.xml
sed ‘s/ teacher/ tutor/g’ groupsList.xml
here g -> represents global.
· To replace all the occurrences of a string stored in a variable ${string_pattern} to ${replace_string} in a file.
sed ‘s/’${string_pattern}’/’ ${replace_string}’/g’ groupsList.xml
· To add a “,” character between the original pattern and replacement string defined by ${string_pattern}’ and ${replace_string} respectively .
sed ‘s/’${string_pattern}’/&,’${ replace_string }’/g’ groupsList.xml
the single quotes enclosing the variable is a necessary.
& variable stores the matched pattern when the string stored in ${string_pattern} is a regular expression.
Example 4)
Sed can be used to edit a particular line of the file. amazing..isnt it..?
/home/UAT$ cat dir_struct
1. 35467 vdn /home/vd2
2. 46788 ghi /var/gh1/gh
3. 89078 bjk /home/vd2
4. 56890 lod /home/lod/inter
5. 33456 bhj /home/bhind
6. 45790 krk /myhome/krk
Now if I want to change the directory structure on line number 4 from /home to /var, sed can be used as follows.
/home/UAT$ sed ‘4 s/home/var/’ dir_struct
1. 35467 vdn /home/vd2
2. 46788 ghi /var/gh1/gh
3. 89078 bjk /home/vd2
4. 56890 lod /var/lod/inter
5. 33456 bhj /home/bhind
6. 45790 krk /myhome/krk
Here we substituted only home with var in the 4th line .If you need to substitute /home/vd2 with /var/vd1 , you would have written the command as follows.
sed ‘4 s/\/home\/vd2/\/var\/vd1/’ dir_struct
looks a little ugly because we used ‘/ ‘ as delimiter and to differentiate the ‘/’ in /home/vd2 and /var/vd1, a backslash “\”was used before it.
instead of using ‘/’as delimiter other characters like : , | , { , # , ! etc can be used for such scenarios.
The above sed can be written using # as follows.
sed ‘4 s#/home/vd2#/var/vd1#’ dir_struct
Example 5)
You have many occurrences of a pattern in a line and you want to replace just the nth pattern or all n+1,n+2.. occurrences . use a number after the ending “/”.
/home/jade$ echo “1st 2nd 3rd 4th 5th 6th”|sed “s/[0-9]th//2”
1st 2nd 3rd 4th 6th # |—–second occurrence of [0-9]th
The command above replaced only the 2nd occurrence of a digit followed by “th”.
If you need to replace all occurrences starting from 2nd,then use the following command.
/home/jade$ echo “1st 2nd 3rd 4th 5th 6th”|sed “s/[0-9]th//2g”
“1st 2nd 3rd 4th ”
rsh
rsh
rsh is used to execute commands on a remote machine.
The rsh command executes the command or a program in another host from current working machine without having to login into that remote machine by entering a password as in ssh. You can run any unix command, or a shell script of a remote host.
Prerequisites
rsh command cannot be executed without making these changes.
1) make a list of the users of both the local and remote hosts and their $LOGNAME s who are going to use rsh to run commands or scripts.
2)Open the required ports which is going to be used for rsh and rcp.
3)make an entry of remote host in /etc/hosts file of the local host and an entry of local host in /etc/hosts of the remote host. This change can done my root user,if you do not have root access then contact your unix admin.
A sample entry is as shown, assume that your hostname is mahaprana and the IP address is 158.0.125.23, Then the remote host’s /etc/hosts file must contain a line
158.0.125.23 mahaprana #backup server.
the comments (3rd column) is optional.
Similarly make an entry of remote host In /etc/hosts file of local host. If remote hostname is alpaprana and its IP address is 158.0.125.45 .
158.0.125.45 alpaprana
4)The list of users are to be placed in $HOME/.rhosts file of both local and remote hosts for every user.
Eg: If the local users are divya and ajay and remote users are bhaskar and ajay.
.
Make entries as shown for respective users in local host
/home/ divya$ hostname
mahaprana
/home/ divya$ echo $HOME
/home/ divya
/home/ divya$ cat .rhosts
158.0.125.45 bhaskar
158.0.125.45 ajay
/home/ajay$echo $HOME
/home/ajay
/home/ ajay$ cat .rhosts
158.0.125.45 bhaskar
158.0.125.45 ajay
And entries shown below in the remote host.
/home/ajay$hostname
alpaprana
/home/ ajay$ cat $HOME/.rhosts
158.0.125.23 ajay
158.0.125.23 divya
/home/bhaskar/ cat $HOME/.rhosts
158.0.125.23 ajay
158.0.125.23 divya
After these changes are made, you can run the rsh command as shown in following examples.
Example 1)
Suppose you are working as a user in local host and the local and remote users are same.
/home/ ajay$ echo $LOGNAME
ajay
/home/ ajay$ rsh 158.0.125.45 “ls inter*”
interbranch.csv.gz
interschool.csv.gz
inter _college.txt
inter-dept.lst
This runs the command ‘ls inter* ‘ in the remote host 158.0.125.45 as user ajay.
By default it runs the commands within double quotes in user (ajay ) ‘s home directory defined by the $HOME variable.
Example 2)
To run the commands or process of a different remote user
/home/ divya$ rsh 158.0.125.45 -l bhaskar “ grep -i beatles Albumlist/old/Film ”
Here comes the sun – BEATLES
Beatles-Twist and shout
To run multiple commands in the remote host use semicolon to separate them.
Eg: If you want to run the above command by making your working directory as Albumlist/old
rsh 158.0.125.45 -l bhaskar “ cd Albumlist/old; grep -i beatles Film” .
Example 3)
you cannot use the environmental variables of the remote user directly inside rsh. For initializing all those variables run .profile
/home/ ajay$ rsh 158.0.125.45 -l bhaskar “ echo $ORACLE_HOME; sqlplus /”
ksh: sqlplus not found
this did not work.
/home/ ajay$ rsh 158.0.125.45 -l bhaskar “. $HOME/.profile 1> /dev/null; echo $ORACLE_HOME; sqlplus /”
/oracle/app/product
SQL>
/dev/null redirects the standard output of running .profile to /dev/null.
rm
rm
rm is used to remove files or directories. It can be used interactively to ask user input while removing multiple files or the contents of a directory.
A great care has to be taken before running rm command. There is no way to recover the files which are removed unless they were backed up on tape.
Use rm to remove these files.
Example 1)
To remove multiple files with a common keyword.
/home/jenny$ls city_DETAIL*.lst
City_DETAIL_MUM.lst
City_DETAIL_DEL.lst
City_DETAIL_BLR.lst
City_DETAIL_KOL.lst
City_DETAIL-ALL
..
To remove all these files except City_DETAIL-ALL enter
/home/jenny$rm City_DETAIL_???.lst
If the file does not belong to the current user, or if the file does not have write permissions, the
rm command asks you before removing ,i:e
rm: remove City_DETAIL_KOL.lst??.lst
If you do not want to answer to this question to all the files, use –f option to remove forcefully.
rm -f City_DETAIL_???.lst
Example 2)
To remove directories and files use –rf option
/home/jenny$rm -rf FILMS
This will remove all the sub directories of and the directory itself. This command is also same as
cd /; rm –rf home/jenny/FILMS
remember that you cannot remove a directory if it is the path of your working directory or a parent when you are inside its subdirectories.
eg: both of these don’t work.
/home/jenny /FILMS$ rm -rf /home/jenny /FILMS #Does not work
/home/jenny /FILMS/K3g$rm -rf /home/jenny /FILMS #Does not work
To view the files which are being removed when files are removed in bulk.
Suppose you have a file containing the list of files to be removed. then
/home/jenny$ head file_to_remove_list
Jj12_2.txt
Df003.log
Awer_cat.lst
Sdifre.post
Mail34.log
TAB/Hourwise_list.doc
Sequence.log
Created/Sqlldr.bad
OLD_FILE/Jenny_old.log
Hearts_jai.mkv
home/jenny$sed ‘s/^/rm -ef /g’ file_to_remove_list > file_to_remove_list.sh
home/jenny$chmod +x file_to_remove_list.sh
/home/jenny$./ file_to_remove_list.sh
removing Jj12_2.txt
removing Df003.log
removing Awer_cat.lst
removing Sdifre.post
removing Mail34.log
removing TAB/Hourwise_list.doc
removing Sequence.log
removing Created/Sqlldr.bad
removing OLD_FILE/Jenny_old.log
removing Hearts_jai.mkv
/home/jenny$
Example 4)
To remove files interactively use -i option.
/home/jenny$ rm -ir FILMS
rm: remove FILMS/ Schindler’s List ?y
rm: remove FILMS/ Chinatown? y
rm: remove FILMS/ Taxi Driver?y
rm: remove FILMS/ The Sixth Sense?n
/home/jenny$
y and n are inputs given by you to remove or not to remove resp.
rcp
rcp
rcp is used to copy or send files or directories between two hosts. It can be used an alternative to ftp and related protocols like sftp, but no password is required to send files unlike ftp.
For rcp to work, various prerequisites are required ,which are same for the rsh command.(refer rsh)
Assume your hostname is Prod-serv and IP is 192.158.0.140 and the remote hostname is
Test-serv and IP is 192.158.0.120 .The local user is kumar and remote user is warren in all of the examples.
Example 1)
To send a file from a local host to remote host.
/home/kumar$ rcp Audit_document.txt warren@192.158.0.120:
This sends the local file Audit_document.txt to the host Test-serv (you can use Test-serv in place of the IP address) and places it in the $HOME directory of warren.
With the name Audit_document.txt and owner of the file as warren.
Example 2)
To receive a file from remote host to local host.
/home/kumar$ rcp warren@192.158.0.120:global/Meal_pass2.prt .
This copies the file Meal_pass2.prt in /home/warren/global directory of 192.158.0.120
to the current directory (/home/kumar denoted here by ‘.’ or dot).
rcp just like cp changes the modification time(time stamp) of the destination file to the latest time. To retain the same time stamp use rcp –p . The above example can be written as
rcp -p warren@192.158.0.120:global/Meal_pass2.prt .
Example 3)
To copy directories between hosts use rcp with –r option.
/home/kumar$ ls -ld localdir
drwxr-xr-x 2 kumar tools 256 Jul 14 2009 localdir
/home/kumar$rcp -r localdir warren@192.15.0.120:
This copies the entire directory localdir(its sub directories) to $HOME directory of warren@192.15.0.120.
More examples:
To copy two files booked_ticket1.txt , booked_ticket2.txt to /var/docs directory of 192.15.0.120
rcp booked_ticket.txt booked_ticket2.txt warren@192.15.0.120: /var/docs
or
rcp booked_ticket?.txt warren@192.15.0.120: /var/docs
To copy the file ‘pattern-styles.tar.gz’ to /home/warren/images directory of 192.15.0.120 as style1.tar.gz with same time stamp.
rcp -p pattern-styles.tar.gz warren@192.15.0.120: images/style1.tar.gz
ps
ps
ps displays all the processes in a unix system. every command, script or an application is considered as a process in unix. All the processes run until the command or application is completed or stopped. Each process has a unique process id ( PID)
It has the same output of windows tasklist command.
Example 1)
to list only your processes (i:e all the commands , and applications run in current session )
/home/priya$ ps
PID TTY TIME CMD
217140 pts/2 0:00 -ksh
594062 pts/2 0:00 -ksh
652138 pts/2 2:03 java webfile.java
820120 pts/2 0:00 ps
Example 2)
To list all the processes running in the server page wise , use –ef option
/home/priya$ ps -ef | pg
UID PID PPID C STIME TTY TIME CMD
priya 184434 1 24 05:32:34 – 0:50 java /usr/bin/java/start-web.java
root 213146 1 0 05:32:31 – 1:21 java /usr/bin/dsmc
matt 327814 893836 0 12:28:09 – 0:00 sleep 120
root 352266 1 0 05:32:38 – 0:14 java /usr/bin/ftpd
priya 368734 1 0 05:32:42 – 0:07
priya 372912 1 0 05:32:43 – 0:10 java /usr/bin/java/start-web3.java
scripts 409806 1 0 05:32:39 – 0:13 java /usr/bin/java/start-web2.java -d final -f clear
scripts 418030 1 0 05:32:35 – 1:08 java /usr/bin/java/start-web.java -d final -f clear
scripts 430084 1 0 05:32:23 – 0:02 /usr/bin/perl -w getfile.cgi
scripts 442592 430084 0 12:17:38 – 0:00 sleep 900
scripts 454810 1 3 05:32:29 – 1:42 java /usr/bin/java/start-web.java -d final -f clear
standard input
Example 3)
To list all the processes which are run by an user,
/home/priya$ ps -fu matt
UID PID PPID C STIME TTY TIME CMD
matt 174431 1 24 05:32:34 – 0:50 wc -l
matt 723156 1 0 05:32:31 pts/2 1:21 cat items
matt 327814 893836 0 12:28:09 – 0:00 sleep 120
matt 1566 1 0 0 5:32:38 pts/1 0:14 sed ‘s/sim//’
matt 668739 1 0 05:32:42 – 0:07 /usr/bin/perl
matt 352912 1 0 05:32:43 – 0:10 java hello.java
All your processes and the processes of other users can be viewed.
In the output of the command above ,the various column heads are defined as follows.
PID
The process ID of the process.
UID
The user ID of the process owner. The login name is printed under the -f flag.
PPID
The process ID of the parent process.
C
CPU utilization of process or thread, incremented each time the system clock ticks and the process or thread is found to be running. Large values indicate a CPU intensive process and result in lower process priority whereas small values indicate an I/O intensive process and result in a more favourable priority.
STIME
The starting time of the process.
TTY
The controlling terminal for the process
if There is a ‘-‘ ,it means the process is not associated with a terminal.
Otherwise indicates The TTY number. For example, the entry pts/2 indicates terminal 2.
CMD
Contains the command name which started the process. The full command name and its parameters are displayed with the -f flag.
paste
paste
paste is an excellent command for text processing, even though its uses are limited. It can mainly be used to append the contents of files. paste is helpful in creating text files by combining
individual files such that all the lines appear on the same row in the destination file i:e linearly.
Example 1)
You have two files as shown.
/users/higgs$ cat Start_Times.log
Getname( ) at Fri Dec 30 22:29:16 IST 2011
Getip( ) at Fri Dec 30 22:30:22 IST 2011
Checkip( ) at Fri Dec 30 22:30:23 IST 2011
Getfile( ) at Fri Dec 30 22:31:22 IST 2011
/users/higgs$ cat End_Times.log
Getname( ) at Fri Dec 30 22:29:17 IST 2011
Getip( ) at Fri Dec 30 22:30:28 IST 2011
Checkip( ) at Fri Dec 30 22:30:24 IST 2011
Getfile( ) at Fri Dec 30 22:31:42IST 2011
If you have to combine these files such that the start and end times of a function appear on a common line, use paste as follows.
/users/higgs$paste Start_Times.log End_Times.log
Getname( ) at Fri Dec 30 22:29:16 IST 2011 Getname( ) at Fri Dec 30 22:29:17 IST 2011
Getip( ) at Fri Dec 30 22:30:22 IST 2011 Getip( ) at Fri Dec 30 22:30:28 IST 2011
Checkip( ) at Fri Dec 30 22:30:23 IST 2011 Checkip( ) at Fri Dec 30 22:30:24 IST 2011
Getfile( ) at Fri Dec 30 22:31:22 IST 2011 Getfile( ) at Fri Dec 30 22:31:42IST 2011
Notice that one space was added in between the two same rows of the two files .This
is default for paste without any option.
In some cases you may need to place a different delimiter between lines of the row.
/users/higgs$ paste -d ‘,’ Start_Times.log End_Times.log >Start_end_times.csv.
/users/higgs$ cat Start_end_times.csv
Getname( ) at Fri Dec 30 22:29:16 IST 2011, Getname( ) at Fri Dec 30 22:29:17 IST 2011
Getip( ) at Fri Dec 30 22:30:22 IST 2011 , Getip( ) at Fri Dec 30 22:30:28 IST 2011
Checkip( ) at Fri Dec 30 22:30:23 IST 2011 , Checkip( ) at Fri Dec 30 22:30:24 IST 2011
Getfile( ) at Fri Dec 30 22:31:22 IST 2011 ,Getfile( ) at Fri Dec 30 22:31:42IST 2011
Open this file Start_end_times.csv in windows excel if you want a better view.
Example 2)
If you want the contents to appear serially,use paste –s.
Imagine you have two files
/users/higgs$ cat Booked.log
Don2
Sherlock_homes
Mission_Impossible4
The_Dirty_picture
/users/higgs$cat Showtime.log
10:30
14:20 4 spaces after each line
17:40
22:45
/users/higgs$paste -s Booked.log Showtime.log
Don2 Sherlock_homes Mission_Impossible4 The_Dirty_picture
10:30 14:20 17:40 22:45
mv
mv
mv command moves or renames files or directories within the same file system or between different file systems. you can also do this interactively when you are moving multiple files.mv is similar to doing cut and paste in windows,
Example 1)
To rename a file or a directory.
/home/yogi$mv yog_LIST jeev_LIST
This renames the file from yog_LIST to jeev_LIST
Similarly a directory can be renamed.
Example 2)
To move a file to a directory.
/home/yogi$mv sqlnet.log oracle_logs
Here oracle_logs is a directory.if you want to save the log with a different name,enter
mv sqlnet.log oracle_logs/sqlnet_2011.log
In this case sqlnet.log is renamed to sqlnet_2011.log and saved in oracle_logs directory.
Example 3)
A file,directory or a group of files or directories can be moved to a
destination path,
/home/yogi$ ls –lrt CONSIGNMNENT*
-rw-r–r– 1 yogi Administ 58 Dec 24 00:40 CONSIGNMENT1
-rw-r–r– 1 yogi Administ 6 Dec 24 00:42 CONSIGNMENT2
-rw-r–r– 1 yogi Administ 978 Dec 25 00:59 CONSIGNMENT3
CONSIGNMNENT-OLD:
-rw-r–r– 1 yogi Administ 27 Jul 25 01:00 lst2.txt
-rw-r–r– 1 yogi Administ 35 Sep 14 09:00 goods_list
/home/yogi$ mv CONSIGNMNENT* /ALL_CONS/DOCS
This moves all the entities listed above to /ALL_CONS/DOCS directory if it exists, otherwise the command gives an error
mv: cannot rename CONSIGNMENT* to /ALL_CONS/DOCS, a file or directory in the path does not exist.
Remember that when there are more than 2 arguments to mv , the last argument is considered the destination directory. When there are only 2 arguments, if the 2nd argument contains a ‘’/” in it, then the directory structure must exist.
Example 3)
If you want mv to ask you interactively which files are to be moved, use mv –i.
/home/yogi$ mv -i Banks_Draft * DRAFTS/LIST
more
more
sometimes, while viewing huge files it would be necessary to view it such that the
contents of the page are displayed as the user examines them and wishes to go
to the next one. user can use various keys in order to go front or back of a page. Move
to few lines ahead or behind.
Example 1)
suppose you have a large file with name greenlife.txt. It contains a total of 1000 lines.
Then to open it and view it page wise enter
/home/g321$ more greenlife.txt
sustainable living
Sustainable living is a lifestyle that attempts to reduce an individual’s or soceity’s
attempt to reduce their carbon footprint by altering methods of transportation, energy
sustainability, in natural balance and respectful of humanity’s symbiotic relaivelyt
ly interrelated with the overall principles of sustainable development.
Lester R. Brown, a prominent environmentalist and founder of the Worldwatch Inst used
reuse/recycle economy with a diversified transport system.”
Contents
1 Definition
2 History
3 Shelter
3.1 List of some sustainable materials
4 Power
4.1 List of organic matter than can be burned for fuel
5 Food
5.1 Environmental impacts of industrial agriculture
5.2 Conventional food distribution and long distance transport
5.3 Local and seasonal foods
5.4 Reducing meat consumption
5.5 Organic farming
5.6 Urban gardening
5.7 Food preservation and storage
6 Transportation
7 Water
7.1 Indoor home appliances
7.1.1 Toilets
7.1.2 Showers
7.1.3 Dishwasher/Sinks
7.1.4 Washing machines
7.2 Outdoor water usage
7.2.1 Conserving Water
7.2.2 Sequestering Water
8 Waste
Sustainable living is fundamentally the application of sustainability to lifestyle
— More (3%) —
Here it waits for user input to go to the next page. The user now has the choice of doing the following few things.(many more options available)
to display the next line ,type return or “Enter” key. type f to go
front 1 line and b to go back one line.
to display the entire next page type space key.
To go 100 lines ahead type 100f and press enter
To go 10 lines behind type 10b
To go to the end of page press shift + g
To exit press q
To search a pattern, type /and enter the keyword.
Example 2)
While searching for patterns using more , if you want to ignore cases use –i option.
/home/g321$ more greenlife.txt
…
….
…… lines
/outdoor
|
|
—— >This will search for all the occurrences of the pattern ‘outdoor’ in the file, ignoring case and highlights them.
Example 3)
You can cause more command to display the specified number of lines in the window by using –n number option.
For example to view 15 lines per window ,
/home/g321$more -15 greenlife.txt
mkdir
mkdir
mkdir is used to create one or more directories. It can create entire directory structure if needed, or can give required permissions to a directory.
Example1)To simply create a directory or a directory structure.
/home/MUM-user$ mkdir channels
you further want to create directory structures. The conventional method of doing it would be
/home/MUM-user$cd channels
/home/MUM-user/ channels$mkdir STAR
/home/MUM-user/ channels$mkdir SONY
/home/MUM-user/ channels$cd SONY
/home/MUM-user/ channels/ SONY$mkdir KBC
….
…..
i:e create all the individual directories in the tree.
mkdir provides an option –p to help you to avoid these commands.
/home/MUM-user$mkdir -p channels/ STAR/NGC/Speed
/home/MUM-user$ mkdir -p channels/ SONY/ KBC
These commands will create all the intermediate parent directories and the final directory.
Example 2)
To change permissions to directories, you can use chmod command. To do it while creating them, use mkdir -p
/home/MUM-user/ $ mkdir -p -m 775 channels/discovery
/home/MUM-user/ $ls –ld channels/discovery
drwxrwxr-x 2 MUM-user Tools 256 NOV 12 22:01 channels/discovery
man
man
Man command internally invokes the sed command to modify the contents of manual pages(documentation written for the commands) and delivers a user readable output.
Examples:
Example 1:
To read documentation on sed command page-wise
/home/b3456/$ man sed | pg
Example 2:
suppose that you want to read the entire documentation of all the commands present in an unix environment and to save all of its pages in a single text file, you can run the following script from the command line to do so.
/home/b3456/$ cd /usr/bin
/usr/bin/$for i in `ls * `
>do
>man $i >>/home/b3456/man_all.txt 2>/dev/null
>done
Note: “>” symbol comes by default in all the lines
Here 2> indicates standard error.It is redirected to /dev/null(which is discarded)
Now you can take a print out of the text file you have just generated( make full use of office printer umm!!) and study all commands at home.
ls
ls
ls command is used to list the contents off the directory.ls writes to standard output
all the information of the files, directories or specifically of the parameters passed to it.
You can display these contents in the alphabetically sorted order, or list only particular files in
Combination with other command. There are many options with ls which you may hardly use in normal cases.
Here ,only few frequently used options are discussed.
Example 1)
To simply display the names of all the files in a directory.
/home/script-user$ ls
Joke-and-fun.xt
Send_script.sh
Mail-file
Rollercoaster
Jackson_note.tar
Example 2)
To display all the files that have a common pattern.
/home/script-user/items $ ls table_*.log
table_111.log
table_x.log
table_te1.log
table_adam.log
Example 3)
To get a long listing of files which contains all the details such as file size, permissions,
Time stamp, user details sorted by the modification time, use ls as follows.
/home/script-user $ ls –lrt
-rw-r–r– 1 script-user Tools 56344 Dec 16 14:37 Joke-and-fun.xt
-rw-r–r– 1 script-user Tools 105977 Dec 16 15:52 Send_script.sh
drw-r–r– 2 script-user Tools 256 Dec 18 21:01 Mail-file
-rw-r–r– 1 script-user Tools 740 Dec 20 01:03 Rollercoaster
-rw-r–r– 1 script-user Tools 58 Dec 24 00:40 Jackson_note.tar
The first field here shows file permissions, which are discussed in chmod,
2nd field is the type of element,(1 for file 2 for directory..etc)
3rd field gives the owner of the element(file,directory)
4th field gives the group which the user belongs to.
5th field gives the size.
6th,7th and 8th field-modification times.
9 th field gives the name of the element.
You can get any field of the output of ls –lrt using awk.
For eg: To get the name of the element from the output of ls –lrt, use awk as follows
/home/script-user $ ls -lrt | awk ‘{print $9}’
Joke-and-fun.xt
Send_script.sh
Mail-file
Rollercoaster
Jackson_note.tar
Example 4)
To display only the directories in a path, use the output of ls –lrt as follows.
/home/script-user $ ls -lrt | grep “^d”
drw-r–r– 2 script-user Tools 256 Dec 18 21:01 Mail-file
The ‘^’ character represents a starting character of a line( a ‘$’ represents end), in fact it’s not physically seen, but an interpreted character.
In this case grep searched for the lines starting with a ‘d’ which stands for directories.
Example 5)
you want to list the directories whose name has a common pattern, use ls –ld
home/script-user/ebooks $ls -ld j*
drwxr-xr-x 2 script-user Tools 256 Nov 9 12:10 j2me
drwxr-xr-x 2 script-user Tools 20480 Nov 9 12:10 java
drwxr-xr-x 2 script-user Tools 4096 Nov 9 12:10 java scripts
Suppose you want to view all the files of the directory java use ls –lrt java instead of doing both cd java and ls –lrt.
Example 6)
To make a case insensitive list on a filename parameter, use ls along with grep as follows.
home/script-user/docs$ ls report*
report1.doc
report_eng.doc
reportslist.txt
home/script-user/docs$ls | grep –i report
report1.doc
report_eng.doc
reportslist.txt
Reports_Bulk.wrt
Business_reports.doc
The above command is equivalent to ls *report* *Report* *REPORT* *Report*………..and so on till all the combination of the word report in Ucase and Lcase.
Example 7)
To list all the directories in a directory, and its subdirectories, use ls –R
In this case ls lists all the files inside the directories starting with j recursively in all its subdirectories.
home/script-user/ebooks $ ls -R j*
j2me:
j2me – the complete reference.pdf j2me—the-complete-reference.htm
java:<---------------------------- directory
java The J2EE Tutorial (Addison Wesley & Sun, 2002).pdf
(Addison Wesley 2002) - Java Network Programming and Distributed Computing.pdf
(Ebook) Java - Borland Jbuilder - Developing Database Applications - Inprise.pdf
(OReilly) Java Network Programming ,2nd Ed.pdf
(OReilly) Java Server Pages ,2nd Ed.pdf
java scripts:
[Java Script] O'Reilly- JavaScript The Definitive Guide.pdf
java scripts/newdir:<------------- subdirectory
[Java Script] O'Reilly - JavaScript The Definitive Guide 2ed.pdf
….
….
…
^C
Example 8)
ls by default does not show hidden files and directories. To show hidden files use –a option.
Eg: files like .profile, .sh_history etc are hidden files. if have not used –a option you wont be
able to see these files.
In all the directories , there are 2 common hidden directories .(a dot) which represents the current directory and ..( 2 dots) represents the parent of the current directory
kill
kill
kill is used to send a signal to the process, though the name suggests that it is used only to terminate a process. But, kill helps to terminate a process more often than not. When some process is run unexpectedly or accidentally, it might have to be aborted ,for which kill helps just like CTRL+ALT+DEL in windows .
Example 1)
ps command gives you the process id s and other details of various commands, if you want to stop any command, just check the corresponding PID of the process(the command) which you want to terminate ,then use kill to stop that process.
Consider the output of ps,
/home/b6789$ps -fu b6789
UID PID PPID C STIME TTY TIME CMD
b6789 352266 1 0 05:32:38 - 0:14 java /usr/bin/wc –l
b6789 368734 1 0 05:32:42 - 0:07 [wc]
------b6789 372912 120 0 05:32:43 - 5:10 java /usr/bin/java/dev2.java
b6789 409806 1 0 05:32:39 - 0:13 java /usr/bin/java/start-atm.java
this process is consuming cpu and has to be terminated,
/home/b6789$kill 372912
This sends the signal SIGTERM to the process 372912, and terminates it.
Sometimes the process may refuse to terminate just by SIGTERM, in that case use SIGKILL signal(kill -9)
/home/b6789$kill -9 372912
This signal cannot be ignored . You can kill any of the process initiated by you, i:e from your user id this way .A root user has the privilege of killing any user’s processes .
Example 2)
You can also get the specific PID for your command without having to search it using grep and awk along with ps. Then kill the PID which you have retrieved.
If you had to kill the CMD start-atm.java above,
/home/b6789$j_pid=`ps –fu $LOGNAME | grep start-atm.java | grep –v grep | awk ’{print $2}’`
/home/b6789$echo $ j_pid
409806
/home/b6789$kill -9 $ j_pid
You can also use xargs to kill process of a particular type all at once .
Example 3)
There is a method to kill all the processes initiated by you,
/home/b6789$kill -9 -1
This will kill all the processes of your user id and also closes your login session. It is a very dangerous command ,so think thrice and check whether any required process is still running using ps before you press enter.
You can send many kinds of signals to a process(not all) by using kill - PID. The various signal numbers and their names are listed below (for AIX) in detail .A short list can also be viewed
by using kill -l (lowercase alphabet L).
Name num Description
SIGHUP 1 hangup, generated when terminal disconnects
SIGINT 2 interrupt, generated from terminal special char
SIGQUIT 3 (*) quit, generated from terminal special char
SIGILL 4 (*) illegal instruction (not reset when caught)
SIGTRAP 5 (*) trace trap (not reset when caught)
SIGABRT 6 (*) abort process
SIGEMT 7 EMT instruction
SIGFPE 8 (*) floating point exception
SIGKILL 9 kill (cannot be caught or ignored)
SIGBUS 10 (*) bus error (specification exception)
SIGSEGV 11 (*) segmentation violation
SIGSYS 12 (*) bad argument to system call
SIGPIPE 13 write on a pipe with no one to read it
SIGALRM 14 alarm clock timeout
SIGTER M 15 software termination signal
SIGURG 16 (+) urgent condition on I/O channel
SIGSTOP 17 (@) stop (cannot be caught or ignored)
SIGTSTP 18 (@) interactive stop
SIGCONT 19 (!) continue (cannot be caught or ignored)
SIGCHLD 20 (+) sent to parent on child stop or exit
SIGTTIN 21 (@) background read attempted from control terminal
SIGTTOU 22 (@) background write attempted to control terminal
SIGIO 23 (+) I/O possible, or completed
SIGXCPU 24 cpu time limit exceeded
SIGXFSZ 25 file size limit exceeded
SIGMSG 27 input data is in the ring buffer
SIGWINCH 28 (+) window size changed
SIGPWR 29 (+) power-fail restart
SIGUSR1 30 user defined signal 1
SIGUSR2 31 user defined signal 2
SIGPROF 32 profiling time alarm
SIGDANGER 33 system crash imminent; free up some page space
SIGVTALRM 34 virtual time alarm
SIGMIGRATE 35 migrate process
SIGPRE 36 programming exception
SIGVIRT 37 AIX virtual time alarm
SIGALRM1 38 m:n condition variables – RESERVED – DON’T USE
SIGTALRM 38 per-thread alarm clock
SIGWAITING 39 m:n scheduling – RESERVED – DON’T USE
SIGRECONFIG 58 Reserved for Dynamic Reconfiguration Operations
SIGCPUFAIL 59 Predictive De-configuration of Processors – (RESERVED – DON’T USE )
SIGKAP 60 keep alive poll from native keyboard
SIGGRANT SIGKAP monitor mode granted
SIGRETRACT 61 monitor mode should be relinquished
SIGSOUND 62 sound control has completed
SIGSAK 63 secure attention key
Example 4)
You can stop a running process in the midway and then continue it from that point using kill command.
Assume you are running a ftp command and want to stop the file transfer.
/home/b6789$ ftp 10.42.67.125
……
…….
put abigfile.txt
#####################################
Now in between if you want to suspend it. take an other session for the same user and run
kill -17 ,
where is the PID of the ftp command. This sends the signal SIGSTOP to the command. you can observe that the file transfer is stopped.
To restart the command,simply run
Kill -19 , which sends the signal SIGCONT which is to continue a process. The file transfer starts from the point where it stopped.
Similarly other commands ,jobs or utilities can be stopped and continued using kill -17 and kill -19 resp.
Example 5)
To Kill all the child processes of a particular process use
kill -9 –
to kill all child processes of PID 204567
kill -9 -204567
head
head
head command is used to view or save a specified portion of the beginning of a file.
It does the same work as tail does for the end of the file.
Example 1)
Head by default displays first 10 lines of a file when no option is specified.
/home/ftpusr$ head blackwater.ns
Blackwater USA was formed in 1997, by Erik Prince in North Carolina, to provide training support to military and law enforcement organizations. In explaining the Blackwater’s purpose, Prince stated that ‘‘We are trying to do for the national security apparatus what FedEx did for the Postal Service.’’After serving SEAL and SWAT teams, Blackwater USA received their first government contract after the bombing of the USS Cole off of the coast of Yemen in October of 2000. After winning the bid on the contract, Blackwater was able to train over 100,000 sailors safely.Prince purchased about 7,000 acres (28 km2) (from Dow Jones Executive, Sean Trotter) of the Great Dismal Swamp, a vast swamp on the North Carolina/Virginia border, now mostly a National Wildlife Refuge. “We needed 3,000 acres to make it safe,” Prince told reporter Robert Young Pelton.There, he created his state-of-the-art private training facility and his contracting company, Blackwater, which he named for the peat-colored water of the swamp. The The Blackwater Lodge and Training Center officially opened on May 15, 1998 with a 6,000 acre facility and cost $6.5 million.
If you wish to view only first two lines of the file, use
/home/ftpusr$ head -2 blackwater.ns
Blackwater USA was formed in 1997, by Erik Prince in North Carolina, to provide training support to military and law enforcement organizations. In explaining the Blackwater’s purpose, Prince stated that ‘‘We are trying to do for the
Example 2)
when you have a very large file(say of 100000 lines) and it is difficult to open such a file in vi, but you are concerned only with first few lines of the file.
In that case redirect the first few lines of that huge file into another file and open that file in vi.
/home/ftpusr$ head -1000 migratory_note > migratory_note_1000
/home/ftpusr$vi migratory_note_1000
..
..
Example 3)
In scripts, there are times where you initialize the output of grep to a variable but grep searches and returns multiple values, you can use only the first returned value by head -1,
Confused..?the example below shows how.
/home/ftpusr$
/home/ftpusr$b=`grep 7 prime_numbers`
/home/ftpusr$echo $b
7
17
37
Suppose you want the first occurring prime number containing 7,
Use head as follows.
/home/ftpusr$b=` grep 7 prime_numbers | head -1`
/home/ftpusr$echo $b
7
Example 4)
To print the nth line of a file, use head and tail in combination as follows.
/home/ftpusr$ head -100 operating-out.sh | tail -1
This prints 100th line of the file.
gzip and tar
gzip and tar
gzip is used to compress files and tar is used to archive a file system. Though these two are different commands ,using them together can accomplish a variety of tasks which seem to be
much simpler than when running them separately. Here we will also discuss about the combined use of tar and cd commands.
First we will discuss with examples the individual command usages of gzip and tar and later use them in combination.
gzip
Example 1)
gzip is a compression utility that uses Huffman coding to compress files and save them in .gz format . The .gz file occupies lesser space than the original file. We will not discuss about the .gz file format , but only show how the command can be used along with its options.
To compress a file stake-holder.csv ,
/home/gr-adm $ ls –lrt stake-holder.csv*
-rw-r–r– 1 gr-adm Administ 56344 Dec 16 14:37 greenlife.txt
/home/gr-adm$ gzip file stake-holder.csv
/home/gr-adm$ l s -lrt stake-holder.csv*
-rw-r–r– 1 kaushik Administ 20758 Dec 16 14:37 greenlife.txt.gz
Example 2)
If you want to retain the original file even after compression or want the .gz file to have a different name, use gzip –c.
To compress the file stake-holder.csv and save the .gz file in some other directory,
/home/gr-adm$ gzip -c stake-holder.csv >/backup/stakes/ stake-holder.csv.gz.
You can also append the compressed file into an existing .gz file.
gzip -c stake-holder.csv >>/backup/stakes/Old-stake-holder.csv.gz
Here , Old-stake-holder.csv.gz can be a pre existing .gz file or a new file with that name will be created.
Example 3)
To decompress the .gz file use gunzip ,
/home/gr-adm$ gunzip /backup/stakes/stake-holder.csv.gz
This command will decompress the file stake-holder.csv.gz and places the original file
stake-holder.csv in /backup/stakes/ itself.
Example 4)
To view the contents of the file without completely decompressing the file , use gzip –dc.
Use pg command to view it pagewise..
/home/gr-adm$ gzip -dc stake-holder.csv.gz | pg
A premji, wipro,67
N R murthy,Infosys,65
N Chandra,TCS,46
…
..
You can redirect the output into another file,
For eg to redirect first 5 lines of the file stake-holder.csv.gz ,
gzip –dc stake-holder.csv.gz | head -5 >stake-holder-five.csv.
tar.
tar is a short form for tape archive. It can create a single file with the contents of a directory structure in one path or extract a file into directory structure in another path.
Example1)
To create a tar file of a directory structure and to view the progress of tar use,
/home/gr-adm$ tar -cvf /archives/archive2/data-history.tar data/notes
a data/notes/Nov-10/exceptions.txt 200 bytes
a data/notes/Nov-12/promotions.csv 123 bytes
a data/notes/N0v-13/oracle-log/ora-errors.log 456 bytes
a data/notes/tea-time.in 2076 bytes
a data/notes/network/virtual-ips.bmp 1290bytes
a data/notes/levy.list 87bytes
/home/gr-adm $
This creates an archive file data-history.tar in the path /archives/archive2 which contains the directory /home/notes and all its files and directory structure.
Example2)
To extract the contents of the archive file into a different location , first change your working directory to the location where you wish to extract the file, then extract and view the progress by running tar with –xvf switch.
/home/gr-adm$ cd /base-app/archive
/home/gr-adm$ tar -xvf /archives/archive2/data-history.tar
x data/notes/Nov-10/exceptions.txt 200 bytes
x data/notes/Nov-12/promotions.csv 123 bytes
x data/notes/N0v-13/oracle-log/ora-errors.log 456 bytes
x data/notes/tea-time.in
x data/notes/network/virtual-ips.bmp
x data/notes/levy.list
/home/gr-adm $
Example3)
Instead of giving entire directory name as a parameter to create a tar file, a list of files which care to be archived can be can be passed to tar with –L option.
A file containing the list of files to take tar is as shown.
/home/gr-adm$ cat to-tar-list
data/notes/Nov-10/exceptions.txt
data/notes/N0v-13/oracle-log/ora-errors.log
data/notes/levy.list
Now to create an archive of only these specific files ,use tar in the following manner.
/home/gr-adm$ tar -cvf /archives/archive2/data-specific.tar -L to-tar-list
a data/notes/Nov-10/exceptions.txt 200 bytes
a data/notes/N0v-13/oracle-log/ora-errors.log 456 bytes
a data/notes/levy.list 87 bytes
Example4)
Sometimes you may need to exclude files from the tar archive. To do this place the files and directories to be excluded in a list file and then use tar –X .
/home/gr-adm$ cat /home/fiuser/excl-list
data/notes/Nov-10/exceptions.txt
data/notes/Nov-12/promotions.csv
data/notes/N0v-13/oracle-log/ora-errors.log
/home/gr-adm$ tar -X /home/fiuser/excl-list -cvf /archives/data-specific2 .tar data/notes
a data/notes/tea-time.in 2076 bytes
a data/notes/network/virtual-ips.bmp 1290bytes
a data/notes/levy.list 87bytes
Example5)
The contents of the tar file cannot be viewed directly by cat or more , but to view the details of the files and directories contained in the tar file ,use tar –tvf
/home/gr-adm$ tar -tvf /archives/data-specific2 .tar
-rw-r–r– gr-adm 2076 2011-12-16 15:52:55 data/notes/tea-time.in
-rw-r–r– gr-adm 1290 2011-12-30 22:44:59 data/notes/network/virtual-ips.bmp
-rw-r–r– gr-adm 87 2011-12-30 22:45:48 data/notes/levy.list
Now that we have gone through tar and gzip individually in detail, let us discuss with examples the use of combined forces of tar and cd command.
· to take the tar of data/notes directory and untar the contents in /archives/archive2
tar cvf – data/notes | (cd /archives/archive2;tar xvf – )
‘ –‘ refers to the console .so it means you are passing the output of tar -cvf to tar –xvf and changing your path to /archives/archive2.
· to take the tar of local directory data/notes and untar the contents in a directory /archives/archive2 of remote machine 10.198.150.45
tar cvf – data/notes | rsh 10.198.150.45 -l remuser “cd /archives/archive2;tar xvf -;”
· to take the tar of files in list file tar_list_1 and untar in /archives/archive2
tar cvf – `cat tar_list_1` | (cd /archives/archive2;tar xvf – )
· to take the tar of files in a local list file tar_list_1 and untar in /archives/archive2 of a remote machine 10.198.150.45
tar cvf – `cat tar_list_1` | rsh 10.198.150.45 – l remuser “cd /archives/archive2;tar xvf -”
· to take the tar of data/notes directory excluding those in excl_list_tar and untar its contents in /archives/archive2
tar –X excl_list_tar cvf – data/notes | (cd /archives/archive2;tar xvf -)
Finally we will discuss with examples the combined use of gzip and tar.
· to gunzip a zipped tar file data_archive.tar.gz and untar It in /archives/archive2 .
gzip -dc data_archive.tar.gz – | (cd /archives/archive2;tar xvf -)
· to gunzip a local zipped tar file data_archive.tar.gz and untar it in /archives/archive2 of a remote machine 10.198.150.45
gzip -dc data_archive.tar.gz – | rsh 10.198.150.45 -l fnsonlad “cd /archives/archive2;tar xvf –“
· To view the contents of a zipped tar file without unzipping it, page wise
gzip -dc data_archive.tar.gz – | tar tvf – |pg
find
find
find command helps you to find files in a directory structure much faster than ls command for
various options. It can widely be used for getting the list of files in a directory and its sub directories
with a given condition on file time stamp, size, name and so on.
Example 1) Suppose that you want to know the entire directory structure in a
Particular path, then use the find command as follows.
/home/Varsha $ find . –type d
./ebooks/perl
./ebooks/perl/1.Strings
./ebooks/perl/10.Subroutines
./ebooks/perl/11.Ref and REcs
./ebooks/perl/12.Packages, Libraries, and Modules
./ebooks/perl/13.Classes, Objects, and Ties
./ebooks/perl/14.Database Access
./songs/hindi/songs collection/15 jab we met
./songs/hindi/songs collection/1942 A Love Story
./songs/hindi/songs collection/20 hey baby
./songs/hindi/songs collection/22 jhoom barabar jhoom
././songs/hindi/songs collection/26 let the music play 3++
./songs/hindi/songs collection/Aa Dekhe Zara
./songs/hindi/songs collection/Aap Kaa Suroor
./songs/hindi/songs collection/Aashiq Banaya Aap Ne
./songs/hindi/songs collection/Rockstar
./songs/hindi/songs collection/3 idiots
.
.
.
.
^C
It seems varsha listens to these songs while her boss is not around .
This command searched all the directories starting from /home/Varsha and displayed it.
Similarly for displaying all the files you can use -type f option. Here the ‘.’ In the command refers to the current directory.
You can use any directory name as the base directory for find to search
To display everything(i:e all
Directories and files) use.
find . – print
Example 2)
You want the output to be similar to that of ls command, use find as follows.
/home/Varsha $ find – type f -ls
244269 33 -rw-r–r– 1 Varsha Trainee 60 Nov 9 23:58 ./ebooks/perl/1.Strings/chapter1
235915 86 -rw-r–r– 1 Varsha Trainee 215 Nov 9 11:26 ./ebooks/perl/1.Strings/chapter2
104841 25 -rw-r–r– 1 Varsha Trainee 554 Nov 9 11:26 ./ebooks/perl/1.Strings/chapter3
104838 32 -rw-r–r– 1 Varsha Trainee 510 Nov 9 11:26 ./ebooks/perl/10.Subroutines/chapter1
104840 33 -rw-r–r– 1 Varsha Trainee 55 Nov 9 11:26 ./ebooks/perl/10.Subroutines/chapter2
..
^C
The output you get are the values included in the following order
I-node number
Size in kilobytes (1024 bytes)
Protection mode
Number of hard links
User
Group
Size in bytes
Modification time
Example 3)
Find also helps in retrieving the files which are a modified n days before or modified after n days. Where n is any integer.
Suppose you want the list of all the files which are modified 5 days ago so that it can be removed after archiving It.
/home/varsha $ date
Fri Dec 23 21:16:06 IST 2011
/home/varsha $ find . –type f -mtime +5 >file_list_5
Then use this file list(file_list_5) to take a tar of the file, or to remove those files and save this file as a copy of ‘which files have been archived’.
Similarly you can use -5 instead of +5 for files modified after 5 days.
Example 4)
to find all the find all the files which are greater or less than a particular size, use –size option.
Suppose you want the list of files which are of size greater than 100kB,use find as follows.
/home/varsha $ find . –size +100c
Example 5)
To find all files which are created after the modification time of a particular file,use find with -newer option.
If you want to list all the files that were created after 2013-05-31 06:25
/home/varsha/$ touch -t 201305310625 test_file
home/varsha/$ ls -lrt test_file
244269 33 -rw-r–r– 1 Varsha Trainee 0 May 31 06:25
home/varsha/$ find . -newer test_file
./BATCH
./BATCH/arguments.bat
./BATCH/arith.bat
./BATCH/for_loop.bat
./BATCH/if_condn.bat
./BATCH/if_condn.bat.bak
./misc
./misc/vim_list.txt
./PERL
./PERL/a1.txt
./PERL/a1.txt.orig
./PERL/a2.txt
./PERL/a2.txt.orig
./PERL/all_dir.txt
You can verify it by using -ls option.
echo
echo
echo is used to display characters into standard output. It can also be used to write to a file
or just to print a newline character.
Example1)
It can be used to print a string and a variable together.
/home/b4578$ echo “ my Home is $HOME”
My Home is /home/b4578
/home/b4578$
To print the $symbol use a backslash “\” character.
ie:-
/home/b4578$ echo “my \$HOME variable is set to $HOME”
my \$HOME variable is set to /home/b4578
Example 2)
To display double quote character “ use a backslash.
Suppose you want to display a html form tag. use echo as shown in a script.
#!/usr/bin/sh
#……….lines of code
echo “”
echo “
”
echo “
”
echo “”
echo “”
when you run the script the output will be
echo “”
echo “
”
echo “
”
echo “”
echo “”
Example 3)
to just display a newline in the output of a script, simply place an echo.
Example 4)
To write the output of echo command to a file use “>” to redirect the output of echo.
/home/b4578$echo “ my logpath is $LOGPATH” >>file_log.txt
/home/b4578$tail -1 file_log.txt
my logpath is /home/b4578/user/logs
Example 5)
To give an audible alert(a bell) use \a inside echo
For example to make an alarm script which beeps continuously when there is any error, the echo command can be used as shown.
/home/b4578$vi check _transactions.sh
….
alarmfunc( )
{
While [ 1 ] # always condition
do
echo “\a There is some problem” #beep continuously with a gap of one second.
sleep 1
done
}
….
…
If [ $tran_time_before -gt 20 ]
alarmfunc
fi
….
…
/home/b4578$
Escape sequences which are recognised by echo are listed below.
\a Displays an alert character.
\b Displays a backspace character.
\c Suppresses the new-line character that otherwise follows the final argument in the output.
All characters following the \c sequence are ignored.
\f Displays a form-feed character.
\n Displays a new-line character.
\r Displays a carriage return character.
\t Displays a tab character.
\v Displays a vertical tab character.
\\ Displays a backslash character.
date
date
date can be used by a lot of applications in various forms. you may require to access a month,day, hour or second of a particular date.
You may use the following options to achieve these functionalities.
%% a literal %
%a locale’s abbreviated weekday name (Sun..Sat)
%A locale’s full weekday name, variable length (Sunday..Saturday)
%b locale’s abbreviated month name (Jan..Dec)
%B locale’s full month name, variable length (January..December)
%c locale’s date and time (Sat Nov 04 12:02:33 EST 1989)
%d day of month (01..31)
%D date (mm/dd/yy)
%e day of month, blank padded ( 1..31)
%h same as %b
%H hour (00..23)
%I hour (01..12)
%j day of year (001..366)
%k hour ( 0..23)
%l hour ( 1..12)
%m month (01..12)
%M minute (00..59)
%n a newline
%p locale’s AM or PM
%r time, 12-hour (hh:mm:ss [AP]M)
%s seconds since 00:00:00, Jan 1, 1970 (a GNU extension)
%S second (00..60)
%t a horizontal tab
%T time, 24-hour (hh:mm:ss)
%U week number of year with Sunday as first day of week (00..53)
%V week number of year with Monday as first day of week (01..52)
%w day of week (0..6); 0 represents Sunday
%W week number of year with Monday as first day of week (00..53)
%x locale’s date representation (mm/dd/yy)
%X locale’s time representation (%H:%M:%S)
%y last two digits of year (00..99)
%Y year (1970…)
%z RFC-822 style numeric timezone (-0500) (a nonstandard extension)
%Z time zone (e.g., EDT), or nothing if no time zone is determinable
To get the the current month,year and day in mmddyyyy format ,enter
date + ‘ %m%d%Y’
cp
cp
cp is used to copy files with a name to a different name in the same path or to a different path.It can also be used to recursively copy files in a directory structure to a different directory.
Example1)
To copy file1 to file2
/home/justin$cp sl_ind.ctl sql_ind2.ctl
This copies the contents of the file sl_ind.ctl to sql_ind2.ctl . If there is already a file named sql_ind2.ctl in the current directory, then the file will be over written ,if it does not exist, the new file will be created.
If you do not want files to be overwritten , use –i option. give y or n as reply.
assume sql_ind2.ctl already exists.
/home/justin$ cp -i sql_ind.ctl sql_ind2.ctl
This will ask you to whether to overwrite or not.
You can also use wildcards like ? and * along with parameters that define a group of source filenames.
Example 2)
when you copy files from source to a different directory ,you do not have to mention the
filename in the destination path if you want to keep the name of the file same. Do it
only when destination filename is to be different.
Suppose /home/raja/libs is a directory,
/home/justin$ cp Jlibs.all.csv /home/raja/libs # this will place Jlibs.all.csv in /home/raja/libs.
/home/justin$ cp Jlibs.all.csv /home/raja/libs/rlibs.all.csv #This will
Create rlibs.all.csv in /home/raja/libs directory
Example 3)
cp has option –p for preserving the file attributes like modification time,without this option, cp by default puts new time stamp for destination files.
/home/justin$ls -lrt flower?.jpg
-rw-r–r– 1 justin Admin 59254 Jan 03 12:14 flower3.jpg
-rw-r–r– 1 justin Admin 59256 Dec 29 17:22 flower2.jpg
/home/justin$cp -p flower3.jpg flower2.jpg
/home/justin$ls -lrt flower?.jpg
-rw-r–r– 1 justin Admin 59254 Jan 03 12:14 flower2.jpg
-rw-r–r– 1 justin Admin 59254 Jan 03 12:14 flower3.jpg
Example 4)
you want to completely copy a file system with all the files and directories in it and create
the same directory structure, use cp with –R option.
/home$ cp -R justin /backup_justin
/home$cd /backup_justin
/backup_justin$find . –type f
Justin/ flower3.jpg
Justin/flower2.jpg
Justin/ sql_ind.ctl
Justin/sql_ind2.ctl
Justin/wildlife/tiger/video2.mpg
Justin/ wildlife/tiger/video3.mpg
Justin/ client-file/voucher1.txt
chmod
chmod
Using chmod you can change the permission of single or multiple directories or files.
In unix, nothing works without proper permissions. To run any script ,the first thing you need to
do after writing it is to give execute permissions .
To view file permissions. use ls –lrt
Example 1)
/home/d456/$ ls –lrt
-rw-r–r– 1 d456 Readers 1181170 Apr 27 2011 apache_manual.tar.gz
-rw-r–r– 1 d456 Readers 1128480 Apr 27 2011 perl cookbook.zip
-rw-r–r– 1 d456 Readers 70228 May 3 2011 IMP.txt
All the files have read and write permissions to the user read to the group and others.
-rw-r–r– can be interpreted in octal notation as 110-100-100 or 644
To change the permissions of the file IMP. txt with read and write to user and
group, only read to others (rw-rw–r–) ,you can use:
D2-/home/d456/$ chmod 664 IMP.txt
D2-/home/d456/$ ls –lrt
rw-r–r– 1 d456 Readers 1181170 Apr 27 2011 apache_manual.tar.gz
-rw-r–r– 1 d456 Readers 1128480 Apr 27 2011 perl cookbook.zip
-rw-rw-r– 1 d456 Readers 70228 May 3 2011 IMP.txt
-rw-r–r– 1 d456 Readers 5748626 May 12 2011 sg245511.pdf
Example 2)
you want to allow all users to run the script named script_name.sh
/home/d476/$ chmod a+x script_name.sh
/home/d476/$ ls –lrt script_name.sh
-rwx -rx–rx– 1 d476 Readers 5748626 Jun 10 2011 script_name.sh
Here a in ‘a+x’ represents all.similarly use ‘u’ is for user, ‘g ‘ for group , ‘o’’ for others ‘+’ to give permissions and ‘-‘ to deny.
Example 3)
You may have to give permissions to all the files in a directory tree use –R option.
To give all files and directories read ,write and execute permissions in /home directory ,
D2- /home $ chmod -R 777 *
chmod: /home/ d476/relativity.txt operation not permitted
chmod : /home/ d476/ source3.sh operation not permitted
You will definitely get these errors if you are not an admin user or logged in as a different user.
cd
cd
cd command helps you to move around directories. various options of cd helps you
in navigating between your current directory to previous working directory, or one directory
behind and so on..
cd in unix provides much more options than the cd in DOS. you will see in these examples.
1) You are working in a directory other than your home and want to go to your home directory($HOME), just enter cd.
/home/dimuser/functions/standard/cpp$ echo $HOME
/home/b768
/home/dimuser/functions/standard/cpp$ cd
/home/b768 $
‘cd ‘ is same as ‘cd $HOME’
2) You need to go to your last working directory.
/home/b768$ cd /var/utilities
/var/utilities$ cd –
/home/b768 #here it echoes the directory into which you are moving
/home/b768 $cd –
/var/utilities
/var/utilities$
3) To move one directory behind use ‘ ..’ ,ie two dots
/home/b768$ cd ..
/home$
Note that ‘.’ Refers to the current directory and ‘..’ refers to a directory behind
4) To interchange similar directory structures as your current working directory.
/home/b768/test/ALG/bin $cd test prod
/home/b768/prod/ALG/bin
/home/b768/prod/ALG/bin$
Ensure here that the destination directory structure exists.
You can also simply change any particular string or a number in a directory name to a new one
and move to the new directory path if it exists.
ie :-
/home/b768/prod/ALG/bin$cd 68 54
/home/b768/prod/ALG/bin
/home/b754/prod/ALG/bin$
cat
cat
cat can be used to open one or a group of files , append files into one file. one of the special functionality is to read an entire file line by line along with the read command. The examples will show us how.
Example 1)
/home/b3456/$ cat branches1.txt (Note: here /home/b3456/$ is the prompt
35467 vdn /home/vd2
46788 ghi /home/gh1/gh
89078 bjk /home/vd2
/home/b3456/$ cat branches2.txt
56890 lod /home/lod/inter
33456 bhj /home/bhind
45790 krk /myhome/krk
Combining these two files into a single file can be done by
/home/b3456/$ cat branches1.txt cat branches2.txt
35467 vdn /home/vd2
46788 ghi /home/gh1/gh
89078 bjk /home/vd2
56890 lod /home/lod/inter
33456 bhj /home/bhind
45790 krk /myhome/krk
Or by
/home/b3456/$ cat branches[1-2].txt
Suppose you have many such files named branches1.txt branches2.txt branches3.txt… branches.txt where is any single numeric or alphabetic character, you can use.
cat branches?.txt to open all the files.
Example 2)
You want to write contents you have copied from a file into another file. one way to do it is to open it in a vi editor and then save it.
Other way to do it is by using cat to open a file and paste the contents.
/home/b3456/$ cat >filetocopy
#the sentence of these lines
were copied from a different file
and pasted here.
#the contents here were typed manually
^D
/home/b3456/$
/home/b3456/$more filetocopy
#the sentence of these lines
were copied from a different file
and pasted here.
#the contents here were typed manually
/home/b3456/$
A ^d character(ctrl + d) must be typed at the end to mark the end of file.
If you already have a file with the same name and you want to append the contents copied into that file,
then use “>>” instead of ‘>’
Example 3)
you can also use cat to read each line of a file into a variable, using read command.
This cannot be achieved with a ‘for in ’ loop since it considers words in a line as separate values assigned to the index variable.
The following commands will tell you how to achieve it using cat and read with a while loop.
Consider that you want to read the file branches1.txt(example 1) ,in such a way that entire string in each line is stored into a variable and then that variable is used to extract each fields(separated by spaces)
/home/b3456/$ cat branches1.txt | while read line # the variable line stores the contents of a line
do
num=`echo $line | awk ‘{print $1}’
path=` echo $line | awk ‘{print $3}’
echo “$path is the path for $num”
done
/home/vd2 is the path for 35467
/home/gh1/gh is the path for 46788
/home/vd2 is the path for 89078
The ‘for in’ loop imposes restrictions on the number of lines of a file which can be by it. But cat has no limitations. It can read any file, however large it is until the system resource is exhausted.
Example 4)
line numbers of a file can be displayed by using cat with –n option.
/home/b3456/$ cat -n branches1.txt cat branches2.txt
1 35467 vdn /home/vd2
2 46788 ghi /home/gh1/gh
3 89078 bjk /home/vd2
4 56890 lod /home/lod/inter
5 33456 bhj /home/bhind
6 45790 krk /myhome/krk
cal
cal
cal command is basically used to view the calendar.When you are working in unix it becomes necessary to have a quick look at the calendar, and you need not minimize your terminal session to view it.jut use cal as follows.
Ex1) To view the entire calendar of a year.
/home/steve $ cal 2011
January February
Sun Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Sat
1 1 2 3 4 5
2 3 4 5 6 7 8 6 7 8 9 10 11 12
9 10 11 12 13 14 15 13 14 15 16 17 18 19
16 17 18 19 20 21 22 20 21 22 23 24 25 26
23 24 25 26 27 28 29 27 28
30 31
March April
Sun Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Sat
1 2 3 4 5 1 2
6 7 8 9 10 11 12 3 4 5 6 7 8 9
13 14 15 16 17 18 19 10 11 12 13 14 15 16
20 21 22 23 24 25 26 17 18 19 20 21 22 23
27 28 29 30 31 24 25 26 27 28 29 30
May June
Sun Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Sat
1 2 3 4 5 6 7 1 2 3 4
8 9 10 11 12 13 14 5 6 7 8 9 10 11
15 16 17 18 19 20 21 12 13 14 15 16 17 18
22 23 24 25 26 27 28 19 20 21 22 23 24 25
29 30 31 26 27 28 29 30
July August
Sun Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Sat
1 2 1 2 3 4 5 6
3 4 5 6 7 8 9 7 8 9 10 11 12 13
10 11 12 13 14 15 16 14 15 16 17 18 19 20
17 18 19 20 21 22 23 21 22 23 24 25 26 27
24 25 26 27 28 29 30 28 29 30 31
31
September October
Sun Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Sat
1 2 3 1
4 5 6 7 8 9 10 2 3 4 5 6 7 8
11 12 13 14 15 16 17 9 10 11 12 13 14 15
18 19 20 21 22 23 24 16 17 18 19 20 21 22
25 26 27 28 29 30 23 24 25 26 27 28 29
30 31
November December
Sun Mon Tue Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Sat
1 2 3 4 5 1 2 3
6 7 8 9 10 11 12 4 5 6 7 8 9 10
13 14 15 16 17 18 19 11 12 13 14 15 16 17
20 21 22 23 24 25 26 18 19 20 21 22 23 24
27 28 29 30 25 26 27 28 29 30 31
Ex 2) To view the calendar of any month of a given year, use cal with month as first parameter, year as second.
/home/steve $ cal 8 1947
August 1947
Sun Mon Tue Wed Thu Fri Sat
1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31
bc
bc
bc is an important utility provided by unix to do mathematical calculations. This utility can be used as a one line command. you can also program bc to perform various arithmetic just like that of a C program. It also has a Math library of its own which has functions like sine, cosine, tan and so on, which can be defined by specifying –l option.
For more details on bc you can refer to these url s on bc.
http://www.gnu.org/software/bc/manual/html_mono/bc.html
http://www.thegeekstuff.com/2009/11/unix-bc-command-line-calculator-in-batch-mode/
Example1)
to perform a simple arithmetic 2 + 2,you can bc as follows.
/home/MS$ bc
2+2——————>type your expression here
4 ———————->Ans ,
Example2)
To run a command containing an expression and evaluate it using bc,
/home/MS$ echo “ 35.2 – 43.6” | bc
-8.4
Example3)
To use decimal digits you need to specify scale variable.
/home/MS$ bc
6/9
0
Scale=4
6/9
.6666
basename
basename
basename is used to get the name of the file given its complete path including the file.
It is also helpful to access the name of a script within it.
Ex ample1)
To get the name of a file from its complete path.
/home/gj876$ echo $CGI_PATH
/utilities/intranet/apache/cgi-docs/cowboy
/home/gj876$ basename $CGI_PATH
cowboy
Example2)
Suppose a script “/home/scripts/linkfiles.sh” is called from another script with its complete path,then to access the base filename of “/home/scripts/linkfiles.sh”, use
#!/usr/bin/ksh
File_name=`basename $0`
echo $File_name
the output will be linkfiles.sh
awk
awk
awk is very good programming language/command in manipulating files which have patterns separated by space or other delimiters. It does these operations faster than other basic unix commands like sed and grep.
awk’s power lies in its “one line” editing option which makes it more a command than a programming language.
Examples:
Example1)
suppose that you have a file which contains 4 columns of an SQL spool file.
/home/b3456/$ cat member_list.txt (Note: here /home/b3456/$ is the prompt string)
Ajit kumar 1000 BLR
sachin Sharma 2500 MUM
Ajay gupta 6560 MUM
Srinivasan R 4509 NDL
Ahmed javed 7845 CHE
To print only the city codes(third field or column ) enter the following command.
/home/b3456/$ awk ‘{print $3}’ member_list.txt
BLR
MUM
MUM
NDL
CHE
Example 2)
you want to get the member whose amount withdrawn ( 2nd column) is greater than or equal to 6000 and less than 7000 ,enter the command
/home/b3456/$ awk ‘ {if ( $2 >= 6000 && $2 < 7000) print $1 }’ member_list.txt
Ajay gupta
Example 3)
To get the total size of a group of files in a directory.
/home/b3456/$ ls –lrt
rw-r--r-- 1 b3456 Administ 19738 Feb 1 2011 NOV_2010_3.pdf
-rw-r--r-- 1 b3456 Administ 18176 Feb 1 2011 DEC_2010.pdf
--rw-r--r-- 1 b3456 Administ 739840 Jul 8 23:09 2011_ITR1_r2.xls
-rw-r--r-- 1 b3456 Administ 11455 Jul 8 23:13 ITR1_AJTPN1033E.xml
-rw-r--r-- 1 b3456 Administ 94058 Jul 9 14:08 09072011020807_13102006
-rw-r--r-- 1 b3456 Administ 101 Jul 9 14:10 address_TAX.txt
-rw-r--r-- 1 b3456 Administ 43237 Jul 9 14:17 XXXPN1033X_ITR-V.zip
-rw-r--r-- 1 b3456 Administ 1046954 Nov 21 14:29 EDU_LOAN_STMT.pdf
Enter the following command.
/home/b3456/$ ls -lrt | awk ' BEGIN {c = 0}{ c += $5 } END{print c}'
973559
Note that you don’t have to use ‘$’ symbol for printing a variable .
Example 4)
Many a times ,you may want to pass a shell variable to awk in the command line or inside
Shell-script and do some conditional manipulations based on the variable .
Consider a situation where you need to compare a name variable of the shell with the names in the file member_list.txt.
The script
#!/usr/bin/sh
……………..#lines of commands
……………..
p_name=`echo $m_name` #p_name Is a shell variable.
awk -v myp_name=$ p_name ‘{ if ( $1 == myp_name) print $0 }’ member_list.txt.
…………….
Here the awk command prints the entire line which contains the name field same as that of value of variable p_name
Example 5)
A single space or a sequence of space characters is the default delimiter for fields, if there are any other use awk with option –F and mention the delimiter.
If the file member_list2.txt contained the following,
/home/b3456/$ cat member_list2.txt
Ajit kumar|1000|BLR
sachin Sharma|2500|MUM
Ajay gupta|6560|MUM
Srinivasan R|4509|NDL
Ahmed javed|7845|CHE
Then to print only the city codes(third field or column ) as in example 1, awk can be used as follows.
/home/b3456/$ awk -F “|” ‘{ print $3 }’ member_list2.txt
BLR
MUM
MUM
NDL
CHE
AWK Examples
1) An awk script which performs arithmetic.
awk '{
x = 3; y = 3;
print ($1/7) + ($2 - y)/x
z = $2 - 4;
print z
}' file_1.txt
2) awk script to print the filename and its size and finally total size occupied by all files.
ls -lrt | awk 'BEGIN { c = 0; siz = 0;print FILENAME BYTES}
NF == 9 && /^-/
{
print $9 " " $5;c++;siz += $5
}
END{
print "\nTOTAL COUNT = " c "\nTOTAL SIZE = " siz
}'
3) consider a file(name_addr.txt) containing names and addresses with records seperated by a blank line and fields seperated by newline.
kaushik
nayak
dl hsg socty
vashi
jose
dmello
ms clny
blore
reema
j l
highway rd
seattle
awk command to extract only first name from this file will be
awk 'BEGIN{FS ="\n";RS=""}
{
print $1
}' name_addr.txt
4)awk command to count the total number of blank lines in a file.
awk ' /^$/
BEGIN{ c = 0}
{
print x += 1
}
END
{
print c
}' file_cont_blkline.txt
5) An awk script to count the total number of occurences of a pattern.
awk -v ptr=0003 'BEGIN{c = 0}{c += gsub(ptr," ",$0) }END{print c} ' sql2.txt
#here the pattern to count is '0003' whose value is passed through variable ptr.
6) An awk program to split output of netstat command into IP and port for a specific port .
netstat -an | awk '$4 ~ /.1521$/ {split($5,arr1,".");print "IP = " arr1[1]"."arr1[2]"."arr1[3]"."arr1[4] "|PORT = "arr1[5]}'
7) an awk program to print lines between two patterns (pattern ZO and a blank line)
awk '$0 ~ /ZO/,/^$/{ print $0}' select_all_bRS.txt
8)An awk program to write conditional output to multiple files .
awk '{ if ($1 ~ /2013/ ){ print $0 > “date.txt”} else if ( $3 ~ /Branch/ ) {print $0 >”brch.txt”} }’ branch_dt.txt
9) An awk script showing the use of next.
awk ‘$1 == “20130426” && $7 == “04060” && substr($2,1,2) == 10 {printf( “%s diff = %d \n”,$0,$4 – $2);next }{print}’ level_2.txt
#if next is not used then $0 will again be printed by {print}.next skips the record
alias
alias
alias is one of those commands for people who want to be lazy. you can use alias
in situations where it is too time consuming to type the same commands again and again.
But avoid aliases to commands like rm, kill etc.
Example. 1)
To always use vim instead of vi, and to make sure that whenever
there is a system crash, network failure etc during editing ,
all the contents are recovered, use the alias as follows.
/home/viru$ alias vi = ‘vim –r’
To make this happen every time you work after logging in, save the above line in
your .profile
i:e in the file $HOME/.profile
after saving it in .profile do not forget to run it.
ie /home/viru$ . $HOME/.profile
|
|
a dot here is necessary.
Example2) after running .profile ,to view all the aliases that are set globally, just enter alias.
/home/viru$ alias
alias ls=’ls –lrt’
alias psu=’ps –fu $LOGNAME’
alias df =‘df –gt’
alias jbin=’cd /home/viru/utils/java/bin’
alias jlib=’cd /home/viru/utils/java/lib’
seems viru is so lazy..!!
Example3)
To prevent the effect of aliases defined for a word, token or a command, use unalias.
/home/viru$ unalias jbin
/home/viru$ jbin
Jbin:not found
There is another way to accomplish this,that is by using quotes (‘’) after a command.the following lines show how.
/home/viru$ alias same
alias same=’/opt/bin/samefile.exe’
/home/viru$ same ‘’
same:not found.
The effect of alias was nullified for the word same. If you use double quotes after an aliased command name ,(eg: alias df =‘df –gt’) then the actual command will be run.(ie df’’ would run /usr/bin/df instead of df -gt)
Aliases which are defined outside a script or in your .profile do not work inside scripts. so make sure not to use aliased words in a shell script to contain their actual values used outside.
xargs
xargs
Xargs is a command which can pass arguments from the output of one command to another. It can run multiline commands by passing arguments in a single file. The most important feature is that it creates a list of arguments to be passed to the command and runs them.
Example1)
Consider that you have many files and each have to be renamed with a common subscript letter.
/home/Krishna $ ls account_[0-9].txt
account_4
account_5
account_7
…
…
Then to rename all these files as < filename>_old , use xargs as follows.
/home/Krishna$ ls account_[0-9].txt | xargs -I { } mv { } { }_prev
/home/Krishna $ls account_[0-9]*
account_4_prev
account_5_prev
account_7_prev
..
Here, xargs passed the output of ls, which is a list as an argument list which is denoted by
‘{ }’ to the mv command .the the entire line can be reconstructed as
mv account_4 account_4_prev
mv account_5 account_5_prev
….
..
Example2)
On few occasions, many unwanted processes of a common type may be running and it is necessary to kill all of them without killing any other process. One method would be to kill all of them with their individual PIDs, but it may be not possible to do this in scripts, xargs does the job.
When it is required to kill all processes run by your login name that have names containing ftp in it,use
ps -fu $LOGNAME | grep ftp | awk ‘{ print $2 }’ | xargs -I { } kill -9 { }
Example 3)
You want to copy a large number of files into a directory placing all the filenames in a list file.
Consider a file containing list of all the filenames .end line contains the destination directory
/home/Krishna$ cat candidates_logs.txt
Arjuna.log
Yudhistir.log
Bhim.log
Nakul.log
Sahadev.log
…
..
/var/tmp/logs
You want to copy all these files to a directory /var/tmp/logs with timestamp. use xargs as follows.
/home/Krishna$ xargs cp –p < candidates_logs.txt
The command structure created by xargs was
cp -p Arjuna.log Yudhistir.log Bhim.log Nakul.log Sahadev.log … candidates_logs.txt
This avoided the typing of all the files in an entire line. You may use this form of xargs in scripts where it is easier to make the list file using simple echo and “>>” .
Example 4)
You are redirecting output containing contents of a file to another and you want a delimiting character after every n words , where n is any natural number, use xargs as follows.
a point to note here is that a word can be a single line if it contains only one word per line.
/home/Krishna$ cat Train-Time.list
S29 F 12 Mumbai_CST Karjat 7:00pm
T113 F 12 (x) Mumbai_CST Thane 7:04pm
K95 S 9 (x) Mumbai_CST Kalyan 7:06pm
A59 S 9 Mumbai_CST Ambernath 7:15pm
BL39 F 12 Mumbai_CST Badlapur 7:17pm
T115 S 9 (x) Mumbai_CST Thane 7:20pm
N27 F 12 Mumbai_CST Kalyan 7:21pm
You can use awk to delimit the fields with a “|” or any other delimiting character, but xargs can perform it too.
/home/Krishna$ cat Train-Time.list | xargs –n1 | xargs -I { } echo “{ }|”
S29| F| 12| Mumbai_CST| Karjat| 7:00pm|
T113| F| 12| Mumbai_CST| Thane| 7:04pm|
K95| S| 9| Mumbai_CST| Kalyan| 7:06pm|
A59| S| 9| Mumbai_CST| Ambernath 7:15pm|
BL39| F| 12| Mumbai_CST| Badlapur| 7:17pm|
T115| S| 9| Mumbai_CST| Thane| 7:20pm|
N27| F| 12| Mumbai_CST| Kalyan| 7:21pm|
You can trim the ending “|” character by using sed .
Here you could have also used values greater than 1 after n so that the character “|” might be inserted after those many words instead of 1.
Example 5)
When running commands by passing multiple arguments , it sometimes becomes necessary to interactively ask you to run it for each argument as seen in the commands cp -i and mv -i.
xargs using its -p option, can perform this on any command you want to run interactively.
Consider a case where you need to take tar of files in a directory by adding every file interactively into the archive file.
/home/Krishna$ ls | xargs -p -n 1 tar -cvf /backup/ALL_KRISHNA.tar
tar -cvf /backup/ALL_KRISHNA.tar account_4_prev ?…y
tar -cvf /backup/ALL_KRISHNA.tar account_5_prev ?…y
tar -cvf /backup/ALL_KRISHNA.tar account_7_prev ?…n ———-> your reply.
tar -cvf /backup/ALL_KRISHNA.tar jokes_old ?…y
Recent Comments