Mohan WebSite

2012 SERVER (64)
2016 windows (9)
AIX (13)
Amazon (40)
Ansibile (19)
Apache (135)
Asterisk (2)
cassandra (2)
Centos (230)
- ftp (5)
- IPTABLES (15)
- SFTP (1)
- swap (3)
Centos RHEL 7 (270)
centos8 (3)
chef (3)
cloud (2)
cluster (3)
Coherence (1)
DB2 (5)
DISK (25)
DNS (9)
Docker (31)
Eassy (11)
ELKS (1)
EXCHANGE (3)
Fedora (6)
GIT (3)
GOD (2)
Grub (1)
Hacking (10)
Hadoop (6)
health (2)
horoscope (23)
Hyper-V (10)
IIS (15)
JAVA (7)
JBOSS (32)
jenkins (1)
Kubernetes (9)
Ldap (5)
Linux (189)
Linux Commands (166)
Load balancer (5)
mariadb (14)
Mongodb (4)
MQ Server (24)
MYSQL (84)
Nagios (5)
NaturalOil (13)
Nginx (36)
Ngix (1)
openldap (1)
Openstack (6)
Oracle (35)
Perl (3)
Postfix (19)
Postgresql (1)
PowerShell (2)
Python (3)
qmail (36)
Redis (12)
RHCE (28)
SCALEIO (1)
Security on Centos (29)
Shell (64)
Solaris (58)
Sql Server 2012 (4)
squid (3)
SSH (10)
SSL (14)
Storage (1)
TIPS on Linux (28)
tomcat (62)
Ubuntu (1)
Uncategorized (30)
Veritas (2)
vfabric (1)
VMware (28)
Weblogic (38)
Websphere (71)
Windows (19)
Windows Software (2)
wordpress (1)
ZIMBRA (17)

Detect failing storage LUN on Linux – multipath

f you login to server and after running dmesg – show kernel log command you get tons of messages like:

# dmesg

end_request: I/O error, dev sdc, sector 0
sd 0:0:0:1: SCSI error: return code = 0×00010000
end_request: I/O error, dev sdb, sector 0
sd 0:0:0:0: SCSI error: return code = 0×00010000
end_request: I/O error, dev sda, sector 0
sd 0:0:0:2: SCSI error: return code = 0×00010000
end_request: I/O error, dev sdc, sector 0
sd 0:0:0:1: SCSI error: return code = 0×00010000
end_request: I/O error, dev sdb, sector 0
sd 0:0:0:0: SCSI error: return code = 0×00010000
end_request: I/O error, dev sda, sector 0
sd 0:0:0:2: SCSI error: return code = 0×00010000
end_request: I/O error, dev sdc, sector 0
sd 0:0:0:1: SCSI error: return code = 0×00010000
end_request: I/O error, dev sdb, sector 0
sd 0:0:0:0: SCSI error: return code = 0×00010000
end_request: I/O error, dev sda, sector 0

In /var/log/messages there are also log messages present like:

# cat /var/log/messages ...

Apr 13 09:45:49 sl02785 kernel: end_request: I/O error, dev sda, sector 0
Apr 13 09:45:49 sl02785 kernel: klogd 1.4.1, ———- state change ———-
Apr 13 09:45:50 sl02785 kernel: sd 0:0:0:1: SCSI error: return code = 0×00010000
Apr 13 09:45:50 sl02785 kernel: end_request: I/O error, dev sdb, sector 0
Apr 13 09:45:55 sl02785 kernel: sd 0:0:0:2: SCSI error: return code = 0×00010000
Apr 13 09:45:55 sl02785 kernel: end_request: I/O error, dev sdc, sector 0

This is a sure sign something is wrong with SCSI hard drives orSCSI controller, to further debug the situation you can use themultipath command, i.e.:

multipath -ll | grep -i failed _ 0:0:0:2 sdc 8:32 [failed][faulty] _ 0:0:0:1 sdb 8:16 [failed][faulty] _ 0:0:0:0 sda 8:0 [failed][faulty]

As you can see all 3 drives merged (sdc, sdb and sda) to show up on 1 physical drive via the remote network connected – LUN to the server is showing as faulty. This is a clear sign something is either wrong with 3 hard drive members of LUN – (Logical Unit Number) (which is less probable) or most likely there is problem with the LUN network SCSI controller. It is common error that LUN SCSI controller optics cable gets dirty and needs a physical clean up to solve it.

In case you don’t know what is storage LUN? – In computer storage, a logical unit number, or LUN, is a number used to identify a logical unit, which is a device addressed by the SCSI protocol or protocols which encapsulate SCSI, such as Fibre Channel or iSCSI. A LUN may be used with any device which supports read/write operations, such as a tape drive, but is most often used to refer to a logical disk as created on a SAN. Though not technically correct, the term “LUN” is often also used to refer to the logical disk itself.

What LUN’s does is very similar to Linux’s Software LVM (Logical Volume Manager).

Mount point suddenly turned into READ ONLY

A RAC system, the GRID mount point on eighth node suddenly turned into read-only mode:

[oracle@node8 ~]$ cd /prod/grid
[oracle@node8 grid]$ touch 1
touch: cannot touch `1': Read-only file system

Instance is down due to that.
We can see below error in OS log:

Sep  5 03:26:13 node8 kernel: EXT3-fs error (device dm-226): ext3_lookup: unlinked inode 426196 in dir #2244752
Sep  5 03:26:13 node8 kernel: EXT3-fs error (device dm-226): ext3_lookup: unlinked inode 426199 in dir #2244752
Sep  5 03:26:13 node8 kernel: ext3_abort called.
Sep  5 03:26:13 node8 kernel: EXT3-fs error (device dm-226): ext3_journal_start_sb: Detected aborted journal
Sep  5 03:26:13 node8 kernel: Remounting filesystem read-only

Something wrong with inode in dir 2244752.
To found out what dir 2244752 is, we can use linux kernel command:
debugfs

But in this case i don’t care what the dir 2244752 is, we only want the issue to be fixed.

So we need to un-mount the mount point, use fsck to fix the Inode issue, and then re-mount the mount point.

We use losf/fuser command to find out all related processes and kill them, and through losetup command we ensure no resclusive device for this mount point, but still when we try to umount the mount point through “mount -f”, it reports mount point busy:

[root@node8 log]# fuser /prod/grid
[root@node8 log]#
[root@node8 log]# lsof |grep /prod/grid
[root@node8 log]#
[root@node8 log]# losetup -a
[root@node8 log]#

Above case means the mount point is holding by kernel, not a process. In this case, we can use “umount -l” to trigger a lazy umount, but with lazy umount, it will damage the file system when we try to fsck.
Also it may prevent us when we want to mount it back.

A clean and safe way is to reboot the server. After that we succeed fixed the mount point read only issue.

ext3_lookup finds an “unlinked inode” and calls ext3_abort which results in a read-only filesystem

Issue

The ext3-filesystem went read-only. In /var/log/messages the following errors are logged:
EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 91626760 in dir #19039379
Aborting journal on device dm-4.
ext3_abort called.
EXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
A couple of days later (after restoring with as written in the resolution):
EXT3-fs error (device dm-4): ext3_free_blocks_sb: bit already cleared for block 43951098
Aborting journal on device dm-4.
EXT3-fs error (device dm-4) in ext3_free_blocks_sb: Journal has aborted
EXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-4) in ext3_truncate: Journal has aborted
EXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-4) in ext3_orphan_del: Journal has aborted
EXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-4) in ext3_delete_inode: Journal has aborted
ext3_abort called.
EXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
__journal_remove_journal_head: freeing b_committed_data
nfs ext3 file system going read-only on two servers which are nfs cross mounted to each other.
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339312 in dir #318520
Aug 16 11:57:17 server1 kernel: Aborting journal on device dm-10.
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339315 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339293 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339321 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339323 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339276 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339322 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339291 in dir #318520
Aug 16 11:57:17 server1 kernel: ext3_abort called.
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_journal_start_sb: Detected aborted journal
Aug 16 11:57:17 server1 kernel: Remounting filesystem read-only
Aug 16 12:02:10 server1 multipathd: dm-10: umount map (uevent)
Aug 16 12:02:11 server1 kernel: ext3_abort called.
Aug 16 12:02:11 server1 kernel: EXT3-fs error (device dm-10): ext3_put_super: Couldn’t clean up the journal
Aug 16 12:03:55 server1 multipathd: dm-10: remove map (uevent)
Aug 16 12:05:01 server1 multipathd: dm-10: add map (uevent)
Aug 16 12:05:22 server1 kernel: kjournald starting. Commit interval 5 seconds
Aug 16 12:05:22 server1 kernel: EXT3-fs warning (device dm-10): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
Aug 16 12:05:22 server1 kernel: EXT3-fs warning (device dm-10): ext3_clear_journal_err: Marking fs in need of filesystem check.
Aug 16 12:05:22 server1 kernel: EXT3-fs warning: mounting fs with errors, running e2fsck is recommended

Solution for this problem

Run this

Quote:

umount -l /tmp
e2fsck /usr/tmpDSK
mount -o loop,rw,noexec,nosuid /usr/tmpDSK /tmp
chmod 1777 /tmp

e2label, fdisk, /etc/fstab, mount, linux rescue, rescue disk, CentOS

Tutorial: e2label, fdisk, /etc/fstab, mount, linux rescue, rescue disk, CentOS

Let’s run through an example of a fresh disk, that needs to be configured, going through partitioning with fdisk, make filesystem, filesystem labelled using e2label, /etc/fstab edited, and mounted using mount. This is relevent for CentOS 3.x, 4.x 5.x; YMMV for other flavours.

First, a run through of each command, with usage examples.

fdisk

if the partitioning of the drive is unknown, fdisk -l can be used to list the partition table of the drive. eg. fdisk -l /dev/sdb

following is an example of a disk without a partition table

# fdisk -l /dev/sdb

Disk /dev/sdb: 74.3 GB, 74355769344 bytes
255 heads, 63 sectors/track, 9039 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn’t contain a valid partition table

and here’s an example of a disk with valid partition table

# fdisk -l /dev/sda

Disk /dev/sda: 299.9 GB, 299974524928 bytes
255 heads, 63 sectors/track, 36469 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device	Boot	Start	End	Blocks	Id	System
/dev/sda1	*	1	13	104391	83	Linux
/dev/sda2		14	268	2048287+	82	Linux swap
/dev/sda3		269	36469	290784532+	83	Linux

to change the partitions on a drive without partition table, such as a new drive, use fdisk
eg. fdisk /dev/sdb

# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won’t be recoverable.

The number of cylinders for this disk is set to 9039.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help):

m will list the options available.
eg. p will print the partition table, n is for adding new partition table;
and after the changes are done, w will commit the changes, and exit the program.

e2label

Usage of e2label is simple. manpage for e2label shows following command line options,

e2label device [ new-label ]

device can be the entire drive or a partition.
eg.
/dev/sda is the entire drive
/dev/sda1 is the first partition of device /dev/sda
/dev/sda2 is the second partition of device /dev/sda

eg. to show the label of /dev/sda1 and /dev/sda3

# e2label /dev/sda1
/boot

# e2label /dev/sda3
/

To change the label for /dev/sda3, do as follows,
# e2label /dev/sda3 changed-label

and to confirm the change was done sucessfully,
# e2label /dev/sda3
changed-label

if you want a / in front of the label,

# e2label /dev/sda3 /changed-label

to verify that the change was done,

# e2label /dev/sda3
/changed-label

if you want the server to be able to boot next time round, ensure changes are made to /etc/fstab as well. Otherwise, on the next boot, you may be prompted with errors, eg.

fsck.ext3: Unable to resolve ‘LABEL=/boot’
fsck.ext3: Unable to resolve ‘LABEL=/’

in which case, you can boot up with a rescue disk, eg. CentOS installation disk, enter “linux rescue” at prompt; chroot /mnt/sysimage and edit /etc/fstab accordingly. Alternatively, if you are prompted for root password, and gets a shell, you may be able to edit /etc/fstab. If the filesystem is readonly, remount the filesytem readwrite. eg. mount -o remount,rw /

mount, umount

# mount /dev/sdb5 /tmp/mnt

will mount /dev/sdb5 on the /tmp/mnt directory
if /tmp/mnt does not exist, it needs to be created, using mkdir

to unmount the device, use umount /dev/sdb5 or umount /tmp/mnt

Example of partitioning, formatting and labelling of a new drive

2 partitions to be created
100MB and rest of the drive

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-9039, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-9039, default 9039): +100M

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (14-9039, default 14):
Using default value 14
Last cylinder or +size or +sizeM or +sizeK (14-9039, default 9039):
Using default value 9039

Command (m for help): p

Disk /dev/sdb: 74.3 GB, 74355769344 bytes
255 heads, 63 sectors/track, 9039 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device	Boot	Start	End	Blocks	Id	System
/dev/sda1		1	13	104391	83	Linux
/dev/sda2		14	9039	72501345	83	Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Next: format both partitions as ext3 filesystems

# mkfs.ext3 /dev/sdb1
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
26104 inodes, 104388 blocks
5219 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
13 block groups
8192 blocks per group, 8192 fragments per group
2008 inodes per group
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729

Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 22 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

# mkfs.ext3 /dev/sdb2
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
9076736 inodes, 18125336 blocks
906266 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
554 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 33 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

next: use e2label to label the volumes. Although both mkfs3.ext and
tune2fs is capable of labelling, with the -L option, I am deliberately
using e2label in this example to demonstrate the use of e2label.

# e2label /dev/sdb1 /drive2-100MB

confirm its done.

# e2label /dev/sdb1
/drive2-100MB

label sdb2 next,

# e2label /dev/sdb2 /drive2-restofdrive
Warning: label too long, truncating.
# e2label /dev/sdb2
/drive2-restofdr

the description used was too long, and got truncated. maximum for label is 16 characters.

next, we try to mount and unmount the 2 filesystems created,

mount shows what has already been mounted. in addition, the -l option shows the filesystem label

# mount -l
/dev/sda3 on / type ext3 (rw) [/]
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw) [/boot]
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

We are mainly concerned with the drives, so let’s grep for the relevent entries

#mount -l | grep /dev/sd
/dev/sda3 on / type ext3 (rw) [/]
/dev/sda1 on /boot type ext3 (rw) [/boot]

We need mount points; the following mount points are created,
# mkdir /mountpt1
# mkdir /mountpt2

to manually mount the drives,
# mount /dev/sdb1 /mountpt1

check that it is mounted,
# mount -l|grep /dev/sd
/dev/sda3 on / type ext3 (rw) [/]
/dev/sda1 on /boot type ext3 (rw) [/boot]
/dev/sdb1 on /mountpt1 type ext3 (rw) [/drive2-100MB]

# mount /dev/sdb2 /mountpt2
# mount -l|grep /dev/sd
/dev/sda3 on / type ext3 (rw) [/]
/dev/sda1 on /boot type ext3 (rw) [/boot]
/dev/sdb1 on /mountpt1 type ext3 (rw) [/drive2-100MB]
/dev/sdb2 on /mountpt2 type ext3 (rw) [/drive2-restofdr]

to unmount, use umount
# umount /dev/sdb1
# umount /dev/sdb2
# mount -l|grep /dev/sd
/dev/sda3 on / type ext3 (rw) [/]
/dev/sda1 on /boot type ext3 (rw) [/boot]

we can also mount the filesystems using the filesystem labels,
# mount -L /drive2-100MB /mountpt1/
# mount -L /drive2-restofdr /mountpt2/
# mount -l |grep /dev/sd
/dev/sda3 on / type ext3 (rw) [/]
/dev/sda1 on /boot type ext3 (rw) [/boot]
/dev/sdb1 on /mountpt1 type ext3 (rw) [/drive2-100MB]
/dev/sdb2 on /mountpt2 type ext3 (rw) [/drive2-restofdr]

unmount can also use the mount point instead of the device, eg.

# umount /mountpt1
# umount /mountpt2
# mount -l |grep /dev/sd
/dev/sda3 on / type ext3 (rw) [/]
/dev/sda1 on /boot type ext3 (rw) [/boot]

next: edit /etc/fstab, add in entries for the new drive if we want it to be mounted at boot up.

following is existing /etc/fstab for the server, which was installed for purpose of this tutorial,

LABEL=/	/	ext3	defaults	1	1
LABEL=/boot	/boot	ext3	defaults	1	2
tmpfs	/dev/shm	tmpfs	defaults	0	0
devpts	/dev/pts	devpts	gid=5,mode=620	0	0
sysfs	/sys	sysfs	defaults	0	0
proc	/proc	proc	defaults	0	0
LABEL=SWAP-sda2	swap	swap	defaults	0	0

add the following entries,

LABEL=/drive2-100MB	/mountpt1	ext3	defaults	1	1
LABEL=/drive2-restofdr	/mountpt2	ext3	defaults	1	2

mount has a -a option which reads /etc/fstab and mounts all filesystems mentioned in the file,

# mount -a
# mount -l |grep /dev/sd
/dev/sda3 on / type ext3 (rw) [/]
/dev/sda1 on /boot type ext3 (rw) [/boot]
/dev/sdb1 on /mountpt1 type ext3 (rw) [/drive2-100MB]
/dev/sdb2 on /mountpt2 type ext3 (rw) [/drive2-restofdr]

next, reboot, and ensure the new drive is mounted automatically, and accessible via the mount points, /mountpt1, /mountpt2

windump

windows environment.

Its exactly the same as tcpdump for linux.

http://www.winpcap.org/windump/

However to get this working you will need WinPcap

http://www.winpcap.org/default.htm

Once installed this is a great tool to watch network traffic.

You can specify IP addresses, subnets, ports, interfaces and combinations of. I provide a few examples, but the documentation is great. And as I said its the equivalent of tcpdump, so commands should work the same.

Watch a particular subnet
windump -n net 192.168.11.0 mask 255.255.255.0

Watch a particular IP and port
windump -n host 192.168.1.226 and tcp port 443

Watch two particular IPs
windump -n host 192.168.11.10 or host 192.168.1.226

Watch a two particular IPs on ports 80 and 443
windump -n (host 192.168.11.10 and (tcp port 80 or 443)) or (host 192.168.1.226 and (tcp port 80 or 443))

List interface and numbers. You need the number to specify an interface to listen on.
windump –D

Watch a particular IP on a particular interface.
windump -i 4 -n host 192.168.17.35

To exclude parameters just append with an exclamation

Watch a particular IP and all traffic except on a specific port

windump -n host 192.168.1.226 and tcp port !443

Convert from CRT to PFX with openssl

In Windows cases you need to merge these files into a PFX file.

Now before I tell you the secret I wish to explain what the difference is between the different files so you know how the certificate works.

•A .csr file is a certificate signing request which initiates your certificate request with a certificate provider and contains administrative information about your organization.
•A .key file is the private key used to encrypt your site’s SSL-enabled requests.
•.pem and .crt extensions are often used interchangeably and are both base64 ASCII encoded files. The technical difference is that .pem files contain both the certificate and key whereas a .crt file only contains the certificate. In reality this distinction is often ignored.
•The PFX extension is used on Windows servers for files containing both the public key files (your SSL certificate files, provided by for instance DigiCert) and the associated private key (generated by your server at the time the CSR was generated).

So in my case I had a crt file for the certificate itself and I had a crt file for the CA and I had the private key within a .key file.
Now I had to “merge” these into a PFX file so I could import it for use for Lync.
Now this is where openssl comes in.

Openssl comes often default with most linux distroes (ubuntu,fedora etc) in my case I had the Netscaler VPX available.
Before you can use openssl on Netscaler you have to type the command shell to enter the regular freebsd shell.

So type the command openssl pkcs12 –export –out certificate.pfx –inkey rsaprivate.key –in certificate.crt –certfile fileca.crt
After that you need to type a password to encrypt the pfx file.

openssl pkcs12 -export -out rmohan.pfx -inkey rmohan.key -in rmohan_com.crt -certfile PositiveSSLCA2.crt

Centos and Rhel 6.5 Hadoop 2.4 3 Node Server Cluster

Centos and Rhel 6.5 Hadoop 2.4 3 Node Server

hadoop word has been popular for many years, the mention of big data will think hadoop, hadoop then what role is it?
Official definition: hadoop is a developing and running large-scale data processing software platform. Core Words is a platform, which means
we have a lot of data, there are several computers, we know the task decomposition to process the data on each computer,
but do not know how to assign tasks, how to recycle a result, hadoop probably help us did it.

Hadoop Distributed File System (HDFS) is a very important part of Hadoop, this paper briefly describes several features of HDFS,
and then analyze the principles behind it, namely how to achieve these characteristics.

1, HDFS

We should be the first consideration is how to save huge amounts of data, how management.
This has been distributed file system, HDFS.

2, Map-Reduce

After the data is saved, how do we deal with these data do, if the way I deal with the complexity, rather than just sort of find how to do this operation?
There is a need to be able to provide a place to write code, let us write their own operations,
which then decompose inside, distribution, recycling and so forth.

3, Hive

Can compile the code is good, but too much trouble to compile the code and database personnel are familiar with SQL statements,
SQL statements can be processed, do not Map-Reduce it, so there are a Hive. And big data is inseparable from the database anyway, is inseparable from the table,
Hive will be able to talk about the data mapped into the data table, and then operate on the convenience, its drawback is slower.

4, HBase

Since Hive slower, then there is no faster database? HBase is that he was born of a query, the query quickly.

5, Sqoop

Did we not have a lot of well-known databases like MySQL, Oracle, data is the existence of this inside me,
and how imported into HDFS as well? Sqoop provides conversion between relational databases and between HDFS.

6, Flume
So much work on the computer, if one little problem, or at which service a problem, how do you know what the worse?
Flume provides a highly reliable log collection system.

7, Mahout

Many large data processing is used for data mining, there are several common that machine learning algorithms,
since the algorithms are fixed and on what kinds of, it developed a thing called Mahout achieve a variety of algorithms, developers can be more quick to use.

8, Zookeeper

ZooKeeper goal is a good package of key services complex and error-prone, easy to use interface and the performance of high efficiency,
stable system functions available to the user. White said that the zoo administrator, he is used to manage the elephant (Hadoop), bees (Hive)’s.
These are the key members of the Hadoop family, there are several less common would not have introduced, that the role of these members,
the whole can do for Hadoop will have a preliminary understanding, the rest is slowly learning the principles of the various parts of and use it.

HDFS Features

1, high fault tolerance. This is the core characteristics of HDFS, and deployed in a large amount of data on inexpensive hardware, even if some of the disks fails, HDFS can quickly recover lost data.

2, simple consistency. This means that HDFS suited write once, read many of the program, the file is written, you do not modify it. Like MapReduce program or web crawler is a perfect fit for this feature.

3, instead of moving the mobile computing data. This good explanation, the data is too big, bad move, HDFS provides interfaces that allow the program to move to their own data from the close position.

4, platform compatibility. Platform should be able to resolve the differences, this allows more extensive use HDFS.

HDFS architecture

1, HDFS is a typical master-slave relationship, the master node is NameNode, DataNode from a node.

NameNode nodes are managers, key management system name space, when the program needs to read the data, first wanted to ask NameNode storage location of data blocks.

There are many DataNode node, usually in the form of tissue rack, the rack and then connected through the switch. DataNode main function is to save the data block, but also to information NameNode report data blocks, not three seconds to send a “heartbeat”, if 10 minutes have not received a heartbeat, then consider this DataNode broken, then you need Data recovered.

2, the following describes the principle DataNode backup, this is one of the reasons HDFS high fault tolerance.

Data on HDFS are not only save what you can, and each file will be copied several times (default 3 times), and then placed in different places in order to avoid data loss.

That is how to save it? There are three copies of each data block, the first one is the data itself, and the second is saved in the same rack (can be understood as the same hard disk) under different DataNode last saved on a different rack on DataNode.

3, except Namenode and DataNode, there is SecondaryNameNode, his role is mainly cyclical merge files stored on NameNode storage location of data blocks, while the NameNode damaged, you can manually recover from SecondaryNameNode in part, but not all.

4, SecondaryNameNode NameNode can not solve a single problem, in order to improve fault tolerance, HDFS as well as HA (high availability) mechanism: Two NameNode. There Federation mechanisms: multiple NameNode.

5, the data block (block), like the Linux system has the smallest unit of data per disk read and write: 512 bytes

The HDFS has the same concept, but the size becomes a 64M, it is because HDFS require multiple read, but reading is to constantly seek, we should try to make a minimum data transfer time compared to seek, if Seek to transmit hundredth time, seek time of 10ms, the transmission speed of 100MB / s, then the block size is 100MB. After the hard disk transfer speed, the block size may be increased. But too much is not good block, a task processing a block, the task will be slower. When the file is less than 64MB, the system will assign a Block consent to this file, but the actual disk resource is no wasted.

6, for a large number of small files, HDFS provides two containers, unified file management: SequenceFile and MapFile.

7, the compression. Compression can reduce the space, there are three: gzip, LZO, Snappy. gzip compression rate highest, but consuming CPU, and slow. Snappy lowest compression ratio, but fast. LZO centered.

HDFS operating

Finally, some common command HDFS

1, hadoop fs – Here are some basic operations:

hadoop fs-mkdir (path) to build folder

hadoop fs-ls (path) list files and directories

hadoop fs-put file upload path

hadoop fs-get file path download

hadoop fs-text file viewer

hadoop fs-rm file deletion

2, hadoop namenode-formate format NameNode

3, hadoop job-submit submit jobs

hadoo job-kill kill jobs

4, hadoop fsck-blocks block print out reports

hadoop fsck-racks print DataNode network topology

Summary

This article describes several features of HDFS and to explain some of its key principles and features of HDFS Finally, common command.
After reading this article HDFS can have a basic understanding of the principles of the specific details or to read a book,
in addition to that there will be HDFS using Java to operate.

Hadoop MapReduce Next Generation – Cluster Setup

192.168.1.40 cluster1.rmohan.com = Master
192.168.1.41 cluster2.rmohan.com = Slave1
192.168.1.42 cluster3.rmohan.com = Slave2

[root@cluster1 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.40 cluster1.rmohan.com cluster1
192.168.1.41 cluster2.rmohan.com cluster2
192.168.1.42 cluster3.rmohan.com cluster3

Add ssh-keygen -t rsa
create in cluster2 and cluster3 update it on the

[root@cluster1 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
ef:a6:45:ea:bd:c5:f7:6e:1e:3a:16:71:50:51:f8:47 root@cluster1.rmohan.com
The key’s randomart image is:
+–[ RSA 2048]—-+
| .++|
| .. E|
| .o |
| . .o|
| S . o .|
| + . . |
| . o o o. |
| . +.. +..o|
| oo+….++|
+—————–+
[root@cluster1 ~]#

[root@cluster1 .ssh]# ls
id_rsa id_rsa.pub known_hosts

enable on all the 3 server

vi /etc/ssh/sshd_config

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

scp .ssh/id_rsa.pub root@cluster2.rmohan.com:/root/.ssh/id_rsa.pub

scp .ssh/id_rsa.pub root@cluster3.rmohan.com:/root/.ssh/id_rsa.pub

[root@cluster3 .ssh]# cat id_rsa.pub >> authorized_keys
cd /root/

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA1mH8BU9kD89womyrIJ10+xNNmXeNszbKX+C1M03ygLynX4ppFo/UX26UnS1GdZFUMTbTKeHAAcIp7n0puHZU4vgF+SAOZMMaeOJT5Qdt6d8CK2neyTi3kYvP6b5+D0ug9ZENG1hc2V+WfYzvimoNsA1lCLz3v8JF3/Ubs9IkU3u+/ipNzKBW0jk/RmDXwGN4SIV7FzoyKVPsdc9kpMKBBb/pX+IlZcd5KvpE/0RaWSKDFh5rE6exNUyw5V3zCNHDLHINAKq+fT+z8dvGCd0ejV7694KrCaxiUKCDWtY2WzVZ43aqvAHqZsqCIkVMOdC7CW5anvvJKcHduuumyTDUwQ== root@cluster2.rmohan.com
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAu+MYbCwHl8NEKmiY4cExIZfVjTv4F2xGlJDXX+/pJKt1Xk2QFZg4i6JCN2h/GreSlpX/GenDU/C21QG/LN3o5hg2Y/88rFStY2O0K5Z44twVwkc+BJxTpBsfNqQfKnqOEVKOS6xGYK7LT3ytr8NaLp/bGV49bLrMoZCNIYuizH5BTW3IqxKsp0JJ5XSqTuuPZh4LPPn8Ecl9HvVDxJ1xBP00FYSTPg48YErvyVUMpercEIoWM3k+rJUh3DqFOyN+O92nqy7/S290rM6dk9p0R6/43MZjVYM61c6AtlxG7m4bt3Gk0dxC8WHbQRTfEdbUuq/jqXN1LPKZV8KCmPGvuQ== root@cluster3.rmohan.com
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAxHQ1ZgJc1Fj65OlHqg89Kli6e+lO9D7gde9HNWKMu6ELWV+jMMbzv9b0o7Q4zfCd5T3+0BuV19iTt3rvEAk2NyVSIQFh5s1XNmIb6mSIt+ORg/agIwES9Hx747gX3iV4uidbhlgGMtvzpBO2NHUeJt4vh7ALTQ26Frkcb3npPEt2KiWmpOcx5C9EJqeMWTt81HFXwZsj/wEX0Vi977KFspvgSnoxcWPd2WTa4WFdq6Lyo8vQYigo85AWv+7CxiLnliR0Zf+W9QWOTALCRRlwnCOYOU0Q+8qEPK3aD0NW4fJ8Ez9gvMZEo5SrStyqIp0xlRrJzQjIlZR0YrmBKPRK8Q== root@cluster1.rmohan.com

Install Java

All 3 Nodes

[root@cluster1 software]# rpm -ivh jdk-7u17-linux-x64.rpm

[root@cluster1 software]# cat /etc/profile.d/java.sh
export JAVA_HOME=/usr/java/jdk1.7.0_17
export JRE_HOME=/usr/java/jdk1.7.0_17/jre
export PATH=$PATH:/usr/java/jdk1.7.0_17
export CLASSPATH=./:/usr/java/jdk1.7.0_17/lib:/usr/java/jdk1.7.0_17/jre/lib
[root@cluster1 software]#

vi /etc/profile.d/hadoop.sh

export HADOOP_HOME=/usr/hadoop/hadoop-2.4.0
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HADOOP_MAPARED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop

mkdir /usr/hadoop
mkdir /usr/hadoop/dfs
mkdir /usr/hadoop/dfs/name
mkdir /usr/hadoop/dfs/data

wget http://mirror.nus.edu.sg/apache/hadoop/common/hadoop-2.4.0/hadoop-2.4.0.tar.gz

tar -xvf hadoop-2.4.0.tar.gz -C /usr/hadoop/

[root@cluster1 etc]# cd hadoop/
[root@cluster1 hadoop]# ls
capacity-scheduler.xml hadoop-env.sh httpfs-env.sh mapred-env.cmd ssl-client.xml.example
configuration.xsl hadoop-metrics2.properties httpfs-log4j.properties mapred-env.sh ssl-server.xml.example
container-executor.cfg hadoop-metrics.properties httpfs-signature.secret mapred-queues.xml.template yarn-env.cmd
core-site.xml hadoop-policy.xml httpfs-site.xml mapred-site.xml.template yarn-env.sh
hadoop-env.cmd hdfs-site.xml log4j.properties slaves yarn-site.xml
[root@cluster1 hadoop]# pwd
/usr/hadoop/hadoop-2.4.0/etc/hadoop

~/hadoop-2.4.0/etc/hadoop/hadoop-env.sh

~/hadoop-2.4.0/etc/hadoop/yarn-env.sh

~/hadoop-2.4.0/etc/hadoop/slaves

~/hadoop-2.4.0/etc/hadoop/core-site.xml

~/hadoop-2.4.0/etc/hadoop/hdfs-site.xml

~/hadoop-2.4.0/etc/hadoop/mapred-site.xml

~/hadoop-2.4.0/etc/hadoop/yarn-site.xml

Add the java details

1) hadoop-env.sh

vi hadoop-env.sh

# The java implementation to use.
export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/usr/java/jdk1.7.0_17

2) yarn-env.sh
vi yarn-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_17

3) slaves

cat slaves
cluster2.rmohan.com
cluster3.rmohan.com

4) core-site.xml

vi core-site.xml

hadoop.tmp.dir
/usr/hadoop/hadoop-2.4.0/tmp/hadoop-${user.name} fs.default.name
hdfs://hadoopmaster:9000

5) mapred-site.xml

cp mapred-site.xml.template mapred-site.xml

mapred.job.tracker
http://hadoopmaster:9001 mapreduce.framework.name
yarn mapreduce.jobhistory.address
hadoopmaster:10020 mapreduce.jobhistory.webapp.address
hadoopmaster:19888 mapred.system.dir
/mapred/system
true mapred.local.dir
/mapred/local
true

6) hdfs-site.xml

vi hdfs-site.xml

dfs.namenode.secondary.http-address
hadoopmaster:9001 dfs.namenode.name.dir
file:/usr/hadoop/dfs/name dfs.datanode.data.dir
file:/usr/hadoop/dfs/data dfs.replication
1 dfs.webhdfs.enabled
true

mapred-site.xml

7) yarn-site.xml

vi yarn-site.xml

yarn.resourcemanager.address
Hadoopmaster:8080 yarn.resourcemanager.scheduler.address
hadoopmaster:8081 yarn.resourcemanager.resource-tracker.address
hadoopmaster:8082 yarn.nodemanager.resource.memory-mb
10240 yarn.nodemanager.remote-app-log-dir
${hadoop.tmp.dir}/nodemanager/remote yarn.nodemanager.log-dirs
${hadoop.tmp.dir}/nodemanager/logs yarn.nodemanager.aux-services
mapreduce_shuffle

Copy the files to Node2 and Node3

scp -r hadoop-2.4.0 root@cluster2.rmohan.com:/usr/hadoop/
scp -r hadoop-2.4.0 root@cluster3.rmohan.com:/usr/hadoop/

/usr/hadoop/hadoop-2.4.0/sbin

[root@cluster1 sbin]# ./start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
14/06/06 17:37:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Starting namenodes on [hadoopmaster]
The authenticity of host ‘hadoopmaster (192.168.1.40)’ can’t be established.
RSA key fingerprint is 42:90:4a:3c:a3:fc:8e:85:ca:f9:d6:10:bb:93:ed:b2.
Are you sure you want to continue connecting (yes/no)? yes
hadoopmaster: Warning: Permanently added ‘hadoopmaster’ (RSA) to the list of known hosts.
hadoopmaster: starting namenode, logging to /usr/hadoop/hadoop-2.4.0/logs/hadoop-root-namenode-cluster1.rmohan.com.out
cluster2.rmohan.com: datanode running as process 2815. Stop it first.
cluster3.rmohan.com: datanode running as process 2803. Stop it first.
Starting secondary namenodes [hadoopmaster]
hadoopmaster: starting secondarynamenode, logging to /usr/hadoop/hadoop-2.4.0/logs/hadoop-root-secondarynamenode-cluster1.rmohan.com.out
14/06/06 17:37:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/hadoop/hadoop-2.4.0/logs/yarn-root-resourcemanager-cluster1.rmohan.com.out
cluster3.rmohan.com: starting nodemanager, logging to /usr/hadoop/hadoop-2.4.0/logs/yarn-root-nodemanager-cluster3.rmohan.com.out
cluster2.rmohan.com: starting nodemanager, logging to /usr/hadoop/hadoop-2.4.0/logs/yarn-root-nodemanager-cluster2.rmohan.com.out

[root@cluster1 sbin]# ./stop-all.sh
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
14/06/06 18:24:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Stopping namenodes on [hadoopmaster]
hadoopmaster: stopping namenode
cluster2.rmohan.com: stopping datanode
cluster3.rmohan.com: stopping datanode
Stopping secondary namenodes [hadoopmaster]
hadoopmaster: stopping secondarynamenode
14/06/06 18:25:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
stopping yarn daemons
stopping resourcemanager
cluster2.rmohan.com: stopping nodemanager
cluster3.rmohan.com: stopping nodemanager
no proxyserver to stop
[root@cluster1 sbin]#

[root@cluster1 hadoop-2.4.0]# /usr/hadoop/hadoop-2.4.0/sbin/slaves.sh /usr/java/jdk1.7.0_17/bin/jps
cluster3.rmohan.com: 3886 NodeManager
cluster2.rmohan.com: 4057 Jps
cluster3.rmohan.com: 4052 Jps
cluster2.rmohan.com: 3789 DataNode
cluster3.rmohan.com: 3784 DataNode
cluster2.rmohan.com: 3891 NodeManager

[root@cluster1 hadoop-2.4.0]# /usr/java/jdk1.7.0_17/bin/jps
5407 NameNode
5726 ResourceManager
5584 SecondaryNameNode
6442 Jps
[root@cluster1 hadoop-2.4.0]#

[root@cluster1 bin]# ./hadoop namenode -format
************************************************************/
[root@cluster1 bin]# ./hadoop namenode -format
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

14/06/06 17:55:05 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = cluster1.rmohan.com/192.168.1.40
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.4.0
STARTUP_MSG: classpath = /usr/hadoop/hadoop-2.4.0/etc/hadoop:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/commons-net-3.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jsr305-1.3.9.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/junit-4.8.2.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jettison-1.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/activation-1.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/commons-io-2.4.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/xz-1.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/paranamer-2.3.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/guava-11.0.2.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/hadoop-auth-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/hadoop-annotations-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/asm-3.2.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/commons-el-1.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/avro-1.7.4.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/lib/zookeeper-3.4.5.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/hadoop-nfs-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/hadoop-common-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/common/hadoop-common-2.4.0-tests.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/commons-el-1.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/hadoop-hdfs-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/hadoop-hdfs-2.4.0-tests.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/guice-3.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/activation-1.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/xz-1.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/jline-0.9.94.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/jackson-xc-1.8.8.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/jackson-jaxrs-1.8.8.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/asm-3.2.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/zookeeper-3.4.5.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/hadoop-yarn-client-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/hadoop-yarn-api-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/hadoop-yarn-server-common-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/hadoop-yarn-common-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/junit-4.10.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/hamcrest-core-1.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.4.0.jar:/usr/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.4.0-tests.jar:/contrib/capacity-scheduler/*.jar:/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common -r 1583262; compiled by ‘jenkins’ on 2014-03-31T08:29Z
STARTUP_MSG: java = 1.7.0_17
************************************************************/
14/06/06 17:55:05 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
14/06/06 17:55:05 INFO namenode.NameNode: createNameNode [-format]
14/06/06 17:55:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Formatting using clusterid: CID-ab82af59-a88b-4e6a-b2b6-9e7ba859e781
14/06/06 17:55:06 INFO namenode.FSNamesystem: fsLock is fair:true
14/06/06 17:55:06 INFO namenode.HostFileManager: read includes:
HostSet(
)
14/06/06 17:55:06 INFO namenode.HostFileManager: read excludes:
HostSet(
)
14/06/06 17:55:06 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
14/06/06 17:55:06 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
14/06/06 17:55:06 INFO util.GSet: Computing capacity for map BlocksMap
14/06/06 17:55:06 INFO util.GSet: VM type = 64-bit
14/06/06 17:55:06 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
14/06/06 17:55:06 INFO util.GSet: capacity = 2^21 = 2097152 entries
14/06/06 17:55:06 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
14/06/06 17:55:06 INFO blockmanagement.BlockManager: defaultReplication = 3
14/06/06 17:55:06 INFO blockmanagement.BlockManager: maxReplication = 512
14/06/06 17:55:06 INFO blockmanagement.BlockManager: minReplication = 1
14/06/06 17:55:06 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
14/06/06 17:55:06 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
14/06/06 17:55:06 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
14/06/06 17:55:06 INFO blockmanagement.BlockManager: encryptDataTransfer = false
14/06/06 17:55:06 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
14/06/06 17:55:06 INFO namenode.FSNamesystem: fsOwner = root (auth:SIMPLE)
14/06/06 17:55:06 INFO namenode.FSNamesystem: supergroup = supergroup
14/06/06 17:55:06 INFO namenode.FSNamesystem: isPermissionEnabled = true
14/06/06 17:55:06 INFO namenode.FSNamesystem: HA Enabled: false
14/06/06 17:55:06 INFO namenode.FSNamesystem: Append Enabled: true
14/06/06 17:55:07 INFO util.GSet: Computing capacity for map INodeMap
14/06/06 17:55:07 INFO util.GSet: VM type = 64-bit
14/06/06 17:55:07 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
14/06/06 17:55:07 INFO util.GSet: capacity = 2^20 = 1048576 entries
14/06/06 17:55:07 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/06/06 17:55:07 INFO util.GSet: Computing capacity for map cachedBlocks
14/06/06 17:55:07 INFO util.GSet: VM type = 64-bit
14/06/06 17:55:07 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
14/06/06 17:55:07 INFO util.GSet: capacity = 2^18 = 262144 entries
14/06/06 17:55:07 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
14/06/06 17:55:07 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
14/06/06 17:55:07 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
14/06/06 17:55:07 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
14/06/06 17:55:07 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
14/06/06 17:55:07 INFO util.GSet: Computing capacity for map NameNodeRetryCache
14/06/06 17:55:07 INFO util.GSet: VM type = 64-bit
14/06/06 17:55:07 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
14/06/06 17:55:07 INFO util.GSet: capacity = 2^15 = 32768 entries
14/06/06 17:55:07 INFO namenode.AclConfigFlag: ACLs enabled? false
Re-format filesystem in Storage Directory /usr/hadoop/dfs/name ? (Y or N) y
14/06/06 17:55:09 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1380935534-192.168.1.40-1402048509097
14/06/06 17:55:09 INFO common.Storage: Storage directory /usr/hadoop/dfs/name has been successfully formatted.
14/06/06 17:55:09 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
14/06/06 17:55:09 INFO util.ExitUtil: Exiting with status 0
14/06/06 17:55:09 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at cluster1.rmohan.com/192.168.1.40
************************************************************/

[root@cluster1 bin]# netstat -antp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2588/sshd
tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 2461/sshd
tcp 0 0 192.168.1.40:9001 0.0.0.0:* LISTEN 3610/java
tcp 0 0 192.168.1.40:22 192.168.1.4:51736 ESTABLISHED 2755/sshd
tcp 0 0 192.168.1.40:22 192.168.1.1:58989 ESTABLISHED 2465/sshd
tcp 0 176 192.168.1.40:22 192.168.1.1:58985 ESTABLISHED 2461/sshd
tcp 0 0 ::ffff:192.168.1.40:8080 :::* LISTEN 3744/java
tcp 0 0 ::ffff:192.168.1.40:8081 :::* LISTEN 3744/java
tcp 0 0 ::ffff:192.168.1.40:8082 :::* LISTEN 3744/java
tcp 0 0 :::22 :::* LISTEN 2588/sshd
tcp 0 0 :::8088 :::* LISTEN 3744/java
tcp 0 0 ::1:6010 :::* LISTEN 2461/sshd
tcp 0 0 :::8033 :::* LISTEN 3744/java
tcp 0 0 ::ffff:192.168.1.40:8082 ::ffff:192.168.1.42:49955 ESTABLISHED 3744/java
tcp 0 0 ::ffff:192.168.1.40:8082 ::ffff:192.168.1.41:55070 ESTABLISHED 3744/java

[root@cluster1 bin]#

URLS To Check the Cluster Details

http://cluster1.rmohan.com:8088/cluster

http://cluster1.rmohan.com:9001/status.jsp

http://cluster1.rmohan.com:50070/dfshealth.jsp

[root@cluster1 hadoop-2.4.0]# ./bin/hdfs dfsadmin -report
14/06/06 18:46:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Configured Capacity: 105689374720 (98.43 GB)
Present Capacity: 92940345344 (86.56 GB)
DFS Remaining: 92940296192 (86.56 GB)
DFS Used: 49152 (48 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

————————————————-
Datanodes available: 2 (2 total, 0 dead)

Live datanodes:
Name: 192.168.1.41:50010 (cluster2.rmohan.com)
Hostname: cluster2.rmohan.com
Decommission Status : Normal
Configured Capacity: 52844687360 (49.22 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 6304174080 (5.87 GB)
DFS Remaining: 46540488704 (43.34 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.07%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Last contact: Fri Jun 06 18:46:15 SGT 2014

Name: 192.168.1.42:50010 (cluster3.rmohan.com)
Hostname: cluster3.rmohan.com
Decommission Status : Normal
Configured Capacity: 52844687360 (49.22 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 6444855296 (6.00 GB)
DFS Remaining: 46399807488 (43.21 GB)
DFS Used%: 0.00%
DFS Remaining%: 87.80%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Last contact: Fri Jun 06 18:46:15 SGT 2014

root@cluster1 hadoop-2.4.0]# ./bin/hdfs fsck / -files -blocks
14/06/06 18:48:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where app
Connecting to namenode via http://hadoopmaster:50070
FSCK started by root (auth:SIMPLE) from /192.168.1.40 for path / at Fri Jun 06 18:48:04 SGT 2014
/

Status: HEALTHY
Total size: 0 B
Total dirs: 1
Total files: 0
Total symlinks: 0
Total blocks (validated): 0
Minimally replicated blocks: 0
Over-replicated blocks: 0
Under-replicated blocks: 0
Mis-replicated blocks: 0
Default replication factor: 3
Average block replication: 0.0
Corrupt blocks: 0
Missing replicas: 0
Number of data-nodes: 2
Number of racks: 1
FSCK ended at Fri Jun 06 18:48:04 SGT 2014 in 10 milliseconds

The filesystem under path ‘/’ is HEALTHY

How to create DFS Directory

[root@cluster1 hadoop-2.4.0]# ./bin/hdfs dfs -mkdir /mohan
14/06/06 18:49:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

[root@cluster1 hadoop-2.4.0]# cat /tmp/test/test1
hello www.google.com
hello1 www.yahoo.com
hello2 www.msn.com
hello3 www.rediff.com
hello4 www.amazon.com
hello5 www.ebay.com

[root@cluster1 hadoop-2.4.0]# cat /tmp/test/test2
the feeling that you understand and share another person’s experiences and emotions : the ability to share someone else’s feelings
He felt great empathy with the poor.
His months spent researching prison life gave him greater empathy towards convicts.
Poetic empathy understandably seeks a strategy of identification with victims Helen Vendler, New Republic, 5 May 2003
Origin of EMPATHY
Greek empatheia, literally, passion, from empaths emotional, from em- + pathos feelings, emotion more at pathos

[root@cluster1 test]# /usr/hadoop/hadoop-2.4.0/bin/hadoop fs -put -f test1.txt /mohan
14/06/06 19:45:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
[root@cluster1 test]# /usr/hadoop/hadoop-2.4.0/bin/hadoop fs -put -f test2.txt /mohan
14/06/06 19:45:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

./bin/hadoop jar ./share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.3.0-sources.jar org.apache.hadoop.examples.WordCount /mohan /output

./bin/hadoop fs -cat /output/part-r-00000
./bin/hadoop fs -cat /output/part-r-00000

Add Proxy Server on Linux and Unix Server

When you need to have proxy connection available to whole system, so dont bother with settings for firefox, yum proxy and similar, its simple as just plain file 😉

just copy & paste following code to terminal window. if you need other shell , like csh , just add another file with csh extension.
This should work for every user on system.

/bin/cat < /etc/profile.d/proxy.sh
export http_proxy=http://host.name:port/
export ftp_proxy=http://host.name:port/
export HTTP_PROXY=http://host.name:port/
export FTP_PROXY=http://host.name:port/
export no_proxy=.localdomain.com
EOF
In case , that you want enable proxy access to only specific user, just add lines to his bash-profile file
echo “export http_proxy=http://hostname.or.ip:port” >> /user_home/.bash_profile
just change hostname/ip to your hostname/ip of proxy server eg. myproxy.company.sk:8080

How-To disable IPv6 on RHEL6 / CentOS 6

in /etc/sysctl.conf : net.ipv6.conf.all.disable_ipv6 = 1

in /etc/sysconfig/network : NETWORKING_IPV6=no

in /etc/sysconfig/network-scripts/ifcfg-eth0 : IPV6INIT=”no”

disable iptables6 – chkconfig –level 345 ip6tables off

reboot

In /etc/grub.conf edit the kernel lines to include:

1
ipv6.disable=1
The “trick” here, as described by TrevorH1 in #linux, is that programs can still load the module as they wish. You can check this by running lsmod or modprobe -l, you will still see the ipv6.ko kernel module. This allows user mode programs to access the kernel module in their code (so they don’t crash); but as far as the kernel is concerned ipv6.disable=1, so the kernel doesn’t really allow much to get through it.

And that’s it. IPv6 is disabled on your box… but if you want to disable a variety of fun things that you might find when seeking to disable IPv6…

In /etc/sysctl.conf change/create entries:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
In /etc/sysconfig/network change/create entries:

NETWORKING_IPV6=no
IPV6INIT=no
In /etc/modprobe.d/blacklist.conf change/create entries:

blacklist net-pf-10
blacklist ipv6
Disable iptables for IPv6:

service ip6tables stop
chkconfig ip6tables off
Disable ipv6 completely:

echo “install ipv6 /bin/true” > /etc/modprobe.d/ipv6_disabled.conf

Aix commands

Commands

CUT:

lspv | cut -c 6-7 lists the hdisks numbers (cuts the 6th and 7tn character (disk nubers))
dlnkmgr view -drv | grep -w 00000 | awk ‘{print $4}’ | cut -d . -f 3

——————————————————————————–

EXPR: integer arithmetic

\* multiplication (\ is needed to tell * is not a special character)
/ division
% remainder
+ addition
– substraction

expr $ 6 + 4 $ / 2 5

——————————————————————————–

FIND: recursively searches the directory tree

find / -xdev -type f -name “cor*” -size +10000c -exec ls -l {} \;

/ where to find
-xdev looks only in the given fs
-type f what type to look for (f: file, d: direcory)
-name “cor*” name to look for
-size +10000c search for the given size (+10000c : greater than 10000 character (larger then 10kbyte))
-size +2 search for greater than 1kbyte (bloxk size is 512 byte, that’s why +2 is given)
-exec ls -l {} \; what command to execute
({}: it represents each findings and \; is needed at the end of line)
e.g. find /home -xdev -name “*.old” -exec rm {} \;
-mtime -1 file modifocation date is less then 1 day (+1 means older than 1 day)

find . -name ‘*.ksh’ -ok rm {} \; this will ask if the command (rm) should be executed or not (much safer if find is not perfect)
find /etc/rc.d -ls it lists the files and dirs in the given dir
find /tmp/bb -exec awk ‘/func/ {print FILENAME}’ {} \; it will check in /tmp/bb if any files contains the word “func” and list those

——————————————————————————–

GREP: (Global Regular Expression Parser) The grep command searches for the pattern.

grep -i ignore case (small letter or capital)
grep -c displays only a count of matching lines
grep -n shows line number at the beginning of the matching line
grep -v invert the sense of matching, to select non-matching lines
grep -w selects only whole words
grep -l lists the name of the files (once) which contain matching line (grep -l paging00 *)

ps -ef | egrep ‘ssh|aix’ select lines with ssh and aix
errpt -a|grep -A1 ‘NAME’ selects lines with NAME and shows 1 line after that (A: after, B: before)

——————————————————————————–

HEAD, TAIL:

head filename displays first 10 lines of the file
head -50 filename displays first 50 lines

tail filename displays last 10 lines of the file
tail -50 filename displays last 50 lines
tail +15 filename displays the file starting from the 15th line
tail -f filename used when we are monitoring the file (interactively shows how the file is changing)

——————————————————————————–

LSOF:

NETWORK
-n no host name resolution
-P not converts port numbers to port names

lsof -i show all network connections
lsof -i| grep LISTEN shows what ports are waiting for connections
lsof -i | grep ESTABLISHED shows current active connections

lsof -i tcp list only TCP connections (works with udp as well)
lsof -i : shows all connections of a given port (lsof -i :22 or lsof -i :ssh)
lsof -i @ shows connections to a specific host (lsof -i @192.168.1.5)
lsof -i protocol:@ip:port this is the syntax for network findings (protocol: tcp or udp, @ip: any ip address, port: a port number)
lsof -a -u -i shows all network activity of a user (-a combines -u and -i)

USER, FILES, PROCESSES
lsof -u shows what is open for a user (lsof -u oracle)
lsof shows what is using a file (lsof /var/adm/syslog.log)
lsof shows what pids are using the filesystem (directory) (it can be good if we cannot umount)
lsof +D shows which file is open by which process in a directory (lsof +D /usr)
(+D will show recurcively the open files, otherwise only device name would be shown)
lsof -N list all NFS files
lsof -c shows files and network connections a command is using (lsof -c ssh)

lsof -p shows what a proccess ID has open (lsof -p 335545)

—————————–

HOW TO FIND WHAT FILES ARE OPENED BY A PID

1.
root@aix20: /root # lsof -p 319660
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ovcd 319660 root 32r VREG 10,5 12153 69705 /usr (/dev/hd2)

(it will show the filesystem and the indode number (69705))

2.
root@aix20: /root # find /usr -inum 69705
/usr/lpp/VV/msg/en_US/sec.cm.client.cat

——————————————————————————–

METACHARACTERS, VARIABLES:

GREP SHELL
single character . ?
multicharacter .* *
character range [x-z] [x-z]
begins a line ^ N/A
ends a line $ N/A
variable N/A $

HOME home directory
$PS1 prompt symbol
set displays the values of all shell variables

——————————————————————————–

RPM:

rpm -qa shows what rpm packages are installed
rpm -ql shows where the files are installed (rpm -qlp .. shows the absolut paths???)
rpm -q –filesbypkg cdrecord list all the files on installed rpm package
rpm -qf /usr/bin/lynx query a file to find the source rpm package
rpm -qi list information on an installed rpm package
rpm -qR list all dependencies on any rpm package

rpm -ivh httpd-2.2.8.aix5.1.rpm install the rpm package

rpm -ivh –force *.rpm
rpm -ivh –force –nodeps does not check dependency (same can be done with “rpm -Uvh…” for upgrade)
rpm -e removes the rpm package

rpm -Va shows which files are missing from the RPM database
rpm -Vv verifies a package
rpm –rebuilddb compress and rebuild the RPM database

/usr/sbin/updtvpkg enables the rpm command to recognize that the libraries have been installed

In some cases you might get an error about failed dependencies when you install RPMs on AIX (for example, error: failed dependencies: libX11.a(shr4.o) is needed by tk-8.3.3-1). Most likely the error occurs because the rpm command does not recognize the shared library. If the error occurs, check to see if the X11 libraries are installed in the directory /usr/lpp/X11/lib. If they are not installed, use the AIX product media to install them. After you have installed the libraries, run the above command (updtvpkg). The command enables the rpm command to recognize that the libraries have been installed.

——————————————————————————–

RSYNC:

Much like cp, rsync copies files from a source to a destination. Unlike cp, the source and destination of an rsync operation can be local or remote. rsync can resume a transfer after interruption; it transfers only those portions of a file that differ between source and destination; and rsync can perform entire or incremental backups.

By default, rsync uses Secure Shell (SSH) as its transport mechanism; you can reuse your machine aliases and public keys with rsync

rsync -av /tmp/bb_rs30 /tmp/bb it will copy the directory (/tmp/bb_rs30) and its contents under /tmp/bb
rsync -av /tmp/bb_rs30/ /tmp/bb it will copy the contents of the named directory but not the directory itself (check / at the end)
-v option enables verbose messages
-a option (stands for archive), is a shorthand for -rlptgoD:
recurse, copy symbolic links as symbolic links, preserve permissions, preserve file times,
preserve group, preserve owner, and preserve devices and special files, respectively

rsync -n -av /tmp/bb_rs30/ /tmp/bb -n: previews what will happen but does not move a single byte (it is good for testing)

rsync -av /tmp/bb_rs30/ aix2:/tmp/bb_rs20 it will copy the contents of the given dir to the other host
(assuming the same login name on the remote machine)

——————————————————————————–

SORT:

cat /etc/passwd | sort -t: -n +2
-t: gives the delimeter character (here :, deafult is the space)
-n numerical order
+2 sort by the 3rd column
-r reverse order

df -m | sort -n +3 | egrep ‘9.%|100%’ sorts by the 4th column (%) of the filesstems, which are 90-100% full
cat animals | sort +0.1 sort the file animals by the second character of the first word
ls -lR |sort -nk 5|tail -20 sorts the 20 largest file
lspv | sort -k1.6n list hdisk in numerical order (lspv|sort -tk +1 -n <--it is the same (delimiter is "k") lspv | sort -k1,1.5 -k1.6n list hdisks then vpaths in numeric order -------------------------------------------------------------------------------- TAR: tar -cvf /tmp/file.tar . saves where we are to the given path (-c: create, -v: verbose, -f: file) tar -cvf conf.tar conf creates conf.tar of the dir conf tar -xvf /tmp/file.tar extracts the tar file (-x: extract) tar -xvf -C /home/bb /tmp/file.tar it will extract the tar file to /home/bb tar -tvf /tmp/file.tar list (verify) the content of a tar file tar -cf - * | gzip -9 > vmi.tar.gz makes tar and zip wehere we are
tar cvf – openssh5.8p1 | gzip -c > openssh5.8p1.tgz creates tgz in one command

tar: 0511-197 DirectorServer: Cannot write data extracted with the tar command:
A file cannot be larger than the value set by ulimit.

I changed these ulimit settings after that it was OK (dont’t forget to logout and login again)
root@bb_lpar: /bb # chuser fsize=-1 fsize_hard=-1 root
root@bb_lpar: /bb # chuser data=-1 data_hard=-1 root

——————————————————————————–

TRUSS:

truss it will show what system calls a command makes
truss -c -p it will trace a process (-c: counts the system calls rather than displaying them, -p: pid)
truss -d -p -o
truss -t open lsps -a shows whatis needed for the given command (here lsps -a)

——————————————————————————–

XARGS:

lspv|awk ‘{print$1}’|xargs it will list the elements in a line separated with space
cat list | xargs -t rm will rm all the files which are in the file list
ls | xargs -t -I {} mv {} {}.old it will rename to .old all the file in the current dir
-t it echoes the constructed command (trace mode) (it is optional)
-I {} insert each line of the ls to that place where the {} symbol appear

lsdev -Cc disk | xargs -n1 rmdev -dl removes all the listed disks
-n1 1 element will be passed each time to xargs (if n2 then the command will be created with 2 elements)

——————————————————————————–

ZIPPING:

gzip-V shows gzip version
gzip file1 zip
gzip -d file1.gz unzip

gzip -9 filename will zip (and remove the file automatically)
gunzip filename unzipping it

gzip -c file > file.gz it creates a zip file but leave the original file as well
> filename after that original file can be emptied

compress file1 the original file will be deleted and a new compressed file will be created with a .Z at the end
zcat file1.Z displays the compressed files (without uncompression)
uncomress file1.Z uncompressing a file (the compressed one will be deleted)

if you receive this:
gunzip SysDir6_2.tar.gz
gunzip: SysDir6_2.tar: File too large

this is because gzip 1.2.4 (or lower gzip versions) need a patch to accept large files, workaround for this problem:
gzip -d -c SysDir6_2.tar.gz| tar xvf –

——————————————————————————–

OTHER:

file shows the type of a file

diff file1 file2 compares only text files
cmp file1 file2 compares all types of files
dircmp dir1 dir2 compares directories

time cp file1 /home show the time to accomplish the command (godd for performance analysis)

ps $$ shows which shell is in use
echo $SHELL shows the current shell
printenv view environmental variables

cat -vet file shows tabs, enters… as viewable characters
-v displays non-printing characters as visible characters
-t displays tab as ^I
-e displays enter as $

!!! sh -xv for troubleshooting, if a command fails, you can see where exactly it failed in the command script!!!
(sh -xv exportvg rootvg)

dd if=/dev/zero of=/home/user/bb/4GB_file bs=1m count=4000 it creates a 4GB file
lmktemp it creates a file with given size

alias dir=’ls’ creates an alias
unalias dir removes an alias
set – unset

df -g | awk ‘{sum = sum + $3} END {print sum}’ it sums up the fs in gigabyte (by the 3rd column)
for i in `ls -l | grep old | awk ‘{print $9}’` ; do du -sk $i; done | awk ‘{sum = sum + $1} END {print sum}’

CPU – PROCESSES:

Physical – Virtual – Logical CPU:

Physical Processors are cores in the machine. Virtual Processors are assigned to an LPAR manually when LPAR is created. Logical Processors are created automatically by AIX, depending on the SMT setting.

Simultaneous Multi-Threading (SMT)
SMT is that feature of a Power Processor, when multiple hardware threads can run on one physical processor at the same time (a processor appears as 2 or 4 logical CPU). Within a CPU (core/cpu/processor are the same thing) there are multiple execution units. For example: floating point arithmetic unit, load and store execution units… A single thread would use only 1 or 2 of those units at any point in time. So most of the executional units within a core will not be utilized. With the ability of multi-threading 2 (or 4) threads could be running in a core at same time. One of them will use the floating processor while the other doing load and store …(If there are collisions, one of them would be delayed but it happens no too often.)

How threads will be dispatched to multiple cores:
First thread will be dispatched to the primary hw thread of a physical cpu. If we have another CPU then next thread will be dispatched there (to avoid collision)

IMPORTANT:

– Prior AIX6 TL4: if only a single hw thread was busy, processor reported as 100% utilized (this is an error because all the secondary threads were not utlized)

– AIX6 TL4 and later: potential capacity of unused hw threads are from TL4 reported as idle time for the processor (it measures the capacity of the unused hw threads.)

————————

SMT behaviour and intelligent SMT threads:

AIX default behaviour is to use all the VPs for maximun performance. If workload grows it will use up all VPs (CPU cores) quickly, but AIX first uses SMT thread 1 on all CPU cores before allocating work to the 2nd, 3rd and 4th SMT threads.

SMT threads can be seen as Logical CPUs on AIX. If SMT=4 then 1 VP shows up as 4 Logical CPU. From Power7 there is a thing called “intelligent SMT threads”. If there are not enough processes to run on all SMT threads (official mode is SMT=4) it will be dynamically switched to 2 or 1.

mpstat or topas -L is showing it:

At the column “lpa” the sign “-” will show turned off SMT threads

# mpstat 2

cpu min maj mpc int cs ics rq mig lpa sysc us sy wa id pc %ec lcs
0 0 0 0 265 35 24 2 0 100 64 100 0 0 0 0.63 31.7 99
1 0 0 0 12 12 0 0 0 100 9 0 0 0 100 0.12 6.1 22
2 0 0 0 9 0 0 0 0 – 0 0 0 0 100 0.12 6.1 20 <--this SMT thread is turned off 3 0 0 0 9 0 0 0 0 - 0 0 0 0 100 0.12 6.1 19 <--this SMT thread is turned off 4 0 0 0 100 12 8 1 0 100 0 100 0 0 0 0.64 31.8 99 5 0 0 0 19 59 0 0 0 100 9 0 0 0 100 0.12 6.1 69 6 0 0 0 9 0 0 0 0 - 0 0 0 0 100 0.12 6.1 9 <--this SMT thread is turned off 7 0 0 0 9 0 0 0 0 - 0 0 0 0 100 0.12 6.1 9 <--this SMT thread is turned off ALL 0 0 0 432 118 32 3 0 0 82 63 0 0 37 2.00 999.8 346 -------------------------------------------------------------------------------- ------------------------ Context Switch: It is inherent in any multiprocessing operating system. Different appl. threads are sharing a CPU. Every time 1 thread is leaving a CPU and a new thread is dispatched to the CPU, a context switch occurs. The environment of the leaving one has to be saved and new environment ha to be reestablished for the new process. High context switch rates can cause many work (overhead) for the CPU, which can be a problem. ------------------------ PROCESS: You use commands to tell the operating system what task you want it to perform. When commands are entered, they are recognized by a command interpreter (also known as a shell), and the task is processed. A program or command that is actually running on the computer is referred to as a process. The commom types of processes: Foreground processes Processes that require a user to start them or to interact with. Programs and commands run as foreground processes by default. Background processes Processes that are run independently of a user. To run a process in the background, type the name of the command with the appropriate parameters and flags, followed by an ampersand (&). When a process is running in the background, you can perform additional tasks by entering other commands at the command prompt. Most processes direct their output to standard output (stdout), even when they run in the background. Because the output from a background process can interfere with your other work on the system, it is usually good practice to redirect the output of a background process to a file. Daemon processes Daemons are processes that run unattended. They are constantly in the background and are available at all times. Daemons are started usually when the system starts, and they run until the system stops. A daemon process typically performs system services. For example qdaemon (provides access to system resources such as printers) and sendmail are daemons. Zombie processes A zombie process is a dead process that is no longer executing but is still recognized in the process table (in other words, it has a PID number). Zombie processe have been killed or have exited and continue to exist in the process table until the parent process dies or the system is shut down and restarted. Zombie processes display as when listed by the ps command. The only way to remove zombies is to reboot the system.

Thread
Each process is made up of one or more kernel threads. A thread is a single sequential flow of control. Rather than duplicating the environment of a parent process, as done via fork, all threads within a process use the same address space and can communicate with each other through variables.

——————————

Process priority

A priority is a number assigned to a thread. The kernel maintains a priority value (0-255). A smaller priority value indicates a more important thread. Real time thread priorities are lower than 40.

Nice value
A nice value is a priority adjustment factor added to the base user priority of 40 (for non-fixed priority threads). The nice value is used by the system to calculate the current priority of a running process. The first process in the system (init) has a nice value of 20, and therefore an effective priority of 60. (PRI heading in the below output) A foreground process has a nice value of 20 (24 for a background process).

ps -el shows process priorities
ps -ekl shows process priorities including kernel processes
ps -kmo THREAD shows processes with their threads priorities

root@aix31: / # ps -el <--shows the nice values under the NI heading (-- means it is running with fixed prio. F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD 200003 A 0 1 0 0 60 20 7037000 784 - 0:39 init 200103 A 0 311326 352456 0 24 -- 81d8400 4676 - 1:15 xmtopas The nice value can be set at process creation time by using the nice command. If the process already created the renice command is used. (ksh will add automatically 4 to the default nice value (20) if a process is started in the background (&)) The nice value can be ranged from 0 to 39, with 39 being the lowest priority. nice -10 add 10 to current nice value (lower priority)
nice –10 subtract 10 from current nice value (higher priority)

The renice value can be -20 to 20. (1-20: lowers the priority, 0:sets to the base scheduling priority, -20 to -1:highers the priority)
renice 10 -p add 10 to the default nice value (20) (lower priority)
renice -n 10 -p add 10 to current nice value (lower priority)
renice -10 -p subtract 10 from the default nice value (20) (higher priority)
renice -n -10 -p subtract 10 from current nice value (higher priority)
(-n: incerment is added to the current nice value, not default)

—————————

CPU infos:

lscfg | grep proc shows how many (virtual) processors we have (lsdev -Cc processor, shows also how many virt. proc we have)
bindprocessor -q shows how many logical (SMT) processors we have
lsattr -El procX shows the processor settings
pmcycles -m shows the processors speed (if smt is enabled it will show for all the logical processors)
smtctl it will show how many processor we have (if smt is turned on or not)

—————————

Process handling:

Ctrl-C or Ctrl-Backspace cancels a foreground process

ps lists processes (by default lists only processes started from the current terminal)
-e every process runnning on the system
-f full listing (username, PPID…)
-L lists all processes which PPID is -u lists all processes running under
-T lists the tree of a given process (shows the children of a given process)

ps -elmo THREAD lists processes and its threads (shows pids and the threads (tid) which belong to a given process)

proctree displays the process tree of the specified process

kill notification to the process to terminate (it is using the default, 15, signal)
kill -9 kills the process without notification
kill -1 restarts the process (rereads the config files as well) (HUP – hangup)
(when a background process is running and you log off a hangup signal is sent)
kill -2 interrupt signal (same as ctrl+c)
kill -l lists all the signals supported by kill (cat /usr/include/sys/signal.h will show as well, with details)

ls -R / > ls.out & starts ls in the background (standard output is ls.out)
nohup ls -R / > ls.out & nohup allows a background process to continue after logging off the system
(if output isn’t redirected, it will create nohup.out)
echo “” | at now this also starts in the background (and you can log off)
jobs lists which processes are running in the background

nohup alt_disk_copy -d hdisk1 -B & can’t be hanged up and in backgound (kill command can stop it)

Restarting a stopped foreground process (jobs command):
1. Ctrl-Z stops a foreground process, its PID is still in the process table (it goes to background)
2. jobs this will list stopped processes
[1] + Stopped (SIGTSTP) ./myscript <--you will see a line like this (here #1 is the job id) 3. fg %1 put given job into foregeound (bg %1 puts into background) Restarting a stopped foreground process (ps -ef ):
1.Ctrl-Z stops a foreground process, its PID is still in the process table (it goes to background)
2.ps -ef | grep find the process ID (PID)
3.fg restarts that stopped proces (it will go to foreground)

Removing a background process:
1.find / -type f > output & run the find command in the background
2.ps lists the PID numbers
3.kill cancel the process

——————————

The operating system allows you to manipulate the input and output (I/O) of data to and from your system. For example you can specify to read input entered on the keyboard (standard input) or to read input from a file. Or you can specify to write output data to the screen (standard output) or to write it to a file.

When a command begins running, it usually expects that the following files are already open: standard input, standard output and standard error. A number, called a file descriptor, is associated with each of these files:

0 represents standard input (stdin)
1 represents standard output (stdout)
2 represents standard error (stderr)

The redirection symbols and their meanings:
< redirects input (stdin) (< filename is added to the end of the command) > redirects output (stdout) (> filename is added to the end of the command)
>> appends output
<< inline input (see pg. 574) 2> redirects output (stderr)
1>&2 redirects stdout to stderr
2>&1 redirects stderr to stdout

mail denise < letter1 sends the file letter1 to user denise with the mail command echo $PATH > path1 saves the value of the PATH variable on the file path1
cat file2 >> file1 append file2 to file1 (the cat commands can concatenate not only display files)
ls -l file1 2> list1 save the stderr to file list1 (if file1 does not exist)
ls *.dat *.txt > files.out 2> files.err (files.out: stdout, file.err: stderr)
command > output 2>&1 saves all the output (stdout and stderr) in one single file

ulimit

Resource limits is the concept where you regulate several resources consumed by a process on an UNIX operating systems. Although the resource limits are set on a per user basis, they are applied per process basis. Therefore, if a user is executing hundreds of processes, the user may consume huge amount of resources, even if the resource setting values for the user are relatively small numbers.

On UNIX systems, the ulimit command controls the limits on system resource, such as process data size, process virtual memory, and process file size. Specifically:

On Solaris systems, by default, the root user has unlimited access to these resources (for example, unlimited).
On AIX, some limits might apply to the root user.

On UNIX systems, each user can either inherit resource limits from the root user or have specific limits defined. When setting resource limits for a process, it is important to know that the limits that apply are those that are in effect for the parent process and not the limits for the user under which the process runs. For example, the IBM Directory server runs under the ldap user account that was created at install time. However, the IBM Directory server is typically started while logged in as the root user. Starting while logged in as the root user means that any limits that are in effect for the ldap user have no effect on the IBM Directory server process unless the IBM Directory server process is started while logged in as the ldap user.

To display the current user’s resource limits, use the ulimit command (see the following example):

# ulimit -Ha
time(seconds) unlimited
file(blocks) 2097151
data(kbytes) unlimited
stack(kbytes) unlimited
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) unlimited

# ulimit -Sa
time(seconds) unlimited
file(blocks) 2097151
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) 2000

-H option instructs the command to display hard resource limits.
-S option instructs the command to display soft resource limits.

The hard resource limit values are set by the root user using the chuser command for each user. The soft resource limit values can be relaxed by the individual user using the ulimit command, as long as the values are smaller than the hard resource limit values.
Increasing process memory size limit
Enter the following command to check the current process data size and virtual memory size limits:

ulimit -d
ulimit -m

It is recommended that the process data size and virtual memory size be set to unlimited. Setting to unlimited can be done by modifying the following lines in the /etc/security/limits file:

default:
data = -1
rss = -1

For changes to the /etc/security/limits file to take effect, the user must log out of the current login session and log back in.

At minimum, set these size limits to 256 MB, which is the value of 256000 in the /etc/security/limits file. Increase these limits when a larger-than-default IBM Directory server cache is to be used. For more information, see the IBM Directory Server documentation.

In addition to the /etc/security/limits file, the process virtual memory size is limited by the number of segments that a process can use. By default, a process can only use one memory segment, which limits a process to 128 MB. AIX support a large memory model that is enabled through the LDR_CNTRL environment variable.
Increase file size limit
Enter the following command to check the current file size limits:

ulimit -f

It is recommended that the file size limit be set to unlimited. Setting to unlimited can be done by modifying the following lines in the /etc/security/limits file:

default:
fsize = -1

Create file systems with large file support
The standard file system on AIX has a 2 GB file size limit, regardless of the ulimit setting. One way to enable files larger than the 2 GB limit is to create the file system with the Large File Enabled option. This option can be found through the Add a Journaled File System option of the smit menu. Refer to AIX documentation for additional information and file system options.
Edit/Change the Ulimit Values for uid:

Edit the limits file under /etc/security/limits (takes effect after reboot)
Use the chuser command to change individual user settings (logout and login required)

Here are a few of the flags that can be set:
chuser rss=-1 username
chuser fsize=-1 username
chuser data=-1 username
chuser nofiles=4000 username
chuser “stack=8388608? username

Recent Posts

Pages

Categories

Archives

Recent Comments

Categories

Detect failing storage LUN on Linux – multipath

Mount point suddenly turned into READ ONLY

Mount point suddenly turned into READ ONLY

e2label, fdisk, /etc/fstab, mount, linux rescue, rescue disk, CentOS

windump

Convert from CRT to PFX with openssl

Centos and Rhel 6.5 Hadoop 2.4 3 Node Server Cluster

Add Proxy Server on Linux and Unix Server

How-To disable IPv6 on RHEL6 / CentOS 6

Aix commands

ulimit

PRODUCTS