Hadoop 2.6.4 distributed cluster environment is built on the actual cluster .
Cluster preparation
There are five machines that have changed the machine name for master, slaver1, slaver2, slaver3, slaver4, and set the face password ssh login. You can refer to here http://www.linuxidc.com/Linux/2016-02/128149.htm
Machine name Ip
Master 192.168.1.1
Slaver1 192.168.1.2
Slaver2 192.168.1.3
Slaver3 192.168.1.4
Slaver4 192.168.1.5
Install the JDK
CentOS 7 is openJDK by default
Uninstall the openJDK under CentOS 7 and install Sun JDK1.7
View the openJDK installation path
rpm -qa | grep java
Uninstall openJDK
rpm -e –nodeps java-1.7.0-openjdk-1.7.0.85-2.6.1.2.el7_1.x86_64
rpm -e –nodeps java-1.8.0-openjdk-devel-1.8.0.60-2.b27.el7_1.x86_64
rpm -e –nodeps java-1.6.0-openjdk-1.6.0.36-1.13.8.1.el7_1.x86_64
rpm -e –nodeps java-1.6.0-openjdk-devel-1.6.0.36-1.13.8.1.el7_1.x86_64
Install Sun JDK1.7
Download from the official website jdk-7u80-linux-x64.rpm, upload to master
Install Sun JDK1.7
rpm -ivh jdk-7u80-linux-x64.rpm
Modify environment variables
In /etc/profile add in
export JAVA_HOME=/usr/java/jdk1.7.0_80
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
source /etc/profile
Install Hadoop 2.6.4
Download, extract
Hadoop 2.6.4 downloaded from the official website and decompress on the master
extraction path of their choice, here is my decompression in
/root/workspace/software/hadoop-2.6.4
Add environment variable
In /etc/profileadd in
export HADOOP_HOME=/root/workspace/software/hadoop-2.6.4/
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
Modify the Hadoop configuration file
Hadoop-env.sh
In the hadoop decompression path below, / etc / hadoop / hadoop-env.sh add the following two lines
export JAVA_HOME=/usr/java/jdk1.7.0_80
export HADOOP_PREFIX=/root/workspace/software/hadoop-2.6.4
Core-site.xml
Under the hadoop decompression path, add the following to / etc / hadoop / core-site.xml
Hdfs-site.xml
In the hadoop decompression path below, / etc / hadoop / hdfs-site.xml add the following
Set here to 3, indicating that the data has 3 copies.
Mapred-site.xml
Under the hadoop decompression path, add the following to / etc / hadoop / mapred-site.xml
Yarn-env.sh
In the hadoop decompression path below, / etc / hadoop / yarn-env.sh increase below, increase the Java-HOME configuration
export JAVA_HOME=/usr/java/jdk1.7.0_80
Yarn-site.xml
Under the hadoop decompression path, add the following to / etc / hadoop / yarn-site.xml
Add some of the port number, easy to remotely from the browser to view the cluster situation, it is recommended to add in accordance with this.
Slaves
Under the hadoop decompression path, add the following to / etc / hadoop / slaves
master
slaver1
slaver2
slaver3
slaver4
Deploy slaver1-slaver4
In accordance with the above process, the slaver1-slaver4 on the environment variables added, and then directly
scp -r /Hadoop2.6.4 root@slaverX:/root/workspace/software/
X is 1 – 4, copied to slaver1 – slaver4, respectively
Start hadoop cluster
Format the file system
hdfs namenode -format
Start NameNode and DateNode
/root/workspace/software/hadoop-2.6.4/sbin, run
start-dfs.sh
Use the jps command to view the Java process on the master
[root@master hadoop]# jps
27130 DataNode
27927 NameNode
12379 Jps
27422 SecondaryNameNode
[root@master hadoop]#
Jps command to view the Java processes on slaver1 – slaver4, respectively
[root@slaver1 hadoop]# jps
6130 DataNode
1264 Jps
View NameNode and NameNode information
Browser input
IP:50070/dfshealth.html#tab-datanode
IP is the IP of your cluster
Start the ResourceManager and NodeManager
Run start-yarn.sh, jps view process is as follows:
[root@master hadoop]# jps
27130 DataNode
28777 ResourceManager
27927 NameNode
12379 Jps
28916 NodeManager
27422 SecondaryNameNode
Switch to slaver1-slaver4, jps view process
[root@slaver1 hadoop]# jps
27130 DataNode
12379 Jps
28916 NodeManager
Succeeded
Hadoop cluster has already started.
Recent Comments