{"id":6786,"date":"2017-06-21T12:48:10","date_gmt":"2017-06-21T04:48:10","guid":{"rendered":"http:\/\/rmohan.com\/?p=6786"},"modified":"2017-06-21T12:48:10","modified_gmt":"2017-06-21T04:48:10","slug":"hadoop-2-6-4-distributed-cluster-environment-is-built-on-the-actual-cluster","status":"publish","type":"post","link":"https:\/\/mohan.sg\/?p=6786","title":{"rendered":"Hadoop 2.6.4 distributed cluster environment is built on the actual cluster ."},"content":{"rendered":"

Hadoop 2.6.4 distributed cluster environment is built on the actual cluster .<\/strong><\/p>\n

Cluster preparation<\/p>\n

There are five machines that have changed the machine name for master, slaver1, slaver2, slaver3, slaver4, and set the face password ssh login. You can refer to here http:\/\/www.linuxidc.com\/Linux\/2016-02\/128149.htm<\/p>\n

Machine name\tIp
\nMaster\t192.168.1.1
\nSlaver1\t192.168.1.2
\nSlaver2\t192.168.1.3
\nSlaver3\t192.168.1.4
\nSlaver4\t192.168.1.5
\nInstall the JDK<\/p>\n

CentOS 7 is openJDK by default<\/p>\n

Uninstall the openJDK under CentOS 7 and install Sun JDK1.7<\/p>\n

View the openJDK installation path<\/p>\n

rpm -qa | grep java<\/p>\n

Uninstall openJDK<\/p>\n

rpm -e –nodeps java-1.7.0-openjdk-1.7.0.85-2.6.1.2.el7_1.x86_64
\n rpm -e –nodeps java-1.8.0-openjdk-devel-1.8.0.60-2.b27.el7_1.x86_64
\n rpm -e –nodeps java-1.6.0-openjdk-1.6.0.36-1.13.8.1.el7_1.x86_64
\n rpm -e –nodeps java-1.6.0-openjdk-devel-1.6.0.36-1.13.8.1.el7_1.x86_64<\/p>\n

Install Sun JDK1.7<\/p>\n

Download from the official website jdk-7u80-linux-x64.rpm, upload to master<\/p>\n

Install Sun JDK1.7<\/p>\n

rpm -ivh jdk-7u80-linux-x64.rpm<\/p>\n

Modify environment variables<\/p>\n

In \/etc\/profile add in<\/p>\n

export JAVA_HOME=\/usr\/java\/jdk1.7.0_80
\nexport CLASSPATH=.:$JAVA_HOME\/jre\/lib\/rt.jar:$JAVA_HOME\/lib\/dt.jar:$JAVA_HOME\/lib\/tools.jar
\nexport PATH=$PATH:$JAVA_HOME\/bin<\/p>\n

source \/etc\/profile<\/p>\n

Install Hadoop 2.6.4<\/p>\n

Download, extract<\/p>\n

Hadoop 2.6.4 downloaded from the official website and decompress on the master
\nextraction path of their choice, here is my decompression in<\/p>\n

\/root\/workspace\/software\/hadoop-2.6.4<\/p>\n

Add environment variable<\/p>\n

In \/etc\/profileadd in<\/p>\n

export HADOOP_HOME=\/root\/workspace\/software\/hadoop-2.6.4\/
\nexport PATH=$PATH:$HADOOP_HOME\/bin:$HADOOP_HOME\/sbin
\nModify the Hadoop configuration file<\/p>\n

Hadoop-env.sh<\/p>\n

In the hadoop decompression path below, \/ etc \/ hadoop \/ hadoop-env.sh add the following two lines<\/p>\n

export JAVA_HOME=\/usr\/java\/jdk1.7.0_80
\nexport HADOOP_PREFIX=\/root\/workspace\/software\/hadoop-2.6.4
\nCore-site.xml<\/p>\n

Under the hadoop decompression path, add the following to \/ etc \/ hadoop \/ core-site.xml<\/p>\n\n fs.defaultFS<\/name>
\n hdfs:\/\/master:9000\/<\/value>\n <\/property>\n \n hadoop.tmp.dir<\/name>
\n \/root\/workspace\/software\/hadoop-2.6.4\/tmp<\/value>\n <\/property>\n<\/configuration>
\nHdfs-site.xml<\/p>\n

In the hadoop decompression path below, \/ etc \/ hadoop \/ hdfs-site.xml add the following<\/p>\n

\n \n dfs.replication<\/name>
\n 3<\/value>\n <\/property>\n<\/configuration>
\nSet here to 3, indicating that the data has 3 copies.<\/p>\n

Mapred-site.xml<\/p>\n

Under the hadoop decompression path, add the following to \/ etc \/ hadoop \/ mapred-site.xml<\/p>\n

\n \n mapreduce.framework.name<\/name>
\n yarn<\/value>\n <\/property>\n<\/configuration>
\nYarn-env.sh<\/p>\n

In the hadoop decompression path below, \/ etc \/ hadoop \/ yarn-env.sh increase below, increase the Java-HOME configuration<\/p>\n

export JAVA_HOME=\/usr\/java\/jdk1.7.0_80
\nYarn-site.xml<\/p>\n

Under the hadoop decompression path, add the following to \/ etc \/ hadoop \/ yarn-site.xml<\/p>\n

\n \n yarn.nodemanager.aux-services<\/name>
\n mapreduce_shuffle<\/value>\n <\/property>\n \n yarn.resourcemanager.hostname<\/name>
\n master<\/value>\n <\/property>\n \n The address of the applications manager interface in the RM.<\/description>
\n Yarn.resourcemanager.address<\/name>
\n master:18040<\/value>\n <\/property>\n\n The address of the scheduler interface.<\/description>
\n Yarn.resourcemanager.scheduler.address<\/name>
\n master:18030<\/value>\n <\/property>\n\n The address of the RM web application.<\/description>
\n Yarn.resourcemanager.webapp.address<\/name>
\n master:18088<\/value>\n <\/property>\n\n The address of the resource tracker interface.<\/description>
\n Yarn.resourcemanager.resource-tracker.address<\/name>
\n master:8025<\/value>\n <\/property>\n<\/configuration>
\nAdd some of the port number, easy to remotely from the browser to view the cluster situation, it is recommended to add in accordance with this.<\/p>\n

Slaves<\/p>\n

Under the hadoop decompression path, add the following to \/ etc \/ hadoop \/ slaves<\/p>\n

master
\nslaver1
\nslaver2
\nslaver3
\nslaver4<\/p>\n

Deploy slaver1-slaver4<\/p>\n

In accordance with the above process, the slaver1-slaver4 on the environment variables added, and then directly<\/p>\n

scp -r \/Hadoop2.6.4 root@slaverX:\/root\/workspace\/software\/
\nX is 1 – 4, copied to slaver1 – slaver4, respectively<\/p>\n

Start hadoop cluster<\/p>\n

Format the file system<\/p>\n

hdfs namenode -format
\nStart NameNode and DateNode<\/p>\n

\/root\/workspace\/software\/hadoop-2.6.4\/sbin, run<\/p>\n

start-dfs.sh
\nUse the jps command to view the Java process on the master<\/p>\n

[root@master hadoop]# jps
\n27130 DataNode
\n27927 NameNode
\n12379 Jps
\n27422 SecondaryNameNode
\n[root@master hadoop]#
\nJps command to view the Java processes on slaver1 – slaver4, respectively<\/p>\n

[root@slaver1 hadoop]# jps
\n6130 DataNode
\n1264 Jps
\nView NameNode and NameNode information<\/p>\n

Browser input<\/p>\n

IP:50070\/dfshealth.html#tab-datanode
\nIP is the IP of your cluster<\/p>\n

Start the ResourceManager and NodeManager<\/p>\n

Run start-yarn.sh, jps view process is as follows:<\/p>\n

[root@master hadoop]# jps
\n27130 DataNode
\n28777 ResourceManager
\n27927 NameNode
\n12379 Jps
\n28916 NodeManager
\n27422 SecondaryNameNode
\nSwitch to slaver1-slaver4, jps view process<\/p>\n

[root@slaver1 hadoop]# jps
\n27130 DataNode
\n12379 Jps
\n28916 NodeManager
\nSucceeded<\/p>\n

Hadoop cluster has already started.<\/p>\n","protected":false},"excerpt":{"rendered":"

Hadoop 2.6.4 distributed cluster environment is built on the actual cluster .<\/p>\n

Cluster preparation<\/p>\n

There are five machines that have changed the machine name for master, slaver1, slaver2, slaver3, slaver4, and set the face password ssh login. You can refer to here http:\/\/www.linuxidc.com\/Linux\/2016-02\/128149.htm<\/p>\n

Machine name Ip Master 192.168.1.1 Slaver1 192.168.1.2 Slaver2 192.168.1.3 Slaver3 192.168.1.4 Slaver4 […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[73],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/6786"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=6786"}],"version-history":[{"count":1,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/6786\/revisions"}],"predecessor-version":[{"id":6787,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/6786\/revisions\/6787"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=6786"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=6786"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=6786"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}