mv ifcfg-eno16777736 ifcfg-eth0
\nvi \/etc\/udev\/rules.d\/90-eno-fix.rules
\n# This file was automatically generated on systemd update
\nSUBSYSTEM==”net”, ACTION==”add”, DRIVERS==”?*”, ATTR{address}==”00:0c:29:9e:8f:95″, NAME=”eno16777736″<\/p>\n
# This file was automatically generated on systemd update
\nSUBSYSTEM==”net”, ACTION==”add”, DRIVERS==”?*”, ATTR{address}==”00:0c:29:9e:8f:95″, NAME=”eth0″<\/p>\n
<\/p>\n
vi \/etc\/hosts<\/p>\n
192.168.1.80 master.rmohan.com master
\n192.168.1.81 lab1.rmohan.com lab1
\n192.168.1.82 lab2.rmohan.com lab2<\/p>\n
Architecture<\/p>\n
IP Address Hostname Role
\n192.168.1.80 master NameNode, ResourceManager
\n192.168.1.81 slave1 SecondaryNameNode, DataNode, NodeManager
\n192.168.1.82 slave2 DataNode, NodeManager<\/p>\n
<\/p>\n
Before we start, we will understand the meaning of the following:<\/p>\n
DataNode<\/strong><\/p>\n A DataNode stores data in the Hadoop File System. A functional file system has more than one DataNode, with the data replicated across them.<\/p>\n NameNode<\/strong><\/p>\n The NameNode is the centrepiece of an HDFS file system. It keeps the directory of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data of these file itself.<\/p>\n NodeManager<\/strong><\/p>\n The NodeManager (NM) is YARN’s per-node agent, and takes care of the individual compute nodes in a Hadoop cluster. This includes keeping up-to date with the ResourceManager (RM), overseeing containers’ life-cycle management; monitoring resource usage (memory, CPU) of individual containers, tracking node-health, log’s management and auxiliary services which may be exploited by different YARN applications.<\/p>\n ResourceManager<\/strong><\/p>\n ResourceManager (RM) is the master that arbitrates all the available cluster resources and thus helps manage the distributed applications running on the YARN system. It works together with the per-node NodeManagers (NMs) and the per-application ApplicationMasters (AMs).<\/p>\n Secondary Namenode<\/strong><\/p>\n Secondary Namenode whole purpose is to have a checkpoint in HDFS. It is just a helper node for namenode.<\/p>\n <\/p>\n <\/p>\n HDFS (Hadoop distributed file system)<\/p>\n GFS paper from Google, published in October 2003, HDFS is GFS clone version.<\/p>\n Is the basis of data storage management in the Hadoop system. It is a highly fault tolerant system that can detect and respond to hardware failures for running on low-cost general-purpose hardware. HDFS simplifies the file consistency model, through streaming data access, providing high-throughput application data access capabilities for applications with large data sets.<\/p>\n HDFS this part of the main part of the composition of a few:<\/p>\n (1), Client: cut file; access HDFS; interact with NameNode, get file location information; interact with DataNode, read and write data.<\/p>\n (2), NameNode: Master node, only one in hadoop1.X, manage the HDFS name space and data block mapping information, configure the replica policy, handle the client request. For large clusters, Hadoop1.x has two of the biggest flaws: 1) for large clusters, namenode memory becomes a bottleneck, namenode’s scalability problem; 2) namenode’s single point of failure problem.<\/p>\n In response to the above two defects, Hadoop2.x later on these two issues were resolved. For the defect 1) proposed Federation namenode to solve, the program is mainly through multiple namenode to achieve multiple namespace to achieve the namingode horizontal expansion. Thereby alleviating the problem of a single namenode memory.<\/p>\n For defect 2), hadoop2.X proposed to achieve two namenode implementation of hot standby HA solution to solve. One of which is in the standby state, one in the active state.<\/p>\n (3), DataNode: Slave node, store the actual data, report the stored information to the NameNode.<\/p>\n (4), Secondary NameNode: Assist NameNode, share its workload; regularly merge fsimage and edits, push to NameNode; in case of emergency, can help restore NameNode, but Secondary NameNode is not a HotDress for NameNode.<\/p>\n At present, the hard disk is not bad, we can through secondarynamenode to achieve namenode recovery.<\/p>\n 3, Mapreduce (distributed computing framework)<\/p>\n Source from Google’s MapReduce thesis, published in December 2004, Hadoop MapReduce is google MapReduce Clone Edition. MapReduce is a computational model used to calculate large amounts of data. Where Map specifies the operation of the independent elements on the dataset to generate key-value pairs of intermediate results. Reduce is the specification of all the “values” of the same “key” in the intermediate result to get the final result. MapReduce such a functional division, is very suitable for a large number of computers in a distributed parallel environment for data processing.<\/p>\n MapReduce calculation framework to the development of two versions of MapReduce API now, for MR1 main components have the following components:<\/p>\n (1), JobTracker: Master node, only one, the main task is the allocation of resources and job scheduling and supervision and management, management of all operations, job \/ task monitoring, error handling, etc .; the task is broken down into a series of tasks, and assigned to TaskTracker.<\/p>\n (2), TaskTracker: Slave node, run Map Task and Reduce Task; and interact with JobTracker, reporting task status.<\/p>\n (3), Map Task: parse each data record, passed to the user to write the map (), and the implementation of the output will be written to the local disk.<\/p>\n (4), Reducer Task: from the implementation of the results of the Map Task, remote read the input data, sort the data, the data passed to the user to write the reduce function implementation.<\/p>\n In this process, there is a shuffle process, which is the key to understanding the MapReduce calculation framework for the process. The process contains the map function output to the reduce function to enter all the operations in this intermediate process, called the shuffle process. In this process, can be divided into the map side and reduce end.<\/p>\n Map side:<\/p>\n 1) After the input data is fragmented, the size of the slice is related to the size of the original file, the size of the file block. Each piece corresponds to a map task.<\/p>\n 2) map task in the implementation process, the results will be stored into memory, when the memory occupies a certain threshold (the threshold can be set), the map will be the middle of the results written to the local disk, the formation of temporary File This process is called overwriting.<\/p>\n 3) map In the process of overflow, will be specified according to the number of designated tasks to reduce the corresponding partition, which is the partition process. Each partition corresponds to a reduce task. And in the process of writing, the corresponding sort. In the process of overflow can also set the conbiner process, the process with the results of the reduction should be consistent, so the application of the process there is a certain limit, need to be used with caution.<\/p>\n 4) At the end of each map, there is only one temporary file as input to reduce, so the Merge operations are merged into multiple temporary files that are overwritten to disk. Finally form a temporary file of an internal partition.<\/p>\n Reduce end:<\/p>\n 1) first to achieve data localization, the need to remote node on the map output to the local copy.<\/p>\n 2) Merge process, the merger process is mainly on the different nodes on the map output results are combined.<\/p>\n 3) continue to copy and merge, the final form of an input file. Reduce stores the final results on HDFS.<\/p>\n For MR2 is the new generation of MR API. It is mainly run on Yarn’s resource management framework.<\/p>\n 4, Yarn (resource management framework)<\/p>\n The framework was hadoop2.x later after the optimization of the JobTracker and TaskTracker models before hadoop1.x, which resulted in the separation of the JobTracker’s resource allocation and job scheduling and oversight. The framework of the main ResourceManager, Applicationmatser, nodemanager. The main work process is as follows: the ResourceManager is mainly responsible for all the application of resource allocation, ApplicationMaster is mainly responsible for the task scheduling of each job, that is, each job corresponds to an ApplicationMaster. Nodemanager is a command that receives Resourcemanager and ApplicationMaster to implement the allocation of resources.<\/p>\n The ResourceManager allocates a Conbiner after receiving the client’s job submission request. Here it is to be noted that the Resoucemanager allocation resource is allocated in units of Conbiner. The first assigned Conbiner starts Applicationmaster, which is primarily responsible for job scheduling. After the Applicationmanager starts, it communicates directly with the NodeManager.<\/p>\n In YARN, resource management is done by the ResourceManager and the NodeManager, where the scheduler in the ResourceManager is responsible for resource allocation and the NodeManager is responsible for resource provisioning and isolation. ResourceManager assigns resources on a NodeManager to a task (this is the so-called “resource scheduling”), the NodeManager needs to provide the appropriate resources for the task as required, and even ensure that these resources should be exclusive, provide the basis for the task of guarantee, This is the so-called resource isolation.<\/p>\n In the Yarn platform can run multiple computing framework, such as: MR, Tez, Storm, Spark and other calculations, the framework.<\/p>\n 5, Sqoop (data synchronization tool)<\/p>\n Sqoop is an abbreviation for SQL-to-Hadoop, which is used primarily for transferring data between a legacy database and Hadoop. Data import and export is essentially a Mapreduce program that takes full advantage of MR parallelism and fault tolerance. Which is the main use of the MP Map task to achieve parallel import, export. Sqoop development has now appeared in two versions, one is sqoop1.xx series, one is sqoop1.99.X series. For the sqoop1 series, mainly through the command line to operate.<\/p>\n Sqoop1 import principle: from the traditional database to obtain metadata information (schema, table, field, field type), the import function is converted to Map Map only map operations, mapreduce a lot of map, each map read a piece of data, and then parallel Complete the copy of the data.<\/p>\n Sqoop1 export Principle: Get the schema of the exported table, meta information, and the field match in Hadoop; multiple map only jobs run at the same time, and the data in hdfs is exported to the relational database.<\/p>\n Sqoop1.99.x is a product of sqoop2, which is not fully functional products, in a test phase, generally will not be used in commercial products.<\/p>\n Sqoop tools, the current understanding of it is that there may be some problems because when the import and export, map task failed, then Applicationmaster will re-schedule another task to run this failed task. But there may be a problem that the data imported by the Map task before the failure and the result of the re-scheduling of the map task will be repeated.<\/p>\n 6, Mahout (data mining algorithm library)<\/p>\n Mahout originated in 2008, initially Apache Lucent subproject, which in a very short period of time has made considerable development, is now Apache’s top project. Compared with the traditional MapReduce programming method to achieve the machine learning algorithm, often need to spend a lot of development time, and the cycle is longer, and Mahout’s main goal is to create some scalable machine learning domain classic algorithm to achieve Developers are more convenient and quick to create intelligent applications.<\/p>\n Mahout now includes a wide range of data mining methods such as clustering, classification, recommendation engine (collaborative filtering), and frequent set mining. In addition to the algorithm, Mahout also includes data input \/ output tools, data mining support architectures such as integration with other storage systems such as databases, MongoDB or Cassandra.<\/p>\n Mahout the following components will generate the corresponding jar package. At this point we need to understand a question: how to use mahout in the end?<\/p>\n In fact, mahout is just a machine learning algorithm library, in which the corresponding machine learning algorithm, such as: recommended system (including user-based and object-based recommendations), clustering and classification algorithm. And some of these algorithms to achieve the MapReduce, spark can be run on the hadoop platform, in the actual development process, only the corresponding jar package can be.<\/p>\n 7, Hbase (distributed inventory database)<\/p>\n Bigtable papers from Google, published in November 2006, the traditional relational database is a row-oriented database. HBase is a Google Bigtable clone that HBase is a scalable, highly reliable, high-performance, distributed and column-oriented dynamic schema database for structured data. Unlike traditional relational databases, HBase uses a BigTable data model: an enhanced sparse sorting table (Key \/ Value), where the keys consist of row keywords, column keywords, and timestamps. HBase provides random, real-time read and write access to large-scale data, while data stored in HBase can be processed using MapReduce, which combines data storage with parallel computing.<\/p>\n Hbase table features<\/p>\n 1), large: a table can have billions of rows, millions of columns;<\/p>\n 2), no model: each row has a sortable primary key and any number of columns, columns can be dynamically increased according to the need, the same table can have different rows of different columns;<\/p>\n 3), oriented columns: oriented column (family) storage and permission control, column (family) independent retrieval;<\/p>\n 4), sparse: null (null) column does not take up storage space, the table can be designed very sparse;<\/p>\n 5), multi-version data: each unit of data can have multiple versions, by default, the version number is automatically assigned, is the cell when the timestamp inserted;<\/p>\n 6), data type single: Hbase in the data are strings, no type.<\/p>\n Hbase physical model<\/p>\n Each column family is stored in a separate file on HDFS, and null values ??are not saved.<\/p>\n Key and Version number have one in each column family;<\/p>\n HBase maintains a multi-level index for each value, ie, its physical storage:<\/p>\n 1, all the rows in the table are sorted by the row key;<\/p>\n 2, Table in the direction of the line is divided into multiple Regions;<\/p>\n 3, Region by size, each table began with only one region, with the data increased, region increasing, when increased to a threshold when the region will be divided into two new regions, then there will be More and more region;<\/p>\n 4, Region is the smallest unit of distributed storage and load balancing in Hbase, and different regions are distributed to different RegionsServer. ,<\/p>\n 5, Region is the smallest unit of distributed storage, but not the smallest unit of storage. Region consists of one or more stores, each store holds a columns family; each Strore is made up of a memStore and 0 to multiple StoreFiles, StoreFile contains HFile; memStore is stored in memory and StoreFile is stored on HDFS.<\/p>\n 8, Zookeeper (distributed collaborative service)<\/p>\n From the Google Chubby paper, published in November 2006, Zookeeper is Chubby clone version, mainly to solve the distributed environment of data management issues: unified naming, state synchronization, cluster management, configuration synchronization.<\/p>\n Zookeeper’s main implementation is two steps: 1), election Leader 2), sync data. This component needs to be used when implementing HA high availability for namenode.<\/p>\n 9, Pig (Hadoop-based data flow system)<\/p>\n By yahoo! Open source, the design motivation is to provide a MapReduce-based ad-hoc (calculated at query time) data analysis tools<\/p>\n Defines a data flow language – Py Latin, which converts scripts into MapReduce tasks on Hadoop. Usually used for off-line analysis.<\/p>\n 10, Hive (Hadoop-based data warehouse)<\/p>\n Open source by facebook, initially used to solve the massive structure of the log data statistics.<\/p>\n Hive defines a SQL-like query language (HQL) that transforms SQL into a MapReduce task on Hadoop. Usually used for off-line analysis.<\/p>\n 11, Flume (log collection tool)<\/p>\n Cloudera open source log collection system, with distributed, highly reliable, high fault tolerance, easy to customize and expand the characteristics.<\/p>\n 3.1. install necessary packages for OS<\/p>\n We pick up CentOS minimal ISO as our installation prototype, once the system installed, we need several more basic packages:<\/p>\n yum install -y net-tools 3.2. setup hostname for all nodes<\/p>\n This step is optional, but important for better self-identify while you use same username to walk through different nodes.<\/p>\n hostnamectl set-hostname master<\/p>\n ex: at master node<\/p>\n re-login to check the effect<\/p>\n 3.3. setup jdk for all nodes<\/p>\n install jdk from oracle official website<\/p>\n wget –header “Cookie: oraclelicense=accept-securebackup-cookie” http:\/\/download.oracle.com\/otn-pub\/java\/jdk\/8u121-b13\/e9e7ea248e2c4826b92b3f075a80e441\/jdk-8u121-linux-x64.rpm [root@master ~]# yum localinstall -y jdk-8u121-linux-x64.rpm Dependencies Resolved<\/p>\n ========================================================================================================================================================================= Transaction Summary Total size: 263 M Installed: Complete! export JAVA_HOME=\/usr\/java\/jdk1.8.0_121 Approach to verification:<\/p>\n java -version [root@master ~]# update-alternatives –config java<\/p>\n There is 1 program that provides ‘java’.<\/p>\n Selection Command 3.4. setup user and user group on all nodes<\/p>\n groupadd hadoop echo ‘192.168.1.80 master.rmohan.com master’ >> \/etc\/hosts ping master 3.6. setup ssh no password login on all nodes<\/p>\n su – hadoop 3.7. stop & disable firewall<\/p>\n systemctl stop firewalld.service P.S. the whole Step 4 operations happens on a single node, let\u2019s say, master. In addition, we\u2019ll login as user hadoop to finish all operations.<\/p>\n su – hadoop 1% [> ] 2,932,536 811KB\/s eta 4m 24s<\/p>\n tar -zxvf hadoop-2.7.2.tar.gz append following content onto ~\/.bashrc<\/p>\n export HADOOP_HOME=\/home\/hadoop\/hadoop-2.7.2 source ~\/.bashrc Add slave node hostnames into $HADOOP_HOME\/etc\/hadoop\/slaves file <configuration> Modify $HADOOP_HOME\/etc\/hadoop\/yarn-site.xml as following mkdir -p $HADOOP_HOME\/tmp scp ~\/.bashrc slave1:~\/ scp -r ~\/hadoop-2.7.2 slave1:~\/ Format namenode for the first time launch<\/p>\n hdfs namenode -format [hadoop@master ~]$ start-dfs.sh Launch yarn distributed computing system [hadoop@master ~]$ start-yarn.sh Shutdown Hadoop Cluster<\/p>\n stop-yarn.sh 6.1. Verify by jps processus<\/p>\n Check jps on each node, and view results.<\/p>\n [hadoop@master ~]$ jps On slave1 node: On slave2 node: 6.2. Verify on Web interface<\/p>\n 6.2. Verify on Web interface<\/p>\n 192.168.1.80:50070 to view hdfs storage status. 7. End<\/p>\n This is all about basic version of hadoop 3 nodes cluster, for high availability version & hadoop relative eco-systems,<\/p>\n I\u2019ll give it on other posts, thanks for contacting me if there is anything mistype or you have any suggestions or anything you don\u2019t understand.<\/p>\n <\/p>\n <\/p>\n How do I add a new node to a Hadoop cluster?<\/strong><\/p>\n 1. Install a new node at hadoop or copy it from another node<\/p>\n 2. Copy the namingode configuration file to the node<\/p>\n 3. Modify the masters and slaves files, increase the node, all nodes have to modify<\/p>\n 4. Set the ssh password to access the node<\/p>\n 5. Start the datanode and tasktracker (hadoop-daemon.sh start datanode \/ tasktracker) on the node separately<\/p>\n 6. Run start-balancer.sh for data load balancing<\/p>\n","protected":false},"excerpt":{"rendered":" CentOS 7.3 under the Hadoop 2.7.2 cluster<\/p>\n how to setup a Hadoop cluster on CentOS linux system. Before you read this article, I assume you already have all basic conceptions about Hadoop and Linux operating system.<\/p>\n mv ifcfg-eno16777736 ifcfg-eth0 vi \/etc\/udev\/rules.d\/90-eno-fix.rules # This file was automatically generated on systemd update SUBSYSTEM==”net”, ACTION==”add”, DRIVERS==”?*”, ATTR{address}==”00:0c:29:9e:8f:95″, NAME=”eno16777736″<\/p>\n […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[73,60],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/6634"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=6634"}],"version-history":[{"count":6,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/6634\/revisions"}],"predecessor-version":[{"id":6645,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/6634\/revisions\/6645"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=6634"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=6634"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=6634"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"\"](\"http:\/\/rmohan.com\/wp-content\/uploads\/2017\/04\/Hadoop.png\")
<\/a> ![\"\"](\"http:\/\/rmohan.com\/wp-content\/uploads\/2017\/04\/hdfsarchitecture.png\")
<\/a>It abstracts the data from the process of generating, transmitting, processing, and ultimately writing to the target path as a data stream. The data source supports the custom data sender in Flume to support the collection of various protocol data. At the same time, Flume data stream provides the ability to process log data easily, such as filtering, format conversion and so on. In addition, Flume also has the ability to write logs to various data targets (customizable). In general, Flume is a scalable, complex environment for the massive log collection system.<\/p>\n
\nyum install -y openssh-server
\nyum install -y wget
\nyum install -y ntpd
\nsystemctl enable ntpd ; systemctl start ntpd
\nntpdate -u 0.centos.pool.ntp.org
\nThe first line is to install ifconfig, while the second one is to be able to be ssh login by remote peer.<\/p>\n
\nyum localinstall -y jdk-8u121-linux-x64.rpm
\nrm jdk-8u121-linux-x64.rpm
\nadd java.sh under \/etc\/profile.d\/<\/p>\n
\nLoaded plugins: fastestmirror
\nExamining jdk-8u121-linux-x64.rpm: 2000:jdk1.8.0_121-1.8.0_121-fcs.x86_64
\nMarking jdk-8u121-linux-x64.rpm to be installed
\nResolving Dependencies
\n–> Running transaction check
\n—> Package jdk1.8.0_121.x86_64 2000:1.8.0_121-fcs will be installed
\n–> Finished Dependency Resolution<\/p>\n
\nPackage Arch Version Repository Size
\n=========================================================================================================================================================================
\nInstalling:
\njdk1.8.0_121 x86_64 2000:1.8.0_121-fcs \/jdk-8u121-linux-x64 263 M<\/p>\n
\n=========================================================================================================================================================================
\nInstall 1 Package<\/p>\n
\nInstalled size: 263 M
\nDownloading packages:
\nRunning transaction check
\nRunning transaction test
\nTransaction test succeeded
\nRunning transaction
\nInstalling : 2000:jdk1.8.0_121-1.8.0_121-fcs.x86_64 1\/1
\nUnpacking JAR files…
\ntools.jar…
\nplugin.jar…
\njavaws.jar…
\ndeploy.jar…
\nrt.jar…
\njsse.jar…
\ncharsets.jar…
\nlocaledata.jar…
\nVerifying : 2000:jdk1.8.0_121-1.8.0_121-fcs.x86_64 1\/1<\/p>\n
\njdk1.8.0_121.x86_64 2000:1.8.0_121-fcs<\/p>\n
\n[root@master ~]#
\njava.sh content:<\/p>\n
\nexport JRE_HOME=\/usr\/java\/jdk1.8.0_121\/jre
\nexport CLASSPATH=$JAVA_HOME\/lib:.
\nexport PATH=$PATH:$JAVA_HOME\/bin
\nre-login, and you\u2019ll find all environment variables, and java is well installed.<\/p>\n
\nls $JAVA_HOME
\necho $PATH
\nif the java version goes wrong, you can<\/p>\n
\n———————————————–
\n*+ 1 \/usr\/java\/jdk1.8.0_121\/jre\/bin\/java<\/p>\n
\nuseradd -d \/home\/hadoop -g hadoop hadoop
\npasswd hadoop
\n3.5. modify hosts file for network inter-recognition on all nodes<\/p>\n
\necho ‘192.168.1.81 lab1.rmohan.com lab1 slave1’ >> \/etc\/hosts
\necho ‘192.168.1.82 lab1.rmohan.com lab2 slave2’ >> \/etc\/hosts
\ncheck the recognition works:<\/p>\n
\nping slave1
\nping slave2<\/p>\n
\nssh-keygen -t rsa
\nssh-copy-id master
\nssh-copy-id slave1
\nssh-copy-id slave2
\nnow you can ssh login to all 3 nodes without passwd, please have a try to check it out.<\/p>\n
\nsystemctl disable firewalld.service
\n4. Hadoop Setup<\/p>\n
\n4.1. Download and untar on the file system.
\n[hadoop@master ~]$ wget http:\/\/mirrors.sonic.net\/apache\/hadoop\/common\/hadoop-2.7.2\/hadoop-2.7.2.tar.gz
\n–2017-04-12 22:49:01– http:\/\/mirrors.sonic.net\/apache\/hadoop\/common\/hadoop-2.7.2\/hadoop-2.7.2.tar.gz
\nResolving mirrors.sonic.net (mirrors.sonic.net)… 69.12.162.27
\nConnecting to mirrors.sonic.net (mirrors.sonic.net)|69.12.162.27|:80… connected.
\nHTTP request sent, awaiting response… 200 OK
\nLength: 212046774 (202M) [application\/x-gzip]
\nSaving to: \u2018hadoop-2.7.2.tar.gz\u2019<\/p>\n
\nrm hadoop-2.7.2.tar.gz
\nchmod 775 hadoop-2.7.2
\n4.2. Add environment variables for hadoop<\/p>\n
\nexport HADOOP_INSTALL=$HADOOP_HOME
\nexport HADOOP_MAPRED_HOME=$HADOOP_HOME
\nexport HADOOP_COMMON_HOME=$HADOOP_HOME
\nexport HADOOP_HDFS_HOME=$HADOOP_HOME
\nexport YARN_HOME=$HADOOP_HOME
\nexport HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME\/lib\/native
\nexport PATH=$PATH:$HADOOP_HOME\/sbin:$HADOOP_HOME\/bin
\nthen make these variables effect:<\/p>\n
\n4.3. Modify configuration files for hadoop<\/p>\n
\necho slave1 > $HADOOP_HOME\/etc\/hadoop\/slaves
\necho slave2 >> $HADOOP_HOME\/etc\/hadoop\/slaves
\nAdd secondary node hostname into $HADOOP_HOME\/etc\/hadoop\/masters file
\necho slave1 > $HADOOP_HOME\/etc\/hadoop\/masters
\nModify $HADOOP_HOME\/etc\/hadoop\/core-site.xml as following
\n<configuration>
\n<property>
\n<name>fs.defaultFS<\/name>
\n<value>hdfs:\/\/master:9000\/<\/value>
\n<description>namenode settings<\/description>
\n<\/property>
\n<property>
\n<name>hadoop.tmp.dir<\/name>
\n<value>\/home\/hadoop\/hadoop-2.7.2\/tmp\/hadoop-${user.name}<\/value>
\n<description> temp folder <\/description>
\n<\/property>
\n<property>
\n<name>hadoop.proxyuser.hadoop.hosts<\/name>
\n<value>*<\/value>
\n<\/property>
\n<property>
\n<name>hadoop.proxyuser.hadoop.groups<\/name>
\n<value>*<\/value>
\n<\/property>
\n<\/configuration>
\nModify $HADOOP_HOME\/etc\/hadoop\/hdfs-site.xml as following
\n<configuration>
\n<property>
\n<name>dfs.namenode.http-address<\/name>
\n<value>master:50070<\/value>
\n<description> fetch NameNode images and edits <\/description>
\n<\/property>
\n<property>
\n<name>dfs.namenode.secondary.http-address<\/name>
\n<value>slave1:50090<\/value>
\n<description> fetch SecondNameNode fsimage <\/description>
\n<\/property>
\n<property>
\n<name>dfs.replication<\/name>
\n<value>2<\/value>
\n<description> replica count <\/description>
\n<\/property>
\n<property>
\n<name>dfs.namenode.name.dir<\/name>
\n<value>file:\/\/\/home\/hadoop\/hadoop-2.7.2\/hdfs\/name<\/value>
\n<description> namenode <\/description>
\n<\/property>
\n<property>
\n<name>dfs.datanode.data.dir<\/name>
\n<value>file:\/\/\/home\/hadoop\/hadoop-2.7.2\/hdfs\/data<\/value>
\n<description> DataNode <\/description>
\n<\/property>
\n<property>
\n<name>dfs.namenode.checkpoint.dir<\/name>
\n<value>file:\/\/\/home\/hadoop\/hadoop-2.7.2\/hdfs\/namesecondary<\/value>
\n<description> check point <\/description>
\n<\/property>
\n<property>
\n<name>dfs.webhdfs.enabled<\/name>
\n<value>true<\/value>
\n<\/property>
\n<property>
\n<name>dfs.stream-buffer-size<\/name>
\n<value>131072<\/value>
\n<description> buffer <\/description>
\n<\/property>
\n<property>
\n<name>dfs.namenode.checkpoint.period<\/name>
\n<value>3600<\/value>
\n<description> duration <\/description>
\n<\/property>
\n<\/configuration>
\nModify $HADOOP_HOME\/etc\/hadoop\/mapred-site.xml as following<\/p>\n
\n<property>
\n<name>mapreduce.framework.name<\/name>
\n<value>yarn<\/value>
\n<\/property>
\n<property>
\n<name>mapreduce.jobtracker.address<\/name>
\n<value>hdfs:\/\/trucy:9001<\/value>
\n<\/property>
\n<property>
\n<name>mapreduce.jobhistory.address<\/name>
\n<value>master:10020<\/value>
\n<description>MapReduce JobHistory Server host:port, default port is 10020.<\/description>
\n<\/property>
\n<property>
\n<name>mapreduce.jobhistory.webapp.address<\/name>
\n<value>master:19888<\/value>
\n<description>MapReduce JobHistory Server Web UI host:port, default port is 19888.<\/description>
\n<\/property>
\n<\/configuration><\/p>\n
\n<configuration>
\n<property>
\n<name>yarn.resourcemanager.hostname<\/name>
\n<value>master<\/value>
\n<\/property>
\n<property>
\n<name>yarn.nodemanager.aux-services<\/name>
\n<value>mapreduce_shuffle<\/value>
\n<\/property>
\n<property>
\n<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class<\/name>
\n<value>org.apache.hadoop.mapred.ShuffleHandler<\/value>
\n<\/property>
\n<property>
\n<name>yarn.resourcemanager.address<\/name>
\n<value>master:8032<\/value>
\n<\/property>
\n<property>
\n<name>yarn.resourcemanager.scheduler.address<\/name>
\n<value>master:8030<\/value>
\n<\/property>
\n<property>
\n<name>yarn.resourcemanager.resource-tracker.address<\/name>
\n<value>master:8031<\/value>
\n<\/property>
\n<property>
\n<name>yarn.resourcemanager.admin.address<\/name>
\n<value>master:8033<\/value>
\n<\/property>
\n<property>
\n<name>yarn.resourcemanager.webapp.address<\/name>
\n<value>master:8088<\/value>
\n<\/property>
\n<\/configuration>
\n4.4. Create necessary folders<\/p>\n
\nmkdir -p $HADOOP_HOME\/hdfs\/name
\nmkdir -p $HADOOP_HOME\/hdfs\/data
\n4.5. Copy hadoop folders and environment settings to slaves<\/p>\n
\nscp ~\/.bashrc slave2:~\/<\/p>\n
\nscp -r ~\/hadoop-2.7.2 slave2:~\/
\n5. Launch hadoop cluster service<\/p>\n
\nLaunch dfs distributed file system
\nstart-dfs.sh<\/p>\n
\n17\/04\/12 23:27:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
\nStarting namenodes on [master]
\nThe authenticity of host ‘master (192.168.1.80)’ can’t be established.
\nECDSA key fingerprint is d8:55:ea:50:b3:bb:8a:fc:90:a2:0e:54:3e:79:60:bc.
\nAre you sure you want to continue connecting (yes\/no)? yes
\nmaster: Warning: Permanently added ‘master,192.168.1.80’ (ECDSA) to the list of known hosts.
\nhadoop@master’s password:
\nmaster: starting namenode, logging to \/home\/hadoop\/hadoop-2.7.2\/logs\/hadoop-hadoop-namenode-master.rmohan.com.out
\nslave2: starting datanode, logging to \/home\/hadoop\/hadoop-2.7.2\/logs\/hadoop-hadoop-datanode-lab2.rmohan.com.out
\nslave1: starting datanode, logging to \/home\/hadoop\/hadoop-2.7.2\/logs\/hadoop-hadoop-datanode-lab1.rmohan.com.out
\nStarting secondary namenodes [slave1]
\nslave1: starting secondarynamenode, logging to \/home\/hadoop\/hadoop-2.7.2\/logs\/hadoop-hadoop-secondarynamenode-lab1.rmohan.com.out
\n17\/04\/12 23:27:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable<\/p>\n
\nstart-yarn.sh<\/p>\n
\nstarting yarn daemons
\nstarting resourcemanager, logging to \/home\/hadoop\/hadoop-2.7.2\/logs\/yarn-hadoop-resourcemanager-master.rmohan.com.out
\nslave2: starting nodemanager, logging to \/home\/hadoop\/hadoop-2.7.2\/logs\/yarn-hadoop-nodemanager-lab2.rmohan.com.out
\nslave1: starting nodemanager, logging to \/home\/hadoop\/hadoop-2.7.2\/logs\/yarn-hadoop-nodemanager-lab1.rmohan.com.out
\n[hadoop@master ~]$<\/p>\n
\nstop-dfs.sh
\n6. Verify the hadoop cluster is up and healthy<\/p>\n
\n3184 ResourceManager
\n3441 Jps
\n2893 NameNode
\n[hadoop@master ~]$<\/p>\n
\n[hadoop@lab1 ~]$ jps
\n3026 NodeManager
\n3127 Jps
\n2811 DataNode
\n2907 SecondaryNameNode
\n[hadoop@lab1 ~]$<\/p>\n
\n[hadoop@lab2 ~]$ jps
\n2722 DataNode
\n2835 NodeManager
\n2934 Jps
\n[hadoop@lab2 ~]$<\/p>\n
\n192.168.1.80:8088 to view yarn computing system resources and application status.<\/p>\n<\/a> <\/a> <\/a><\/p>\n