{"id":5593,"date":"2016-02-04T10:09:20","date_gmt":"2016-02-04T02:09:20","guid":{"rendered":"http:\/\/rmohan.com\/?p=5593"},"modified":"2016-02-04T12:43:58","modified_gmt":"2016-02-04T04:43:58","slug":"hdfs-command-syntax","status":"publish","type":"post","link":"https:\/\/mohan.sg\/?p=5593","title":{"rendered":"HDFS Command Syntax"},"content":{"rendered":"

HDFS Command Syntax Overview: <\/b>
\nhadoop fs <\/b><\/code>: <\/code>Ex.: hadoop fs -ls<\/code>\u00a0<\/code><\/code>
\nhadoop version<\/b> : check hadoop installed properly<\/code><\/code>
\n<\/b><\/p>\n

HELP:<\/b>
\nhelp [cmd]<\/b>: hopefully this is self-describing\u00a0<\/b><\/code><\/p>\n

Inspect files:<\/b>
\nls\/lsr : list all files in
\ncat : print on stdout
\ntail [-f] : output the last part of the<\/code>
\ntest : return attributes of file and directory<\/code>
\ntouchz : create new emty file size 0
\ndu\/dus : show space utilization<\/code>
\ncount : no. of directories, files, and bytes<\/code>
\nsetrep : (-r) change the replication factor of file\/directory<\/code>
\nstat : info about the specified path<\/code>
\nCreate\/remove files:<\/b>
\nmkdir : create a directory
\nmv : move (rename) files
\ncp : copy files
\nrm\/rmr : remove files
\nCopy\/Put files from remote m\/c into the HADOOP cluster:<\/b>
\ncopyFromLocal : copy a local file to the HDFS
\ncopyToLocal : copy a file on the HDFS to the local disk<\/code>
\ncp : copies one or more files<\/code>
\nget : copies files to the local file system<\/code>
\nput : copies files from the local file system<\/code>
\nmv : moves one or more files<\/code><\/p>\n

Hadoop Namenode Commands:<\/b>
\nhadoop namenode -format: Format HDFS filesystem from Namenode
\nhadoop namenode -upgrade: Upgrade the NameNode
\nstart-dfs.sh Start: HDFS Daemons
\nstop-dfs.sh Stop: HDFS Daemons
\nstart-mapred.sh: Start: MapReduce Daemons
\nstop-mapred.sh Stop: MapReduce Daemons
\nhadoop namenode -recover -force: Recover namenode metadata after a cluster failure (may lose data)\u00a0<\/code><\/p>\n

Hadoop Configuration Files:<\/b>
\ncore-site.xml : Parameters for entire Hadoop cluster
\nhdfs-site.xml : Parameters for HDFS and its clients
\nmapred-site.xml : Parameters for MapReduce and its clients<\/code>
\nyarn-site.xml : <\/code>Parameters for nodemanager and resource manager<\/code> <\/code>
\nmasters : Host machines for secondary Namenode
\nslaves : List of slave hosts<\/code>
\nhadoop-env.sh : Sets ENV variables for Hadoop\u00a0<\/code><\/code>
\nset JAVA_HOME=%JAVA_HOME%
\nset HADOOP_PREFIX=D:\\Hadoop<\/code><\/b>
\nHadoop Job Commands<\/b>
\nhadoop job -submit : Submit the job
\nhadoop job -status : Print job status completion percentage
\nhadoop job -list all : List all jobs
\nhadoop job -list-active-trackers : List all available TaskTrackers
\nhadoop job -set-priority : Set priority for a job. Valid priorities : VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW
\nhadoop job -kill-task : Kill a task
\nhadoop job -history : Display job history including job details, failed and killed jobs
\nHadoop mradmin Commands<\/b>
\nhadoop mradmin -safemode get : Check Job tracker status
\nhadoop mradmin -refreshQueues : Reload mapreduce configuration
\nhadoop mradmin -refreshNodes : Reload active TaskTrackers
\nhadoop mradmin -refreshServiceAcl : Force Jobtracker to reload service ACL
\nhadoop mradmin -refreshUserToGroupsMappings : Force jobtracker to reload user group mappings
\nHadoop fsck Commands<\/b>
\nhadoop fsck \/ : Filesystem check on HDFS
\nhadoop fsck \/ -files : Display files during check
\nhadoop fsck \/ -files -blocks : Display files and blocks during check
\nhadoop fsck \/ -files -blocks -locations : Display files, blocks and its locationhadoop fsck \/ -files -blocks -locations -racks : Display network topology for data-node locations
\nhadoop fsck -delete : Delete corrupted files
\nhadoop fsck -move : Move corrupted files to \/lost+found directory<\/code>
\nHadoop Balancer Commands<\/b>
\nstart-balancer.sh : Balance the cluster
\nhadoop dfsadmin -setBalancerBandwidth : Adjust bandwidth used by the balancer
\nhadoop balancer -threshold 20 : Limit balancing to only 20% resources in the cluster<\/code>
\nHadoop Safe Mode (Maintenance Mode) Commands<\/b>
\nThe following dfsadmin commands helps the cluster to enter or leave safe mode, which is also called as maintenance mode.
\nIn this mode, Namenode does not accept any changes to the name space, it does not replicate or delete blocks.
\nhadoop dfsadmin -safemode enter : Enter safe mode
\nhadoop dfsadmin -safemode leave : Leave safe mode
\nhadoop dfsadmin -safemode get : Get the status of mode
\nhadoop dfsadmin -safemode wait : Wait until HDFS finishes data block replication
\nhadoop dfsadmin -report : total usage on the cluster<\/code><\/p>\n

Launching Hadoop Jobs:<\/b>
\nhadoop jar [mainClass] args... : <\/code>Launch job via jar file<\/code><\/code>
\nhadoop jar com.twitter.scalding.Tool [mainClass] args <\/code>: A Scalding job is launched using<\/code><\/code>\u00a0<\/code><\/code>
\nmapred job -kill <\/code>: If you need to kill a map-reduce job\u00a0<\/code>\u00a0<\/code>
\n<\/code><\/p>\n

Commonly Used Administration Commands:<\/b>
\nFormat the namenode: hadoop namenode -format
\nStarting Secondary namenode: hadoop secondrynamenode
\nRun namenode : hadoop namenode
\nRun data node: hadoop datanode
\nCluster Balancing: hadoop balancer
\nRun MapReduce job tracker node: hadoop jobtracker
\nRun MapReduce task tracker node: hadoop tasktracker<\/code><\/p>\n

Start\/Stop Yarn (starts resourcemanager and nodemanager)and DFS (Starts namenode and data node) from sbin directory:<\/b>
\nstart-yarn, stop-yarn
\nstart-dfs, stop-dfs<\/b><\/code>
\n
\nStart and Stop ALL daemon from sbin directory:<\/b>
\nstart-all, stop-all<\/b> <\/code><\/p>\n


\nCheck All 5 daemons (Namenode,Secoundary Node,Job Tracker, DataNode, Task Tracker ) are up:<\/b><\/code>
\njps <\/b><\/code><\/p>\n","protected":false},"excerpt":{"rendered":"

HDFS Command Syntax Overview: hadoop fs : Ex.: hadoop fs -ls hadoop version : check hadoop installed properly <\/p>\n

HELP: help [cmd]: hopefully this is self-describing <\/p>\n

Inspect files: ls\/lsr : list all files in cat : print on stdout tail [-f] : output the last part of the test : return attributes of file […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[60],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/5593"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5593"}],"version-history":[{"count":1,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/5593\/revisions"}],"predecessor-version":[{"id":5594,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/5593\/revisions\/5594"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5593"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5593"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5593"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}