大数据入门与实战-HDFS的常用操作

1 命令

1.1 dfs命令

bin/hdfs dfs
Usage: hadoop fs [generic options]
    [-appendToFile <localsrc> ... <dst>]
    [-cat [-ignoreCrc] <src> ...]
    [-checksum <src> ...]
    [-chgrp [-R] GROUP PATH...]
    [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
    [-chown [-R] [OWNER][:[GROUP]] PATH...]
    [-copyFromLocal [-f] [-p] [-l] [-d] <localsrc> ... <dst>]
    [-copyToLocal [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
    [-count [-q] [-h] [-v] [-t [<storage type>]] [-u] [-x] <path> ...]
    [-cp [-f] [-p | -p[topax]] [-d] <src> ... <dst>]
    [-createSnapshot <snapshotDir> [<snapshotName>]]
    [-deleteSnapshot <snapshotDir> <snapshotName>]
    [-df [-h] [<path> ...]]
    [-du [-s] [-h] [-x] <path> ...]
    [-expunge]
    [-find <path> ... <expression> ...]
    [-get [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
    [-getfacl [-R] <path>]
    [-getfattr [-R] {-n name | -d} [-e en] <path>]
    [-getmerge [-nl] [-skip-empty-file] <src> <localdst>]
    [-help [cmd ...]]
    [-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [<path> ...]]
    [-mkdir [-p] <path> ...]
    [-moveFromLocal <localsrc> ... <dst>]
    [-moveToLocal <src> <localdst>]
    [-mv <src> ... <dst>]
    [-put [-f] [-p] [-l] [-d] <localsrc> ... <dst>]
    [-renameSnapshot <snapshotDir> <oldName> <newName>]
    [-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ...]
    [-rmdir [--ignore-fail-on-non-empty] <dir> ...]
    [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
    [-setfattr {-n name [-v value] | -x name} <path>]
    [-setrep [-R] [-w] <rep> <path> ...]
    [-stat [format] <path> ...]
    [-tail [-f] <file>]
    [-test -[defsz] <path>]
    [-text [-ignoreCrc] <src> ...]
    [-touchz <path> ...]
    [-truncate [-w] <length> <path> ...]
    [-usage [cmd ...]]

上面的命令的作用基本上和linux文件操作等命令的效果差不多

文件夹创建与删除 上传文件

1.2 常用启动关闭命令

• sbin/hadoop-daemon.sh start namenode:

启动namenode

• sbin/hadoop-daemon.sh start datanode

启动datanode

• sbin/yarn-daemon.sh start resourcemanager

启动resourcemanager

• sbin/yarn-daemon.sh start nodemanager

启动nodemanager

• sbin/mr-jobhistory-daemon.sh start historyserver

启动historyserver

• jps

查看进程

与start对应的stop为停止相关的进程，比如sbin/hadoop-daemon.sh stop namenode

1.3常用的Web UI地址

• 文件操作系统
  http://192.168.31.135:50070/dfshealth.html#tab-datanode
  
• 监控集群任务运行状况
  http://192.168.31.135:8088/cluster/apps
  
• 历史任务管理器
  http://192.168.31.135:19888/jobhistory/app
  

1.4 其他命令

• bin/yarn jar

执行jar文件
例如下面的词频统计实例：
bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar wordcount /user/root/mapreduce/input/news.txt /user/root/mapreduce/output/

2 配置文件

2.1 yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
     <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>quincyqiang.com</value>
    </property>
    
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>True</value>
    </property>
    
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>604800</value>
    </property>
</configuration>

• yarn.log-aggregation-enable：打开日志聚合
• yarn.log-aggregation.retain-seconds：日志保留时间