一、工作中常用的HDFS的命令整理

说明：

20200527新增命令 setfacl 和 getfacl,见命令(15) 和 (16)

20200702新增hdfs dfs -ls 和 hdfs dfs -ls /的区别，见命令（17）

（1）设置文件副本

dsf.host.hz.hwinfo.io:tianyafu:/data/home/tianyafu:>hdfs dfs -setrep

-setrep: Not enough arguments: expected 2 but got 0

Usage: hadoop fs [generic options] -setrep [-R] [-w] <rep> <path> ...

示例：hdfs dfs -setrep -w -R 1 /tmp/tianyafu/sxf

（2）平衡DataNode之间的存储

dsf.host.hz.hwinfo.io:tianyafu:/data/home/tianyafu:>hdfs balancer --help

Usage: hdfs balancer

        [-policy <policy>]      the balancing policy: datanode or blockpool

        [-threshold <threshold>]        Percentage of disk capacity

        [-exclude [-f <hosts-file> | <comma-separated list of hosts>]]  Excludes the specified datanodes.

        [-include [-f <hosts-file> | <comma-separated list of hosts>]]  Includes only the specified datanodes.

        [-source [-f <hosts-file> | <comma-separated list of hosts>]]  Pick only the specified datanodes as source nodes.

        [-idleiterations <idleiterations>]      Number of consecutive idle iterations (-1 for Infinite) before exit.

        [-runDuringUpgrade]    Whether to run the balancer during an ongoing HDFS upgrade.This is usually not desired since it will not affect used space on over-utilized machines.

Generic options supported are

-conf <configuration file>    specify an application configuration file

-D <property=value>            use value for given property

-fs <local|namenode:port>      specify a namenode

-jt <local|resourcemanager:port>    specify a ResourceManager

-files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster

-libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.

-archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is

bin/hadoop command [genericOptions] [commandOptions]

常用示例：hdfs balancer -threshold 5

 # 这个是机器资源使用率的差额，如果设置太小，可能不会迁移；这个5是一个百分比，表示各个DataNode的磁盘使用率的偏差在5%左右

（3）使用hdfs用户

有时候权限不够，需要临时切换到hdfs用户

常用示例：sudo -u hdfs hdfs dfs -chown -R 775 path

（4）查看

dsd.host.hz.hwinfo.io:tianyafu:/home/tianyafu:>hdfs dfs -ls -help

-ls: Illegal option -help

Usage: hadoop fs [generic options] -ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [<path> ...]

常用示例：hdfs dfs -ls path

（5）创建目录

dsd.host.hz.hwinfo.io:tianyafu:/home/tianyafu:>hdfs dfs -mkdir -help

-mkdir: Illegal option -help

Usage: hadoop fs [generic options] -mkdir [-p] <path> ...

常用示例：hdfs dfs -mkdir path

（6）上传下载

dsd.host.hz.hwinfo.io:tianyafu:/home/tianyafu:>hdfs dfs -put -help

-put: Illegal option -help

Usage: hadoop fs [generic options] -put [-f] [-p] [-l] <localsrc> ... <dst>

dsd.host.hz.hwinfo.io:tianyafu:/home/tianyafu:>hdfs dfs -get -help

-get: Illegal option -help

Usage: hadoop fs [generic options] -get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>

常用示例：

hdfs dfs -put ./xxx.zip /tmp/

hdfs dfs -get /tmp/xxx.zip /home/hadoop/download

（7）移动和复制

dsd.host.hz.hwinfo.io:tianyafu:/home/tianyafu:>hdfs dfs -cp -help

-cp: Illegal option -help

Usage: hadoop fs [generic options] -cp [-f] [-p | -p[topax]] <src> ... <dst>

dsd.host.hz.hwinfo.io:tianyafu:/home/tianyafu:>hdfs dfs -mv -help

-mv: Illegal option -help

Usage: hadoop fs [generic options] -mv <src> ... <dst>

常用示例：

hdfs dfs -cp /tmp/xxx.zip /tmp/hadoop/

hdfs dfs -mv /tmp/xxx.zip /tmp/hadoop/

（8）删除

dsd.host.hz.hwinfo.io:tianyafu:/home/tianyafu:>hdfs dfs -rm -help

-rm: Illegal option -help

Usage: hadoop fs [generic options] -rm [-f] [-r|-R] [-skipTrash] <src> ...

常用示例： hdfs dfs -rm -r /tmp/xxx.zip

（9）修改权限

dsd.host.hz.hwinfo.io:tianyafu:/home/tianyafu:>hdfs dfs -chmod -R

-chmod: Not enough arguments: expected 2 but got 0

Usage: hadoop fs [generic options] -chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...

常用示例：hdfs dfs -chmod -R 775 /tmp/zhangsan

(10) 修改所属用户和所属组

dsd.host.hz.hwinfo.io:tianyafu:/home/tianyafu:>hdfs dfs -chown -R

-chown: Not enough arguments: expected 2 but got 0

Usage: hadoop fs [generic options] -chown [-R] [OWNER][:[GROUP]] PATH...

常用示例：hdfs dfs -chown -R  zhangsan:etluser /tmp/zhangsan

（11）查看目录或文件大小

dsd.host.hz.hwinfo.io:tianyafu:/home/tianyafu:>hdfs dfs -du -help

-du: Illegal option -help

Usage: hadoop fs [generic options] -du [-s] [-h] [-x] <path> ...

常用示例：hdfs dfs -du -s -h /tmp/zhangsan

（12）目录中文件个数统计

dsd.host.hz.hwinfo.io:tianyafu:/home/tianyafu:>hdfs dfs -count

-count: Not enough arguments: expected 1 but got 0

Usage: hadoop fs [generic options] -count [-q] [-h] [-v] [-x] <path> ...

常用示例：hdfs dfs -count -h /tmp/zhangsan/

（13）查看文件内容

dsd.host.hz.hwinfo.io:tianyafu:/home/tianyafu:>hdfs dfs -cat -help

-cat: Illegal option -help

Usage: hadoop fs [generic options] -cat [-ignoreCrc] <src> ...

常用示例：hdfs dfs -cat /tmp/1.log

（14） 查看目录的备份信息等

dsd.host.hz.hwinfo.io:tianyafu:/home/tianyafu:>hdfs fsck -help

Usage: DFSck <path> [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]] [-maintenance]

        <path>  start checking from this path

        -move  move corrupted files to /lost+found

        -delete delete corrupted files

        -files  print out files being checked

        -openforwrite  print out files opened for write

        -includeSnapshots      include snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it

        -list-corruptfileblocks print out list of missing blocks and files they belong to

        -blocks print out block report

        -locations      print out locations for every block

        -racks  print out network topology for data-node locations

常用示例：hdfs fsck /tmp/zhangsan

（15）设置HDFS的ACL条目

hdfs除了可以像linux设置所属用户和所属组外，还有ACL控制条目

一个HDFS路径设置了ACL控制条目后，对应的权限后面会有个 + 号

通过 hdfs dfs -setfacl 命令可以设置ACL条目

[azkaban@dsf ~]$ hdfs dfs -setfacl

-setfacl: <path> is missing

Usage: hadoop fs [generic options] -setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]

常用示例：hdfs dfs-setfacl-R -m user:hadoop:rwx <path>

（16）获取HDFS的ACL条目

通过 hdfs dfs -getfacl 命令可以获取ACL条目

-bash-4.1$ hdfs dfs -getfacl

-getfacl: <path> is missing

Usage: hadoop fs [generic options] -getfacl [-R] <path>

常用示例：hdfs dfs -getfacl <path>

（17）hdfs dfs -ls 和 hdfs dfs -ls / 的区别

hdfs dfs -ls 是一种简写方式，默认是查看到 hdfs dfs -ls /user/当前用户 这个目录的

hdfs dfs -ls / 是查看根目录