关于HDFS文件块丢失/损坏的相关问题

作者: 海边的贝壳林 | 来源:发表于2018-05-10 15:45 被阅读0次

关于HDFS文件块丢失/损坏的相关问题
文件丢失？损坏？兼容性问题？到底是什么导致了错误
hdfs块丢失导致的异常问题
hadoop 3.x 案例3:datanode 异常
HDFS概述 -学习笔记一
断电导致HDFS 块损坏修复
HDFS的Federation特性
HDFS
Hadoop系列之HDFS初识、理论基础与读写流程
HDFS学习

打开Ambari看到hdfs报警[alert]: Total Blocks:[*], Missing Blocks:[*], 发现是有些文件块损坏了. 启动hdfs的时候发现也起不来了, 日志一直循环下面的东西.

Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://test01.bigdata.hbh:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.

NameNode一直处于安全模式

[root@test01 ~]# sudo -u hdfs hdfs dfsadmin -fs hdfs://test01.bigdata.hbh:8020 -safemode get
Safe mode is ON

打开NameNode UI可以看到如下的描述:

Safe mode is ON. The reported blocks 4156 needs additional 2 blocks to reach the threshold 1.0000 of total blocks 4157. The number of live datanodes 4 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.

说明我们的损坏的文件比例超过了阈值, 这个阈值配置在hdfs中, 下图是从Ambari的配置管理, 这里配置的是100%, 也就是说不允许任何一个块损坏掉. 如果我们配置成99%应该就不会触发safemode了.

image.png

问题描述: 测试集群上的硬盘容量很小, 只有几十G, 之前做基准测试的时候就把磁盘写满了, 导致数据块丢失, 系统启动都是有问题的, 一直说hdfs在safe mode.

基础

什么是safe mode

怎么样触发safe mode

丢了一部分副本的数据

检查

[hdfs@test01 ~]$ hadoop fsck /user/root/.staging/job_1515575016190_0003/job.jar -files -blocks -locations -racks
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Connecting to namenode via http://test01.bigdata.hbh:50070/fsck?ugi=hdfs&files=1&blocks=1&locations=1&racks=1&path=%2Fuser%2Froot%2F.staging%2Fjob_1515575016190_0003%2Fjob.jar
FSCK started by hdfs (auth:SIMPLE) from /172.16.201.200 for path /user/root/.staging/job_1515575016190_0003/job.jar at Fri Jan 26 16:11:15 CST 2018
/user/root/.staging/job_1515575016190_0003/job.jar 272019 bytes, 1 block(s):  Under replicated BP-1912246748-192.168.89.173-1513143837848:blk_1073751971_11222. Target Replicas is 10 but found 4 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
0. BP-1912246748-192.168.89.173-1513143837848:blk_1073751971_11222 len=272019 repl=4 [/default-rack/172.16.201.200:50010, /default-rack/172.16.201.201:50010, /default-rack/172.16.201.202:50010, /default-rack/172.16.201.204:50010]

Status: HEALTHY
 Total size:    272019 B
 Total dirs:    0
 Total files:   1
 Total symlinks:        0
 Total blocks (validated):  1 (avg. block size 272019 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:    0 (0.0 %)
 Under-replicated blocks:   1 (100.0 %)
 Mis-replicated blocks:     0 (0.0 %)
 Default replication factor:    3
 Average block replication: 4.0
 Corrupt blocks:        0
 Missing replicas:      6 (60.0 %)
 Number of data-nodes:      4
 Number of racks:       1
FSCK ended at Fri Jan 26 16:11:15 CST 2018 in 0 milliseconds


The filesystem under path '/user/root/.staging/job_1515575016190_0003/job.jar' is HEALTHY
[hdfs@test01 ~]$

脏数据

[hdfs@test01 ~]$ hadoop fsck /apps/hbase/data/oldWALs/test01.bigdata.hbh%2C16020%2C1515637923065.default.1515745933793 -files -blocks -locations -racks
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Connecting to namenode via http://test01.bigdata.hbh:50070/fsck?ugi=hdfs&files=1&blocks=1&locations=1&racks=1&path=%2Fapps%2Fhbase%2Fdata%2FoldWALs%2Ftest01.bigdata.hbh%252C16020%252C1515637923065.default.1515745933793
FSCK started by hdfs (auth:SIMPLE) from /172.16.201.200 for path /apps/hbase/data/oldWALs/test01.bigdata.hbh%2C16020%2C1515637923065.default.1515745933793 at Fri Jan 26 16:12:22 CST 2018
/apps/hbase/data/oldWALs/test01.bigdata.hbh%2C16020%2C1515637923065.default.1515745933793 91 bytes, 1 block(s):
/apps/hbase/data/oldWALs/test01.bigdata.hbh%2C16020%2C1515637923065.default.1515745933793: CORRUPT blockpool BP-1912246748-192.168.89.173-1513143837848 block blk_1073753448
 MISSING 1 blocks of total size 91 B
0. BP-1912246748-192.168.89.173-1513143837848:blk_1073753448_12711 len=91 MISSING!

Status: CORRUPT
 Total size:    91 B
 Total dirs:    0
 Total files:   1
 Total symlinks:        0
 Total blocks (validated):  1 (avg. block size 91 B)
  ********************************
  UNDER MIN REPL'D BLOCKS:  1 (100.0 %)
  dfs.namenode.replication.min: 1
  CORRUPT FILES:    1
  MISSING BLOCKS:   1
  MISSING SIZE:     91 B
  CORRUPT BLOCKS:   1
  ********************************
 Minimally replicated blocks:   0 (0.0 %)
 Over-replicated blocks:    0 (0.0 %)
 Under-replicated blocks:   0 (0.0 %)
 Mis-replicated blocks:     0 (0.0 %)
 Default replication factor:    3
 Average block replication: 0.0
 Corrupt blocks:        1
 Missing replicas:      0
 Number of data-nodes:      4
 Number of racks:       1
FSCK ended at Fri Jan 26 16:12:22 CST 2018 in 1 milliseconds


The filesystem under path '/apps/hbase/data/oldWALs/test01.bigdata.hbh%2C16020%2C1515637923065.default.1515745933793' is CORRUPT

处理问题

查到具体哪个DataNode的哪些文件是丢失/损坏了的

[root@test01 ~]# sudo -u hdfs hdfs fsck /apps/hbase/data/oldWALs/ | egrep -v '^\.+$' | egrep -v '^$'
Connecting to namenode via http://test01.bigdata.hbh:50070/fsck?ugi=hdfs&path=%2Fapps%2Fhbase%2Fdata%2FoldWALs
FSCK started by hdfs (auth:SIMPLE) from /172.16.201.200 for path /apps/hbase/data/oldWALs at Thu Feb 08 09:57:58 CST 2018
/apps/hbase/data/oldWALs/test02.bigdata.hbh%2C16020%2C1515637922143..meta.1515745955950.meta: CORRUPT blockpool BP-1912246748-192.168.89.173-1513143837848 block blk_1073753450
/apps/hbase/data/oldWALs/test02.bigdata.hbh%2C16020%2C1515637922143..meta.1515745955950.meta: MISSING 1 blocks of total size 91 B..
/apps/hbase/data/oldWALs/test05.bigdata.hbh%2C16020%2C1515637921765.default.1515745929606: CORRUPT blockpool BP-1912246748-192.168.89.173-1513143837848 block blk_1073753446
/apps/hbase/data/oldWALs/test05.bigdata.hbh%2C16020%2C1515637921765.default.1515745929606: MISSING 1 blocks of total size 91 B.Status: CORRUPT
 Total size:    182 B
 Total dirs:    1
 Total files:   2
 Total symlinks:        0
 Total blocks (validated):  2 (avg. block size 91 B)
  ********************************
  UNDER MIN REPL'D BLOCKS:  2 (100.0 %)
  dfs.namenode.replication.min: 1
  CORRUPT FILES:    2
  MISSING BLOCKS:   2
  MISSING SIZE:     182 B
  CORRUPT BLOCKS:   2
  ********************************
 Minimally replicated blocks:   0 (0.0 %)
 Over-replicated blocks:    0 (0.0 %)
 Under-replicated blocks:   0 (0.0 %)
 Mis-replicated blocks:     0 (0.0 %)
 Default replication factor:    3
 Average block replication: 0.0
 Corrupt blocks:        2
 Missing replicas:      0
 Number of data-nodes:      4
 Number of racks:       1
FSCK ended at Thu Feb 08 09:57:58 CST 2018 in 1 milliseconds
The filesystem under path '/apps/hbase/data/oldWALs' is CORRUPT

[root@test01 ~]# sudo -u hdfs hadoop fs -rm /apps/hbase/data/oldWALs/test05.bigdata.hbh%2C16020%2C1515637921765.default.1515745929606
18/02/08 09:58:17 INFO fs.TrashPolicyDefault: Moved: 'hdfs://test01.bigdata.hbh:8020/apps/hbase/data/oldWALs/test05.bigdata.hbh%2C16020%2C1515637921765.default.1515745929606' to trash at: hdfs://test01.bigdata.hbh:8020/user/hdfs/.Trash/Current/apps/hbase/data/oldWALs/test05.bigdata.hbh%2C16020%2C1515637921765.default.1515745929606
[root@test01 ~]# sudo -u hdfs hdfs fsck /apps/hbase/data/oldWALs/ | egrep -v '^\.+$' | egrep -v '^$'
Connecting to namenode via http://test01.bigdata.hbh:50070/fsck?ugi=hdfs&path=%2Fapps%2Fhbase%2Fdata%2FoldWALs
FSCK started by hdfs (auth:SIMPLE) from /172.16.201.200 for path /apps/hbase/data/oldWALs at Thu Feb 08 09:58:24 CST 2018
/apps/hbase/data/oldWALs/test02.bigdata.hbh%2C16020%2C1515637922143..meta.1515745955950.meta: CORRUPT blockpool BP-1912246748-192.168.89.173-1513143837848 block blk_1073753450
/apps/hbase/data/oldWALs/test02.bigdata.hbh%2C16020%2C1515637922143..meta.1515745955950.meta: MISSING 1 blocks of total size 91 B.Status: CORRUPT
 Total size:    91 B
 Total dirs:    1
 Total files:   1
 Total symlinks:        0
 Total blocks (validated):  1 (avg. block size 91 B)
  ********************************
  UNDER MIN REPL'D BLOCKS:  1 (100.0 %)
  dfs.namenode.replication.min: 1
  CORRUPT FILES:    1
  MISSING BLOCKS:   1
  MISSING SIZE:     91 B
  CORRUPT BLOCKS:   1
  ********************************
 Minimally replicated blocks:   0 (0.0 %)
 Over-replicated blocks:    0 (0.0 %)
 Under-replicated blocks:   0 (0.0 %)
 Mis-replicated blocks:     0 (0.0 %)
 Default replication factor:    3
 Average block replication: 0.0
 Corrupt blocks:        1
 Missing replicas:      0
 Number of data-nodes:      4
 Number of racks:       1
FSCK ended at Thu Feb 08 09:58:24 CST 2018 in 1 milliseconds
The filesystem under path '/apps/hbase/data/oldWALs' is CORRUPT

关于HDFS文件块丢失/损坏的相关问题
打开Ambari看到hdfs报警[alert]: Total Blocks:[*], Missing Blocks...
文件丢失？损坏？兼容性问题？到底是什么导致了错误
在日常的维护中，免不了和文件打交道，文件涉及的问题有很多类，这里讨论：文件丢失，损坏，兼容性问题。而对于文件丢失...
hdfs块丢失导致的异常问题
原文：https://blog.csdn.net/u010316405/article/details/51518...
hadoop 3.x 案例3:datanode 异常
一. 问题描述 hdfs web页面显示有的文件是损坏的，需要使用fsck来检查二. 解决方案通过fsck检查...
HDFS概述 -学习笔记一
1.1 HDFS产出背景及定义 1.2 HDFS优缺点 1.3 HDFS组成架构 1.4 HDFS文件块大小
断电导致HDFS 块损坏修复
一、现象断电导致HDFS服务不正常或者显示块损坏二、直接DN节点上删除文件一个block的三个副本(3副本）...
HDFS的Federation特性
背景 hdfs主要有两层架构：命名空间（Namespace）由目录，文件和块组成。它支持所有与命名空间相关的文件...
HDFS
HDFS：Hadoop分布式文件系统。文件系统的块（Block）：是磁盘块的整数倍，HDFS默认块的大小为64M...
Hadoop系列之HDFS初识、理论基础与读写流程
1 HDFS初识考虑这样一个问题：文件切成很多小文件块散列存储在集群中时，是如何知道每个小文件块存储的位置的呢？...
HDFS学习
1. HDFS：hadoop分布式文件管理系统，以流式数据访问模式来存储超大文件 2. HDFS数据块：HDFS上...