HDFS文件损坏解决方法

作者: 喵星人ZC | 来源:发表于2019-05-30 22:41 被阅读0次

HDFS文件损坏解决方法
hadoop 3.x 案例3:datanode 异常
通过API访问HDFS
docker中安装hadoop过程及错误解决
mac文件已损坏，是否移到废纸篓
Hadoop HDFS 的 I/O 流操作
分布式常用命令
kill掉hadoop进程，datanode无法启动
Hadoop Java API
mysql 备份恢复-mysqldump

image.png

自己虚拟机搭建的伪分布Hadoop，没有停止进程，不小心点到关机。再次启动时就一直处于安全模式，而且说损坏了20个块

[hadoop@hadoop000 hadoop]$ hdfs fsck /

Connecting to namenode via http://hadoop000:50070
FSCK started by hadoop (auth:SIMPLE) from /192.168.245.100 for path / at Thu May 30 22:25:35 CST 2019
........
/spark_eventLog/application_1557764445673_0005: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741882

/spark_eventLog/application_1557764445673_0005: MISSING 1 blocks of total size 46746 B..
/spark_eventLog/application_1557764445673_0006: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741888

/spark_eventLog/application_1557764445673_0006: MISSING 1 blocks of total size 44885 B..
/spark_eventLog/application_1557764445673_0007: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741893

/spark_eventLog/application_1557764445673_0007: MISSING 1 blocks of total size 44932 B..
/spark_eventLog/application_1557764445673_0008: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741898

/spark_eventLog/application_1557764445673_0008: MISSING 1 blocks of total size 44935 B..
/spark_eventLog/application_1557764445673_0009: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741903

/spark_eventLog/application_1557764445673_0009: MISSING 1 blocks of total size 44931 B..
/spark_eventLog/application_1557764445673_0010: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741908

/spark_eventLog/application_1557764445673_0010: MISSING 1 blocks of total size 44935 B..
/spark_eventLog/application_1557764445673_0011: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741913

/spark_eventLog/application_1557764445673_0011: MISSING 1 blocks of total size 44931 B..
/spark_eventLog/application_1557764445673_0012: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741918

/spark_eventLog/application_1557764445673_0012: MISSING 1 blocks of total size 62124 B..
/spark_eventLog/application_1557764445673_0013: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741923

/spark_eventLog/application_1557764445673_0013: MISSING 1 blocks of total size 44932 B.........................
/spark_eventLog/local-1558243986730: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741876

/spark_eventLog/local-1558243986730: MISSING 1 blocks of total size 38056 B..
/spark_eventLog/local-1558244355032: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741877

/spark_eventLog/local-1558244355032: MISSING 1 blocks of total size 38056 B..........
/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0002-1558020903556-hadoop-select+domain+%2C+time+%2C+row_num...domain_info%28Stage-1558020926785-1-1-SUCCEEDED-root.hadoop-1558020910773.jhist: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741843

/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0002-1558020903556-hadoop-select+domain+%2C+time+%2C+row_num...domain_info%28Stage-1558020926785-1-1-SUCCEEDED-root.hadoop-1558020910773.jhist: MISSING 1 blocks of total size 35395 B..
/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0002.summary: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741842

/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0002.summary: MISSING 1 blocks of total size 394 B..
/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0002_conf.xml: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741844

/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0002_conf.xml: MISSING 1 blocks of total size 223006 B..
/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0003-1558021068294-hadoop-select+%0Adomain%2C+%0Atime%2C+%0Arow_nu...domain_info%28Stage-1558021086767-1-1-SUCCEEDED-root.hadoop-1558021073405.jhist: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741856

/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0003-1558021068294-hadoop-select+%0Adomain%2C+%0Atime%2C+%0Arow_nu...domain_info%28Stage-1558021086767-1-1-SUCCEEDED-root.hadoop-1558021073405.jhist: MISSING 1 blocks of total size 35481 B..
/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0003.summary: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741855

/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0003.summary: MISSING 1 blocks of total size 397 B..
/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0003_conf.xml: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741857

/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0003_conf.xml: MISSING 1 blocks of total size 223167 B..
/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0004-1558021143709-hadoop-select+%0Adomain%2C+%0Atime%2C+%0Arow_nu...domain_info%28Stage-1558021160715-1-1-SUCCEEDED-root.hadoop-1558021148130.jhist: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741869

/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0004-1558021143709-hadoop-select+%0Adomain%2C+%0Atime%2C+%0Arow_nu...domain_info%28Stage-1558021160715-1-1-SUCCEEDED-root.hadoop-1558021148130.jhist: MISSING 1 blocks of total size 35551 B..
/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0004.summary: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741868

/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0004.summary: MISSING 1 blocks of total size 397 B..
/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0004_conf.xml: CORRUPT blockpool BP-1926021031-192.168.245.100-1557764272932 block blk_1073741870

/tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1557764445673_0004_conf.xml: MISSING 1 blocks of total size 223322 B...Status: CORRUPT
 Total size:    102274952 B
 Total dirs:    29
 Total files:   60
 Total symlinks:                0
 Total blocks (validated):      57 (avg. block size 1794297 B)
  ********************************
  CORRUPT FILES:        20
  MISSING BLOCKS:       20
  MISSING SIZE:         1276573 B
  CORRUPT BLOCKS:       20
  ********************************
 Minimally replicated blocks:   37 (64.91228 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    1
 Average block replication:     0.64912283
 Corrupt blocks:                20
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          1
 Number of racks:               1
FSCK ended at Thu May 30 22:25:35 CST 2019 in 11 milliseconds


The filesystem under path '/' is CORRUPT

解决方法：

离开安全模式(处于安全模式是无法删除损坏块的)
hadoop dfsadmin -safemode leave


删除损坏块
hdfs fsck  /  -delete

如果是生产请尝试使用hdfs debug

HDFS文件损坏解决方法
自己虚拟机搭建的伪分布Hadoop，没有停止进程，不小心点到关机。再次启动时就一直处于安全模式，而且说损坏了20个...
hadoop 3.x 案例3:datanode 异常
一. 问题描述 hdfs web页面显示有的文件是损坏的，需要使用fsck来检查二. 解决方案通过fsck检查...
通过API访问HDFS
通过API操作HDFS 今天的主要内容 HDFS获取文件系统 HDFS文件上传 HDFS文件下载 HDFS目录创建...
docker中安装hadoop过程及错误解决
本文主要记录build 支持hdfs的docker过程中遇到的问题，以及解决方法。自己写的Dockerfile文件...
mac文件已损坏，是否移到废纸篓
Mac提示应用文件已损坏解决方法遇到这种情况，其实并不是说程序真的损坏了，而是你在打开这个程序的时候触发了mac...
Hadoop HDFS 的 I/O 流操作
HDFS 文件上传 HDFS 文件下载定位文件读取文件合并
分布式常用命令
Hadoop创建文件：上传文件到hdfs 查看hdfs文件删除hdfs文件/文件夹解压zip文件到指定目录 ...
kill掉hadoop进程，datanode无法启动
问题日志：查看对应的启动日志; 解决方法：找到hadoop配置文件中 hdfs-site.xml 清除掉集群中da...
Hadoop Java API
获取HDFS客户端对象创建文件夹读取HDFS文件创建文件、写文件重命名文件从本地复制文件到hdfs 复制...
mysql 备份恢复-mysqldump
备份恢复 mysql数据损坏类型 1.物理损坏磁盘损坏：硬件，磁道坏，dd，格式化文件损坏：数据文件损坏，re...