IT博客汇 | HDFS丢失块:不良 : 群集中有 1 个丢失块问题处理

HDFS丢失块:不良 : 群集中有 1 个丢失块问题处理

牧人发表于 2024-06-06 13:53:00

Cloudera Manager管理的集群中hdfs报错

不良 : 群集中有 1 个 丢失块 块。群集中共有 6,956 个块。百分比 丢失块: 0.01%。 临界阈值：任意。

看来是有文件损坏了，解决方法：
1、检查缺失块:

[root@devnode01 ~]# hdfs fsck /
Connecting to namenode via http://devnode02:9870/fsck?ugi=root&path=%2F
FSCK started by root (auth:SIMPLE) from /10.23.74.196 for path / at Thu Jun 06 13:11:54 CST 2024
Status: CORRUPT
 Number of data-nodes:  3
 Number of racks:               1
 Total dirs:                    3969
 Total symlinks:                0

Replicated Blocks:
 Total size:    8730303943 B
 Total files:   6953
 Total blocks (validated):      6962 (avg. block size 1253993 B)
  ********************************
  UNDER MIN REPL'D BLOCKS:      1 (0.014363688 %)
  dfs.namenode.replication.min: 1
  CORRUPT FILES:        1
  CORRUPT BLOCKS:       1
  CORRUPT SIZE:         35557852 B
  ********************************
 Minimally replicated blocks:   6961 (99.985634 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       6 (0.08618213 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     2.9987073
 Missing blocks:                0
 Corrupt blocks:                1
 Missing replicas:              6 (0.028727377 %)
 Blocks queued for replication: 0

Erasure Coded Block Groups:
 Total size:    0 B
 Total files:   0
 Total block groups (validated):        0
 Minimally erasure-coded block groups:  0
 Over-erasure-coded block groups:       0
 Under-erasure-coded block groups:      0
 Unsatisfactory placement block groups: 0
 Average block group size:      0.0
 Missing block groups:          0
 Corrupt block groups:          0
 Missing internal blocks:       0
 Blocks queued for replication: 0
FSCK ended at Thu Jun 06 13:11:54 CST 2024 in 137 milliseconds


The filesystem under path '/' is CORRUPT

2、查看缺失块的详情

[root@devnode01 ~]# hdfs fsck / | egrep -v '^\.+$' | grep -v eplica
Connecting to namenode via http://devnode02:9870/fsck?ugi=root&path=%2F
FSCK started by root (auth:SIMPLE) from /10.23.74.196 for path / at Thu Jun 06 13:14:50 CST 2024


/sparkforyarn/jars/hive-exec-2.1.1-cdh6.1.1.jar: CORRUPT blockpool BP-977273909-10.23.74.199-1715236315519 block blk_1073745302

/sparkforyarn/jars/hive-exec-2.1.1-cdh6.1.1.jar: CORRUPT 1 blocks of total size 35557852 B.

Status: CORRUPT
 Number of data-nodes:  3
 Number of racks:               1
 Total dirs:                    3969
 Total symlinks:                0

 Total size:    8730303943 B
 Total files:   6953
 Total blocks (validated):      6962 (avg. block size 1253993 B)
  ********************************
  UNDER MIN REPL'D BLOCKS:      1 (0.014363688 %)
  CORRUPT FILES:        1
  CORRUPT BLOCKS:       1
  CORRUPT SIZE:         35557852 B
  ********************************
 Missing blocks:                0
 Corrupt blocks:                1

Erasure Coded Block Groups:
 Total size:    0 B
 Total files:   0
 Total block groups (validated):        0
 Minimally erasure-coded block groups:  0
 Over-erasure-coded block groups:       0
 Under-erasure-coded block groups:      0
 Unsatisfactory placement block groups: 0
 Average block group size:      0.0
 Missing block groups:          0
 Corrupt block groups:          0
 Missing internal blocks:       0
FSCK ended at Thu Jun 06 13:14:50 CST 2024 in 104 milliseconds


The filesystem under path '/' is CORRUPT

在hdfs的web管理界面上也会直接显示：

/sparkforyarn/jars/hive-exec-2.1.1-cdh6.1.1.jar文件损坏了.

3、删除损坏的文件

[root@devnode01 ~]# hdfs fsck -delete /sparkforyarn/jars/hive-exec-2.1.1-cdh6.1.1.jar
Connecting to namenode via http://devnode02:9870/fsck?ugi=root&delete=1&path=%2Fsparkforyarn%2Fjars%2Fhive-exec-2.1.1-cdh6.1.1.jar
FSCK started by root (auth:SIMPLE) from /10.23.74.196 for path /sparkforyarn/jars/hive-exec-2.1.1-cdh6.1.1.jar at Thu Jun 06 13:20:13 CST 2024

/sparkforyarn/jars/hive-exec-2.1.1-cdh6.1.1.jar: CORRUPT blockpool BP-977273909-10.23.74.199-1715236315519 block blk_1073745302

/sparkforyarn/jars/hive-exec-2.1.1-cdh6.1.1.jar: CORRUPT 1 blocks of total size 35557852 B.
Status: CORRUPT
 Number of data-nodes:  3
 Number of racks:               1
 Total dirs:                    0
 Total symlinks:                0

Replicated Blocks:
 Total size:    35557852 B
 Total files:   1
 Total blocks (validated):      1 (avg. block size 35557852 B)
  ********************************
  UNDER MIN REPL'D BLOCKS:      1 (100.0 %)
  dfs.namenode.replication.min: 1
  CORRUPT FILES:        1
  CORRUPT BLOCKS:       1
  CORRUPT SIZE:         35557852 B
  ********************************
 Minimally replicated blocks:   0 (0.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     0.0
 Missing blocks:                0
 Corrupt blocks:                1
 Missing replicas:              0
 Blocks queued for replication: 0

Erasure Coded Block Groups:
 Total size:    0 B
 Total files:   0
 Total block groups (validated):        0
 Minimally erasure-coded block groups:  0
 Over-erasure-coded block groups:       0
 Under-erasure-coded block groups:      0
 Unsatisfactory placement block groups: 0
 Average block group size:      0.0
 Missing block groups:          0
 Corrupt block groups:          0
 Missing internal blocks:       0
 Blocks queued for replication: 0
FSCK ended at Thu Jun 06 13:20:13 CST 2024 in 12 milliseconds


The filesystem under path '/sparkforyarn/jars/hive-exec-2.1.1-cdh6.1.1.jar' is CORRUPT

不带文件路径可自动删除所有损坏的文件块。

最后复制hive-exec-2.1.1-cdh6.1.1.jar文件到对应/sparkforyarn/jars/路径下即可，如果不能删除，需要从上面命令中找到发生在哪台机器上，查看日志，在具体分析。

HDFS丢失块:不良 : 群集中有 1 个 丢失块 问题处理

HDFS丢失块:不良 : 群集中有 1 个丢失块问题处理