问题描述

新装的hdp集群，hdfs启动后，datanode马上全部死掉，查看日志报错信息如下：

2020-06-22 00:37:20,594 INFO  datanode.DataNode (DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50
2020-06-22 00:37:20,595 ERROR datanode.DataNode (DataNode.java:secureMain(2883)) - Exception in secureMain
java.lang.RuntimeException: Although a UNIX domain socket path is configured as /var/lib/hadoop-hdfs/dn_socket, we cannot start a localDataXceiverServer because libhadoop cannot be loaded.
    at org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:1192)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:1161)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1416)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:500)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2782)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2690)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2732)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2876)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2900)

问题排查

执行命令hadoop checknative -a

hadoop checknative -a

Java HotSpot(TM) Client VM warning: You have loaded library /usr/hdp/3.1.0.0-78/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
20/06/22 23:42:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Native library checking:
hadoop:  false
zlib:    false
zstd  :  false
snappy:  false
lz4:     false
bzip2:   false
openssl: false
ISA-L:   false
20/06/22 23:42:14 INFO util.ExitUtil: Exiting with status 1: ExitException

load native-hadoop library 失败，但实际上 /usr/hdp/3.1.0.0-78/hadoop/lib/native/libhadoop.so.1.0.0等文件都存在。

重新安装一个jdk,本人使用的上openjdk1.8 开发版。
yum install -y java-1.8.0-openjdk-devel-1.8.0.252.b09-2.el7_8.x86_64

重新初始化ambari

ambari-server setup