在hadoop的配置文件 core-site.xml中,需要设置 fs.default.name 或 fs.defaultFS ,具体应该使用哪一个,会造什么样的错误,需要在实际操作中去验证。官网有下面这段描述

原文 中文 core-site.xml所在目录
This section deals with important parameters to be specified in the given configuration files 本节讨论在给定的配置文件中指定的重要参数 etc/hadoop/core-site.xml
Parameter Value Notes
fs.defaultFS NameNode URI hdfs://host:port/
io.file.buffer.size 131072 Size of read/write buffer used in SequenceFiles.
We need to have only one of the two (fs.default.name or fs.defaultFS, and the former is deprecated). If both are present then NN HA will be enabled/disabled based on what is read first.

Following is description from NN HA in Apache docs:

fs.defaultFS - the default path prefix used by the Hadoop FS client when none is given.
Optionally, you may now configure the default path for Hadoop clients to use the new HA-enabled logical URI. If you used "mycluster" as the nameservice ID earlier, this will be the value of the authority portion of all of your HDFS paths. This may be configured like so, in your core-site.xml file:

<property>
    <name>fs.defaultFS</name>
    <value>hdfs://mycluster</value>
</property>

从文档中可以看出,使用 fs.default.name 还是 使用 fs.defaultFS ,要首先判断是否开启了 NN 的HA (namenode 的 highavaliable),如果开启了NN HA,那么就用fs.defaultFS,在单一namenode的情况下,就用 fs.default.name , 如果在单一namenode节点的情况使用 fs.defaultFS ,系统将报

20/05/17 10:34:33 ERROR namenode.NameNode: Failed to start namenode.
java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.
        at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:626)
        at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddressCheckLogical(DFSUtilClient.java:655)
        at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:617)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.getRpcServerAddress(NameNode.java:563)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:694)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:714)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:953)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:932)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.j

hadoop查看配置的 fs.default.name名字:
hdfs getconf -confKey fs.default.name

Logo

魔乐社区(Modelers.cn) 是一个中立、公益的人工智能社区,提供人工智能工具、模型、数据的托管、展示与应用协同服务,为人工智能开发及爱好者搭建开放的学习交流平台。社区通过理事会方式运作,由全产业链共同建设、共同运营、共同享有,推动国产AI生态繁荣发展。

更多推荐