kerberos mysql配置_mysql到HDFS,kerberos配置报错 Can't get Kerberos realm

报错日志如下，麻烦解惑2018-07-10 08:10:13.144 [main] INFOVMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl2018-07-10 08:10:13.161 [main] INFOEngine - the machine info=>osIn...

贫血王子

3149人浏览 · 2021-02-01 07:32:55

贫血王子 · 2021-02-01 07:32:55 发布

报错日志如下，麻烦解惑

2018-07-10 08:10:13.144 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl

2018-07-10 08:10:13.161 [main] INFO Engine - the machine info =>

osInfo:Oracle Corporation 1.8 25.102-b14

jvmInfo:Linux amd64 3.10.0-514.el7.x86_64

cpu num:4

totalPhysicalMemory:-0.00G

freePhysicalMemory:-0.00G

maxFileDescriptorCount:-1

currentOpenFileDescriptorCount:-1

GC Names[PS MarkSweep, PS Scavenge]

MEMORY_NAME | allocation_size | init_size

PS Eden Space | 512.50MB | 512.50MB

Code Cache | 240.00MB | 2.44MB

Compressed Class Space | 1,024.00MB | 0.00MB

PS Survivor Space | 85.00MB | 85.00MB

PS Old Gen | 1,365.50MB | 1,365.50MB

Metaspace | -0.00MB | 0.00MB

2018-07-10 08:10:13.204 [main] INFO Engine -

{

"content":[

{

"reader":{

"name":"mysqlreader",

"parameter":{

"column":[

"connection":[

{

"jdbcUrl":[

"jdbc:mysql://170.100.104.54:3306/uisftech_test"

"table":[

"postal_address"

]

}

"password":"*****",

"username":"root"

}

"writer":{

"name":"hdfswriter",

"parameter":{

"column":[

{

"name":"CONTACT_MECH_ID",

"type":"STRING"

{

"name":"TO_NAME",

"type":"STRING"

{

"name":"ATTN_NAME",

"type":"STRING"

{

"name":"ADDRESS1",

"type":"STRING"

{

"name":"ADDRESS2",

"type":"STRING"

{

"name":"ADDRESS3",

"type":"STRING"

{

"name":"DIRECTIONS",

"type":"STRING"

{

"name":"CITY",

"type":"STRING"

{

"name":"POSTAL_CODE",

"type":"STRING"

{

"name":"POSTAL_CODE_EXT",

"type":"STRING"

{

"name":"COUNTRY_GEO_ID",

"type":"STRING"

{

"name":"STATE_PROVINCE_GEO_ID",

"type":"STRING"

{

"name":"COUNTY_GEO_ID",

"type":"STRING"

{

"name":"POSTAL_CODE_GEO_ID",

"type":"STRING"

{

"name":"GEO_POINT_ID",

"type":"STRING"

{

"name":"LAST_UPDATED_STAMP",

"type":"STRING"

{

"name":"LAST_UPDATED_TX_STAMP",

"type":"STRING"

{

"name":"CREATED_STAMP",

"type":"STRING"

{

"name":"CREATED_TX_STAMP",

"type":"STRING"

}

"compress":"gzip",

"defaultFS":"hdfs://170.100.100.95:9000",

"fieldDelimiter":"\t",

"fileName":"dataxtest",

"fileType":"text",

"haveKerberos":"true",

"kerberosKeytabFilePath":"/var/run/cloudera-scm-agent/process/3398-hive-HIVESERVER2/hive.keytab",

"kerberosPrincipal":"hive/bigdatanode01.csebank.com@CSEBANK.COM",

"path":"/user/hive/warehouse/uisftech_test.db/postal_address",

"writeMode":"append"

}

"setting":{

"speed":{

"channel":"2"

}

2018-07-10 08:10:13.235 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null

2018-07-10 08:10:13.237 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0

2018-07-10 08:10:13.238 [main] INFO JobContainer - DataX jobContainer starts job.

2018-07-10 08:10:13.242 [main] INFO JobContainer - Set jobId = 0

2018-07-10 08:10:13.786 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://170.100.104.54:3306/uisftech_test?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.

2018-07-10 08:10:13.788 [job-0] WARN OriginalConfPretreatmentUtil - 您的配置文件中的列配置存在一定的风险. 因为您未配置读取数据库表的列，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.

2018-07-10 08:10:15.731 [job-0] ERROR JobContainer - Exception when job run

java.lang.IllegalArgumentException: Can't get Kerberos realm

at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65) ~[hadoop-common-2.7.1.jar:na]

at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:275) ~[hadoop-common-2.7.1.jar:na]

at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:311) ~[hadoop-common-2.7.1.jar:na]

at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.kerberosAuthentication(HdfsHelper.java:90) ~[hdfswriter-0.0.1-SNAPSHOT.jar:na]

at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.getFileSystem(HdfsHelper.java:64) ~[hdfswriter-0.0.1-SNAPSHOT.jar:na]

at com.alibaba.datax.plugin.writer.hdfswriter.HdfsWriter$Job.init(HdfsWriter.java:47) ~[hdfswriter-0.0.1-SNAPSHOT.jar:na]

at com.alibaba.datax.core.job.JobContainer.initJobWriter(JobContainer.java:704) ~[datax-core-0.0.1-SNAPSHOT.jar:na]

at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:304) ~[datax-core-0.0.1-SNAPSHOT.jar:na]

at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113) ~[datax-core-0.0.1-SNAPSHOT.jar:na]

at com.alibaba.datax.core.Engine.start(Engine.java:92) [datax-core-0.0.1-SNAPSHOT.jar:na]

at com.alibaba.datax.core.Engine.entry(Engine.java:171) [datax-core-0.0.1-SNAPSHOT.jar:na]

at com.alibaba.datax.core.Engine.main(Engine.java:204) [datax-core-0.0.1-SNAPSHOT.jar:na]

Caused by: java.lang.reflect.InvocationTargetException: null

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_102]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_102]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_102]

at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_102]

at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:84) ~[hadoop-auth-2.7.1.jar:na]

at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63) ~[hadoop-common-2.7.1.jar:na]

... 11 common frames omitted

Caused by: sun.security.krb5.KrbException: Cannot locate default realm

at sun.security.krb5.Config.getDefaultRealm(Config.java:1029) ~[na:1.8.0_102]

... 17 common frames omitted

2018-07-10 08:10:15.742 [job-0] ERROR Engine -

经DataX智能分析,该任务最可能的错误原因是:

com.alibaba.datax.common.exception.DataXException: Code:[Framework-02], Description:[DataX引擎运行过程出错，具体原因请参看DataX运行结束时的错误诊断信息 .]. - java.lang.IllegalArgumentException: Can't get Kerberos realm

at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)

at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:275)

at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:311)

at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.kerberosAuthentication(HdfsHelper.java:90)

at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.getFileSystem(HdfsHelper.java:64)

at com.alibaba.datax.plugin.writer.hdfswriter.HdfsWriter$Job.init(HdfsWriter.java:47)

at com.alibaba.datax.core.job.JobContainer.initJobWriter(JobContainer.java:704)

at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:304)

at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113)

at com.alibaba.datax.core.Engine.start(Engine.java:92)

at com.alibaba.datax.core.Engine.entry(Engine.java:171)

at com.alibaba.datax.core.Engine.main(Engine.java:204)

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:84)

at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)

... 11 more

Caused by: KrbException: Cannot locate default realm

at sun.security.krb5.Config.getDefaultRealm(Config.java:1029)

... 17 more