hive-jdbc的默认超时时间被mysql超时时间覆盖

现象hive-jdbc.2.1.1版本在循环遍历查询数据，并且同时处理发现偶尔会出现SocketException异常，推测异常是由于超时导致的，结果验证果然超时时间被设置为了mysql的30sError retrieving next rowjava.sql.SQLException: Error retrieving next rowat org.apache.hive.jdbc.HiveQu

项哥

5914人浏览 · 2021-07-13 20:51:30

项哥 · 2021-07-13 20:51:30 发布

现象

hive-jdbc.2.1.1版本

在循环遍历查询数据，并且同时处理发现偶尔会出现SocketException异常，推测异常是由于超时导致的，结果验证果然超时时间被设置为了mysql的30s

Error retrieving next row
java.sql.SQLException: Error retrieving next row
        at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:396)
        .....
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
        at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)

原因

org.apache.hive.jdbc.HiveConnection的setupLoginTimeout方法可以看出超时时间默认取的是DriverManager.getLoginTimeout()，loginTimeout最终就是Socket的setSoTimeout时间

  
  private void setupLoginTimeout() {
    long timeOut = TimeUnit.SECONDS.toMillis(DriverManager.getLoginTimeout());
    if (timeOut > Integer.MAX_VALUE) {
      loginTimeout = Integer.MAX_VALUE;
    } else {
      loginTimeout = (int) timeOut;
    }
  }
    private TTransport createUnderlyingTransport() throws TTransportException {       
     ...
            transport = HiveAuthFactory.getSocketTransport(this.host, this.port, this.loginTimeout);
            ...
    }

// 下面是HiveAuthFactory类中的方法
  public static TTransport getSocketTransport(String host, int port, int loginTimeout) {
    return new TSocket(host, port, loginTimeout);
  }

// 下面是thrift的TSocket类中的方法
  private void initSocket() {
    socket_ = new Socket();
    try {
      socket_.setSoLinger(false, 0);
      socket_.setTcpNoDelay(true);
      socket_.setKeepAlive(true);
      socket_.setSoTimeout(socketTimeout_);
    } catch (SocketException sx) {
      LOGGER.error("Could not configure socket.", sx);
    }
  }

在DriverManager.setLoginTimeout方法上打断点调试可以看到DriverManager.getLoginTimeout()的超时时间被覆盖
HikariDataSource.getConnection()---->new HikariPool(this)------>PoolBase.initializeDataSource()----->PoolBase.setLoginTimeout----->DriverDataSource.setLoginTimeout()----->DriverManager.setLoginTimeout

HikariDataSource extends HikariConfig

private static final long CONNECTION_TIMEOUT = SECONDS.toMillis(30);
 public HikariConfig()
   {
      connectionTimeout = CONNECTION_TIMEOUT;
   }

PoolBase.setLoginTimeout

   private void setLoginTimeout(final DataSource dataSource)
   {
      if (connectionTimeout != Integer.MAX_VALUE) {
         try {
            dataSource.setLoginTimeout(Math.max(1, (int) MILLISECONDS.toSeconds(500L + connectionTimeout)));
         }
         catch (Throwable e) {
            LOGGER.info("{} - Failed to set login timeout for data source. ({})", poolName, e.getMessage());
         }
      }
   }