spring cloud alibaba 应用无法注册到sentinel dashboard

寂夜了无痕

5158人浏览 · 2023-08-11 17:17:19

寂夜了无痕 · 2023-08-11 17:17:19 发布

一。技术背景

由于升级jdk17的需要我们将项目中的 spring cloud spring cloud alibaba 以及springboot进行了升级各版本如下
spring cloud 2021.0.5
spring cloud alibaba 2021.0.5.0
spring boot 2.6.13

二。问题表现

当启动项目服务后，服务无法注册到 sentinel-dashboard

三。错误排查

首先检查sentinel-dashboard 启动状态启动成功并且可以正常访问且不存在网络问题
环境配置检查

<!-- 依赖检查 无误 -->
<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-gateway</artifactId>
</dependency>

<dependency>
  <groupId>com.alibaba.cloud</groupId>
  <artifactId>spring-cloud-starter-alibaba-sentinel</artifactId>
</dependency>

<dependency>
  <groupId>com.alibaba.cloud</groupId>
  <artifactId>spring-cloud-alibaba-sentinel-gateway</artifactId>
</dependency>

配置检查

#配置也正常
spring.cloud.sentinel.transport.dashboard=localhost:8080
spring.cloud.sentinel.transport.port=8719

第三步源码追踪

接下来开始漫长源码分析步骤

然后点击 spring.cloud.sentinel.transport.dashboard 这条配置跳转 com.alibaba.cloud.sentinel.SentinelProperties.Transport#setDashboard

然后点击 getDashboard() 方法查看在哪里调用最后来到了 com.alibaba.cloud.sentinel.custom.SentinelAutoConfiguration#init

在如下代码中

@Configuration(proxyBeanMethods = false)
@ConditionalOnProperty(name = "spring.cloud.sentinel.enabled", matchIfMissing = true)
@EnableConfigurationProperties(SentinelProperties.class)
public class SentinelAutoConfiguration {
	@PostConstruct
	private void init() {
        ///省略部分逻辑	
		if (StringUtils.isEmpty(System.getProperty(TransportConfig.SERVER_PORT))
				&& StringUtils.isNotBlank(properties.getTransport().getPort())) {
			System.setProperty(TransportConfig.SERVER_PORT,
					properties.getTransport().getPort());
		}
		if (StringUtils.isEmpty(System.getProperty(TransportConfig.CONSOLE_SERVER))
				&& StringUtils.isNotBlank(properties.getTransport().getDashboard())) {
			System.setProperty(TransportConfig.CONSOLE_SERVER,
					properties.getTransport().getDashboard());
		}
	}
}

断点时发现配置成功被设置到系统的属性配置中
接下来再来看心跳发送具体类 HeartbeatSenderInitFunc
com.alibaba.csp.sentinel.transport.init.HeartbeatSenderInitFunc#init

@Override
public void init() {
   HeartbeatSender sender = HeartbeatSenderProvider.getHeartbeatSender();
   if (sender == null) {
       RecordLog.warn("[HeartbeatSenderInitFunc] WARN: No HeartbeatSender loaded");
       return;
   }

   initSchedulerIfNeeded();
   long interval = retrieveInterval(sender);
   setIntervalIfNotExists(interval);
   //定时调度发送心跳
   scheduleHeartbeatTask(sender, interval);
}

这里有个重要的逻辑
HeartbeatSender sender = HeartbeatSenderProvider.getHeartbeatSender();
此次调用触发 HeartbeatSenderProvider 静态方法触发加载

static {
    resolveInstance();
}

private static void resolveInstance() {
	//spi加载类 并初始化
    HeartbeatSender resolved = SpiLoader.of(HeartbeatSender.class).loadHighestPriorityInstance();
    if (resolved == null) {
        RecordLog.warn("[HeartbeatSenderProvider] WARN: No existing HeartbeatSender found");
    } else {
        heartbeatSender = resolved;
        RecordLog.info("[HeartbeatSenderProvider] HeartbeatSender activated: {}", resolved.getClass()
            .getCanonicalName());
    }
}

/**
 * 调用的方法
 * Get resolved {@link HeartbeatSender} instance.
 *
 * @return resolved {@code HeartbeatSender} instance
 */
public static HeartbeatSender getHeartbeatSender() {
    return heartbeatSender;
}

其中在 SpiLoader.of(HeartbeatSender.class).loadHighestPriorityInstance(); 方法中创建了实例调用了SimpleHttpHeartbeatSender 构造方法

public SimpleHttpHeartbeatSender() {
    // Retrieve the list of default addresses.
    //获取控制台server地址列表
    List<Endpoint> newAddrs = TransportConfig.getConsoleServerList();
    if (newAddrs.isEmpty()) {
        RecordLog.warn("[SimpleHttpHeartbeatSender] Dashboard server address not configured or not available");
    } else {
        RecordLog.info("[SimpleHttpHeartbeatSender] Default console address list retrieved: {}", newAddrs);
    }
    this.addressList = newAddrs;
}

在构造方法中获取连接配置中的地址列表

TransportConfig.getConsoleServerList();
//com.alibaba.csp.sentinel.transport.config.TransportConfig#getConsoleServerList
public static List<Endpoint> getConsoleServerList() {
	//从sentinel配置中加载配置 
    String config = SentinelConfig.getConfig(CONSOLE_SERVER);
    List<Endpoint> list = new ArrayList<Endpoint>();
    if (StringUtil.isBlank(config)) {
        return list;
    }
    //省略部分逻辑
    return list;
}

SentinelConfig.getConfig(CONSOLE_SERVER);
获取配置的具体细节

public static String getConfig(String key) {
    AssertUtil.notNull(key, "key cannot be null");
    return props.get(key);
}
//此方法调用前会触发 SentinelConfig静态构造初始化

static {
    try {
        initialize();
        loadProps();
        resolveAppName();
        resolveAppType();
        RecordLog.info("[SentinelConfig] Application type resolved: {}", appType);
    } catch (Throwable ex) {
        RecordLog.warn("[SentinelConfig] Failed to initialize", ex);
        ex.printStackTrace();
    }
}
//我们来看核心的加载配置的方法 loadProps();
private static void loadProps() {
	//又调用了SentinelConfigLoader的类型的获取属性方法
    Properties properties = SentinelConfigLoader.getProperties();
    for (Object key : properties.keySet()) {
        setConfig((String) key, (String) properties.get(key));
    }
}

我们再来看下 SentinelConfigLoader.getProperties(); 具体的实现

public static Properties getProperties() {
    return properties;
}
//此方法的调用又会触发 SentinelConfigLoader的静态构造

static {
    try {
        load();
    } catch (Throwable t) {
        RecordLog.warn("[SentinelConfigLoader] Failed to initialize configuration items", t);
    }
}

//核心load方法
private static void load() {
    // Order: system property -> system env -> default file (classpath:sentinel.properties) -> legacy path
    String fileName = System.getProperty(SENTINEL_CONFIG_PROPERTY_KEY);
    if (StringUtil.isBlank(fileName)) {
        fileName = System.getenv(SENTINEL_CONFIG_ENV_KEY);
        if (StringUtil.isBlank(fileName)) {
            fileName = DEFAULT_SENTINEL_CONFIG_FILE;
        }
    }

    Properties p = ConfigUtil.loadProperties(fileName);
    if (p != null && !p.isEmpty()) {
        RecordLog.info("[SentinelConfigLoader] Loading Sentinel config from {}", fileName);
        properties.putAll(p);
    }

    for (Map.Entry<Object, Object> entry : new CopyOnWriteArraySet<>(System.getProperties().entrySet())) {
        String configKey = entry.getKey().toString();
        String newConfigValue = entry.getValue().toString();
        String oldConfigValue = properties.getProperty(configKey);
        properties.put(configKey, newConfigValue);
        if (oldConfigValue != null) {
            RecordLog.info("[SentinelConfigLoader] JVM parameter overrides {}: {} -> {}",
                    configKey, oldConfigValue, newConfigValue);
        }
    }
}

核心逻辑主要有两步

从sentinel配置文件中加载配置不为空则放入 properties 中
从系统的配置中获取并放入properties 如果已经存在则覆盖

这步完成后就给 SimpleHttpHeartbeatSender 的 addressList 属性就行赋值操作
从这里就可以看到这里加载的配置如果被提前触发那么在自动装配类设置的控制台地址等参数也无法生效

具体再来的调度逻辑在scheduleHeartbeatTask中方法每5秒发送一次心跳

private void scheduleHeartbeatTask(/*@NonNull*/ 
						final HeartbeatSender sender, 
						/*@Valid*/ long interval) {
    pool.scheduleAtFixedRate(new Runnable() {
        @Override
        public void run() {
            try {
                sender.sendHeartbeat();
            } catch (Throwable e) {
                RecordLog.warn("[HeartbeatSender] Send heartbeat error", e);
            }
        }
    }, 5000, interval, TimeUnit.MILLISECONDS);
    RecordLog.info("[HeartbeatSenderInit] HeartbeatSender started: "
        + sender.getClass().getCanonicalName());
}

我们再来看下具体的实现
sender.sendHeartbeat();
实现类只有一个 SimpleHttpHeartbeatSender

com.alibaba.csp.sentinel.transport.heartbeat.SimpleHttpHeartbeatSender#sendHeartbeat

@Override                                                                                                     
public boolean sendHeartbeat() throws Exception {                                                             
    if (TransportConfig.getRuntimePort() <= 0) {                                                              
        RecordLog.info("[SimpleHttpHeartbeatSender] Command server port not initialized, won't send heartbeat"
        return false;                                                                                         
    }                                                                                                         
    Endpoint addrInfo = getAvailableAddress();                                                                
    if (addrInfo == null) {                                                                                   
        return false;                                                                                         
    }                                                                                                         
                                                                                                              
    SimpleHttpRequest request = new SimpleHttpRequest(addrInfo, TransportConfig.getHeartbeatApiPath());       
    request.setParams(heartBeat.generateCurrentMessage());                                                    
    try {                                                                                                     
        SimpleHttpResponse response = httpClient.post(request);                                               
        if (response.getStatusCode() == OK_STATUS) {                                                          
            return true;                                                                                      
        } else if (clientErrorCode(response.getStatusCode()) || serverErrorCode(response.getStatusCode())) {  
            RecordLog.warn("[SimpleHttpHeartbeatSender] Failed to send heartbeat to " + addrInfo              
                + ", http status code: " + response.getStatusCode());                                         
        }                                                                                                     
    } catch (Exception e) {                                                                                   
        RecordLog.warn("[SimpleHttpHeartbeatSender] Failed to send heartbeat to " + addrInfo, e);             
    }                                                                                                         
    return false;                                                                                             
}

以上就是发送心跳的逻辑
核心逻辑

获取有效的链接
创建连接发送心跳请求
响应以及异常处理

但是在断点过程中有效的链接列表居然是空的这就是导致代码无法继续向下
在这里插入图片描述
然后我们继续围绕这个点进行排查

获取有效的地址列表方法如下

private Endpoint getAvailableAddress() {
    if (addressList == null || addressList.isEmpty()) {
      return null;
    }
    if (currentAddressIdx < 0) {
      currentAddressIdx = 0;
    }
    int index = currentAddressIdx % addressList.size();
    return addressList.get(index);
}

发现使用的成员变量 addressList 从上面可以看到配置是来自于SentinelConfigLoader类的properties 属性中。在这属性初始化时，如果系统中没有设置控制台地址等配置，那么后续也是无法获取到的。

看到这里就大概明白了，正常的情况，在自动配置类SentinelAutoConfiguration的init方法中
设置控制台地址等配置然后开启饥饿加载情况下初始化所有的 InitFunc 实现类其中的心跳发送也会在这里初始化，然后在进行后续流程。最终是加载 SentinelConfigLoader类的properties
因为我们在SentinelAutoConfiguration的init 那么SentinelConfigLoader 的properties属性中也会存在控制台地址等配置。
到这里我们就怀疑SentinelConfigLoader的properties 被提前加载了我们在断点观察

在这里插入图片描述

在这里插入图片描述
可以看到当自动配置类SentinelAutoConfiguration的init方法还没执行到时，发现我们自己实现的sentinelConfig 配置中引起了源码中·sentinelConfig的构造方法已经被调用同时这个调用又会触发SentinelConfigLoader 的静态构造调用当我们定义的sentinelConfig 类执行完成后才去执行的 SentinelAutoConfiguration的init方法
所以说，后续的心跳发送执行器，无法获取到控制台地址等配置，就导致应用无法注册到dashboard上

至于为什么导致这个加载顺序问题已经向官方提交了 issue 寻求帮助
https://github.com/alibaba/Sentinel/issues/3201
https://github.com/alibaba/spring-cloud-alibaba/issues/3425

四。解决办法

idea 启动类增加参数指定dashboard server地址以及应用名称

-Dcsp.sentinel.dashboard.server=localhost:8080 -Dcsp.sentinel.app.name=gateway-service

启动类设置系统变量

@SpringBootApplication
@EnableDiscoveryClient
public class GatewayServiceApplication {

    public static void main(String[] args) {
        System.setProperty("csp.sentinel.dashboard.server","localhost:8080");
        System.setProperty("csp.sentinel.app.name","gateway-service");
        SpringApplication.run(GatewayServiceApplication.class, args);
    }

}