dubbo在停机时通过注册jvm关闭钩子来执行自身优雅停机工作,但当dubbo与spring一同运行时,由于spring也通过jvm关闭钩子注册:
public abstract class AbstractApplicationContext:
@Override
public void registerShutdownHook() {
if (this.shutdownHook == null) {
// No shutdown hook registered yet.
this.shutdownHook = new Thread() {
@Override
public void run() {
synchronized (startupShutdownMonitor) {
doClose();
}
}
};
Runtime.getRuntime().addShutdownHook(this.shutdownHook);
}
}
而jvm钩子函数的执行是并发执行
class Shutdown:
/* Run all registered shutdown hooks
*/
private static void runHooks() {
for (int i=0; i < MAX_SYSTEM_HOOKS; i++) {
try {
Runnable hook;
synchronized (lock) {
// acquire the lock to make sure the hook registered during
// shutdown is visible here.
currentRunningHook = i;
hook = hooks[i];
}
if (hook != null) hook.run();
} catch(Throwable t) {
if (t instanceof ThreadDeath) {
ThreadDeath td = (ThreadDeath)t;
throw td;
}
}
}
}
如果spring的关闭钩子先于dubbo注册的关闭钩子执行,则dubbo关闭钩子执行时会报IllegalStateException,比如:
[DubboShutdownHook] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [com.alibaba.dubbo.remoting.exchange.support.DefaultFuture]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [com.alibaba.dubbo.remoting.exchange.support.DefaultFuture]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1311)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1299)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1158)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeServer.isRunning(HeaderExchangeServer.java:88)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeServer.close(HeaderExchangeServer.java:109)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol.destroy(DubboProtocol.java:384)
at com.alibaba.dubbo.rpc.protocol.ProtocolListenerWrapper.destroy(ProtocolListenerWrapper.java:72)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper.destroy(ProtocolFilterWrapper.java:105)
at com.alibaba.dubbo.config.ProtocolConfig.destroyAll(ProtocolConfig.java:153)
at com.alibaba.dubbo.config.AbstractConfig$1.run(AbstractConfig.java:82)
at java.lang.Thread.run(Thread.java:748)
以上错误主要在dubbo 2.5.x版本发生,在2.6.x版本中,除了注册关闭钩子,还监听了spring关闭事件:
SpringExtensionFactory:
private static class ShutdownHookListener implements ApplicationListener {
@Override
public void onApplicationEvent(ApplicationEvent event) {
if (event instanceof ContextClosedEvent) {
// we call it anyway since dubbo shutdown hook make sure its destroyAll() is re-entrant.
// pls. note we should not remove dubbo shutdown hook when spring framework is present, this is because
// its shutdown hook may not be installed.
DubboShutdownHook shutdownHook = DubboShutdownHook.getDubboShutdownHook();
shutdownHook.destroyAll();
}
}
}
在监听到容器关闭事件时,执行关闭函数。
同时jvm的关闭钩子也没有去掉,这是为了兼容非spring容器环境下的情况。
在2.7.4版本中则更进一步:
public class SpringExtensionFactory:
public static void addApplicationContext(ApplicationContext context) {
CONTEXTS.add(context);
if (context instanceof ConfigurableApplicationContext) {
((ConfigurableApplicationContext) context).registerShutdownHook();
DubboShutdownHook.getDubboShutdownHook().unregister();
}
BeanFactoryUtils.addApplicationListener(context, SHUTDOWN_HOOK_LISTENER);
}
在使用容器上下文时,如果上下文实现了spring的ConfigurableApplicationContext,则在此上下文中显式注册spring关闭钩子,并在容器中添加dubbo ShutdownHookListener,同时注销dubbo在jvm中注册的关闭钩子。
并且用
private final AtomicBoolean registered = new AtomicBoolean(false);
保证了即使在添加了ApplicationListener后才调用jvm注册钩子时,也不会重复注册。做到了同时兼容有无spring上下文的情况。
前文说到在2.5.x版本中,spring关闭钩子和dubbo钩子并发执行时导致的问题,要在此版本下解决此问题,自行在监听spring关闭事件中调用dubbo关闭函数即可:
@Component("dubboDestroyHandler")
public class DubboDestroyHandler implements ApplicationListener {
private static final Logger LOGGER = LoggerFactory.getLogger(DubboDestroyHandler.class);
@Override
public void onApplicationEvent(ApplicationEvent event) {
//如果是容器关闭事件
if (event instanceof ContextClosedEvent) {
LOGGER.info("显式调用dubbo destroyAll");
ProtocolConfig.destroyAll();
}
}
}
但是这样还存在一个隐患,在2.6.3版本之前,判断客户端是否仍持有连接的方法中:
public class HeaderExchangeServer:
private boolean isRunning() {
Collection<Channel> channels = getChannels();
for (Channel channel : channels) {
if (DefaultFuture.hasFuture(channel)) {
return true;
}
}
return false;
}
DefaultFuture.hasFuture(channel)无法准确的判断客户端仍持有连接,导致channel过早关闭,可能导致业务未完成。直到2.6.3之前改成了:
if (channel.isConnected()) {
return true;
}
才解决了此问题,在2.5.9之后的2.5系列版本中,是在服务端从注册中心解注册之后,等待一段时间,好让consumer收到服务变化的通知
public class ProtocolConfig extends AbstractConfig:
// Wait for registry notification
try {
Thread.sleep(ConfigUtils.getServerShutdownTimeout());
} catch (InterruptedException e) {
logger.warn("Interrupted unexpectedly when waiting for registry notification during shutdown process!");
}
而在2.5.9之前的版本,就只能指望消费者能及时收到注册中心的通知,将channel禁用了。在集群规模,注册中心负担不大的情况下,一般没有问题。但是稳妥起见,升级到更高版本才能从根本上解决问题