文章目录

  • 1.概要
  • 1.1 dubbo调用主流程
  • 1.2 dubbo集群配置
  • 2. 集群容错方案
  • 2.1 集群容错方案简介
  • AvailableCluster
  • BroadcastCluster
  • FailbackCluster
  • FailfastCluster
  • FailoverCluster(默认)
  • FailsafeCluster
  • ForkingCluster
  • MergeableCluster
  • RegistryAwareCluster
  • 2.2 dubbo消费端启动及调用过程引用集群
  • 2.3 AbstractClusterInvoker的select方法


1.概要

1.1 dubbo调用主流程
  • Cluster 将 Directory 中的多个 Invoker 伪装成一个Invoker,对上层透明,伪装过程包含了容错逻辑
  • Router 负责从多个 Invoker 中按路由规则选出子集,比如读写分离,应用隔离等
  • LoadBalance 负责从多个 Invoker 中选出具体的一个用于本次调用,选的过程包含了负载均衡算法

dubbo 自定义容错策略 dubbo 容错机制_dubbo 自定义容错策略

1.2 dubbo集群配置
<!-- 服务端配置 failover/failfast/failsafe/failback/forking-->
<dubbo:service cluster="failsafe"/>

<!-- 消费端配置 failover/failfast/failsafe/failback/forking-->
<dubbo:reference cluster="failsafe"/>

2. 集群容错方案

2.1 集群容错方案简介

dubbo集群容群方案有以下几种:AvailableCluster、BroadcastCluster、FailbackCluster、FailfastCluster、FailoverCluster、FailsafeCluster、ForkingCluster、MergeableCluster、RegistryAwareCluster,默认是FailoverCluster。

@SPI(FailoverCluster.NAME)
public interface Cluster {
  @Adaptive
  <T> Invoker<T> join(Directory<T> directory) throws RpcException;
}
mock=org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterWrapper
failover=org.apache.dubbo.rpc.cluster.support.FailoverCluster
failfast=org.apache.dubbo.rpc.cluster.support.FailfastCluster
failsafe=org.apache.dubbo.rpc.cluster.support.FailsafeCluster
failback=org.apache.dubbo.rpc.cluster.support.FailbackCluster
forking=org.apache.dubbo.rpc.cluster.support.ForkingCluster
available=org.apache.dubbo.rpc.cluster.support.AvailableCluster
mergeable=org.apache.dubbo.rpc.cluster.support.MergeableCluster
broadcast=org.apache.dubbo.rpc.cluster.support.BroadcastCluster
registryaware=org.apache.dubbo.rpc.cluster.support.RegistryAwareCluster
AvailableCluster

AvailableCluster,服务可用cluster,从提供者invoker列表中直接返回第一个可用的invoker,如果当前没有可用的实例,则抛出异常。

public class AvailableCluster implements Cluster {

    public static final String NAME = "available";

    @Override
    public <T> Invoker<T> join(Directory<T> directory) throws RpcException {
      return new AbstractClusterInvoker<T>(directory) {
        @Override
        public Result doInvoke(Invocation invocation, List<Invoker<T>> invokers, LoadBalance loadbalance) throws RpcException {
          for (Invoker<T> invoker : invokers) {
            if (invoker.isAvailable()) {
              return invoker.invoke(invocation);
            }
          }
          throw new RpcException("No provider available in " + invokers);
        }};
    }
}
BroadcastCluster

BroadcastCluster,广播调用所有提供者,逐个调用,任意一台报错则报错。通常用于通知所有提供者更新缓存或日志等本地资源信息。

public class BroadcastCluster implements Cluster {

    @Override
    public <T> Invoker<T> join(Directory<T> directory) throws RpcException {
        return new BroadcastClusterInvoker<T>(directory);
    }

}

public class BroadcastClusterInvoker<T> extends AbstractClusterInvoker<T> {
  
  private static final Logger logger = LoggerFactory.getLogger(BroadcastClusterInvoker.class);
  
  // 构造器
  public BroadcastClusterInvoker(Directory<T> directory) {
    super(directory);
  }
  
  @Override
  @SuppressWarnings({"unchecked", "rawtypes"})
  public Result doInvoke(final Invocation invocation, List<Invoker<T>> invokers, LoadBalance loadbalance) throws RpcException {
    checkInvokers(invokers, invocation);
    RpcContext.getContext().setInvokers((List) invokers);
    RpcException exception = null;
    Result result = null;
    
    for (Invoker<T> invoker : invokers) {
      try {
        result = invoker.invoke(invocation);
      } catch (RpcException e) {
        exception = e;
        logger.warn(e.getMessage(), e);
      } catch (Throwable e) {
        exception = new RpcException(e.getMessage(), e);
        logger.warn(e.getMessage(), e);
      }
    }
    
    if (exception != null) {
      throw exception;
    }
    return result;
  }

}
FailbackCluster

FailbackCluster,失败自动恢复,后台记录失败请求,定时重发。通常用于消息通知操作。

public class FailbackCluster implements Cluster {

    public final static String NAME = "failback";

    @Override
    public <T> Invoker<T> join(Directory<T> directory) throws RpcException {
        return new FailbackClusterInvoker<T>(directory);
    }

}

public class FailbackClusterInvoker<T> extends AbstractClusterInvoker<T> {
  
  @Override
  protected Result doInvoke(Invocation invocation, List<Invoker<T>> invokers, LoadBalance loadbalance) throws RpcException {
    Invoker<T> invoker = null;
    try {
      checkInvokers(invokers, invocation);
      invoker = select(loadbalance, invocation, invokers, null);
      return invoker.invoke(invocation);
    } catch (Throwable e) {
      logger.error("Failback to invoke method " + invocation.getMethodName() + ", wait for retry in background. Ignored exception: " + e.getMessage() + ", ", e);
      // 加入到失败定时任务中
      addFailed(loadbalance, invocation, invokers, invoker);
      return new RpcResult(); // ignore
    }
  }
  
  
  private void addFailed(LoadBalance loadbalance, Invocation invocation, List<Invoker<T>> invokers, Invoker<T> lastInvoker) {
    // DCL创建HashedWheelTimer
    if (failTimer == null) {
      synchronized (this) {
        if (failTimer == null) {
          failTimer = new HashedWheelTimer(new NamedThreadFactory("failback-cluster-timer", true), 1, TimeUnit.SECONDS, 32, failbackTasks);
        }
      }
    }
    // 创建重试task
    RetryTimerTask retryTimerTask = new RetryTimerTask(loadbalance, invocation, invokers, lastInvoker, retries, RETRY_FAILED_PERIOD);
    // 加入到时间轮中,并定时执行(默认5ms)
    failTimer.newTimeout(retryTimerTask, RETRY_FAILED_PERIOD, TimeUnit.SECONDS);
  }
  //......
}
FailfastCluster

FailfastCluster,快速失败,只发起一次调用,失败立即报错。通常用于非幂等性的写操作,比如新增记录。

public class FailfastCluster implements Cluster {

    public final static String NAME = "failfast";

    @Override
    public <T> Invoker<T> join(Directory<T> directory) throws RpcException {
        return new FailfastClusterInvoker<T>(directory);
    }

}

public class FailfastClusterInvoker<T> extends AbstractClusterInvoker<T> {
  
  public FailfastClusterInvoker(Directory<T> directory) {
    super(directory);
  }
  
  @Override
  public Result doInvoke(Invocation invocation, List<Invoker<T>> invokers, LoadBalance loadbalance) throws RpcException {
    checkInvokers(invokers, invocation);
    Invoker<T> invoker = select(loadbalance, invocation, invokers, null);
    try {
      return invoker.invoke(invocation);
    } catch (Throwable e) {
      if (e instanceof RpcException && ((RpcException) e).isBiz()) { // biz exception.
        throw (RpcException) e;
      }
      throw new RpcException(......);
    }
  }
}
FailoverCluster(默认)

FailoverCluster,当调用Invoker失败,则打印错误日志,同时尝试其他的Invoker,重试n次表示最多执行n个不同的invoker。通常用于读操作,但重试会带来更长延迟,可通过retries="2"来设置重试次数(不含第一次)。

public class FailoverCluster implements Cluster {

    public final static String NAME = "failover";

    @Override
    public <T> Invoker<T> join(Directory<T> directory) throws RpcException {
        return new FailoverClusterInvoker<T>(directory);
    }

}

public class FailoverClusterInvoker<T> extends AbstractClusterInvoker<T> {
  
  public FailoverClusterInvoker(Directory<T> directory) {
    super(directory);
  }
  
  @Override
  @SuppressWarnings({"unchecked", "rawtypes"})
  public Result doInvoke(Invocation invocation, final List<Invoker<T>> invokers, LoadBalance loadbalance) throws RpcException {
    List<Invoker<T>> copyInvokers = invokers;
    checkInvokers(copyInvokers, invocation);
    String methodName = RpcUtils.getMethodName(invocation);
    // 获取"retries"参数值,默认为2
    int len = getUrl().getMethodParameter(methodName, Constants.RETRIES_KEY, Constants.DEFAULT_RETRIES) + 1;
    if (len <= 0) {
      len = 1;
    }
    
    RpcException le = null;
    List<Invoker<T>> invoked = new ArrayList<Invoker<T>>(copyInvokers.size());
    Set<String> providers = new HashSet<String>(len);
    for (int i = 0; i < len; i++) {
      // 重新选择并校验Invoker列表,避免Invoker列表发生更新
      if (i > 0) {
        checkWhetherDestroyed();
        copyInvokers = list(invocation);
        checkInvokers(copyInvokers, invocation);
      }
      // 选择Invoker,不包括invoked列表中的
      Invoker<T> invoker = select(loadbalance, invocation, copyInvokers, invoked);
      invoked.add(invoker);
      RpcContext.getContext().setInvokers((List) invoked);
      try {
        Result result = invoker.invoke(invocation);
        // 正常执行返回result,不会执行下面finally,即不会把正常执行的invoker中的url加入到providers
        return result; 
      } catch (RpcException e) {
        if (e.isBiz()) { // biz exception.
          throw e;
        }
        le = e;
      } catch (Throwable e) {
        le = new RpcException(e.getMessage(), e);
      } finally {
        // 把执行失败的invoker中的url加入到providers
        providers.add(invoker.getUrl().getAddress());
      }
    }
    // 全部重试失败,抛出异常
    throw new RpcException(......);
  }

}
FailsafeCluster

FailsafeCluster,失败安全,出现异常时,直接忽略。通常用于写入审计日志等操作。

public class FailsafeCluster implements Cluster {

    public final static String NAME = "failsafe";

    @Override
    public <T> Invoker<T> join(Directory<T> directory) throws RpcException {
        return new FailsafeClusterInvoker<T>(directory);
    }

}

public class FailsafeClusterInvoker<T> extends AbstractClusterInvoker<T> {
  
  public FailsafeClusterInvoker(Directory<T> directory) {
    super(directory);
  }
  
  @Override
  public Result doInvoke(Invocation invocation, List<Invoker<T>> invokers, LoadBalance loadbalance) throws RpcException {
    try {
      checkInvokers(invokers, invocation);
      Invoker<T> invoker = select(loadbalance, invocation, invokers, null);
      return invoker.invoke(invocation);
    } catch (Throwable e) {
      logger.error("Failsafe ignore exception: " + e.getMessage(), e);
      return new RpcResult(); // ignore
    }
  }
}
ForkingCluster

ForkingCluster,并行调用多个服务器,只要一个成功即返回。通常用于实时性要求较高的读操作,但需要浪费更多服务资源。可通过 forks=“2” 来设置最大并行数。

public class ForkingCluster implements Cluster {

    public final static String NAME = "forking";

    @Override
    public <T> Invoker<T> join(Directory<T> directory) throws RpcException {
        return new ForkingClusterInvoker<T>(directory);
    }

}


public class ForkingClusterInvoker<T> extends AbstractClusterInvoker<T> {
  
  private final ExecutorService executor = Executors.newCachedThreadPool(
            new NamedInternalThreadFactory("forking-cluster-timer", true));
  
  // 构造器
  public ForkingClusterInvoker(Directory<T> directory) {
    super(directory);
  }
  
  @Override
  @SuppressWarnings({"unchecked", "rawtypes"})
  public Result doInvoke(final Invocation invocation, List<Invoker<T>> invokers, LoadBalance loadbalance) throws RpcException {
    try {
      checkInvokers(invokers, invocation);
      final List<Invoker<T>> selected;
      // 从“forks”key中获取并行数,默认2
      final int forks = getUrl().getParameter(Constants.FORKS_KEY, Constants.DEFAULT_FORKS);
      final int timeout = getUrl().getParameter(Constants.TIMEOUT_KEY, Constants.DEFAULT_TIMEOUT);
      
      // 选择 forks 个invoker
      if (forks <= 0 || forks >= invokers.size()) {
        selected = invokers;
      } else {
        selected = new ArrayList<>();
        for (int i = 0; i < forks; i++) {
          Invoker<T> invoker = select(loadbalance, invocation, invokers, selected);
          if (!selected.contains(invoker)) {
            selected.add(invoker);
          }
        }
      }
      
      /** 
      ** 开启 forks 个线程执行,若线程执行成功就加入到 ref 阻塞队列中,若执行过程中抛出异常,则对count进行自增,极端情况下所有的线程都抛出异常时,此时value==selected.size(),把最后一个执行抛出的异常加入到队列中(此时队列中就只一个异常)
      */
      RpcContext.getContext().setInvokers((List) selected);
      final AtomicInteger count = new AtomicInteger();
      final BlockingQueue<Object> ref = new LinkedBlockingQueue<>();
      for (final Invoker<T> invoker : selected) {
        executor.execute(new Runnable() {
          @Override
          public void run() {
            try {
              Result result = invoker.invoke(invocation);
              ref.offer(result);
            } catch (Throwable e) {
              int value = count.incrementAndGet();
              if (value >= selected.size()) {
                ref.offer(e);
              }
            }
          }});
      }
      
      try {
        // 取出第一个
        Object ret = ref.poll(timeout, TimeUnit.MILLISECONDS);
        if (ret instanceof Throwable) {
          // 异常抛出......
          Throwable e = (Throwable) ret;
          throw new RpcException(......)
        }
        return (Result) ret;
      } catch (InterruptedException e) {
        throw new RpcException("Failed to forking invoke provider " + selected + ", but no luck to perform the invocation. Last error is: " + e.getMessage(), e);
      }
    } finally {
      RpcContext.getContext().clearAttachments();
    }
  }
}
MergeableCluster

MergeableCluster,聚合集群,将集群中的调用结果聚合起来返回结果。消费方需从每个group中调用一次返回结果,合并结果返回,这样就可以实现聚合结果。

配置方式

<dubbo:reference interface="com.xxx.BarService" group="*" merger="true" />

<dubbo:reference interface="com.xxx.BarService" group="aaa,bbb" merger="true"/>

<!--指定合并方法,将调用返回结果的指定方法进行合并,合并方法的参数类型必须是返回结果类型本身-->
<dubbo:reference interface="com.xxx.BarService" group="*">
    <dubbo:method name="getItems" merger=".addAll" />
</dubbo:service>

源码分析

public class MergeableCluster implements Cluster {

    public static final String NAME = "mergeable";

    @Override
    public <T> Invoker<T> join(Directory<T> directory) throws RpcException {
        return new MergeableClusterInvoker<T>(directory);
    }

}

public class MergeableClusterInvoker<T> extends AbstractClusterInvoker<T> {
  
  private ExecutorService executor = Executors.newCachedThreadPool(new NamedThreadFactory("mergeable-cluster-executor", true));
  
  // 构造器
  public MergeableClusterInvoker(Directory<T> directory) {
    super(directory);
  }
  
  @Override
  protected Result doInvoke(Invocation invocation, List<Invoker<T>> invokers, LoadBalance loadbalance) throws RpcException {
    checkInvokers(invokers, invocation);
    String merger = getUrl().getMethodParameter(invocation.getMethodName(), Constants.MERGER_KEY);
    if (ConfigUtils.isEmpty(merger)) {
      for (final Invoker<T> invoker : invokers) {
        if (invoker.isAvailable()) {
          return invoker.invoke(invocation);
        }
      }
      return invokers.iterator().next().invoke(invocation);
    }
    
    Class<?> returnType;
    try {
      returnType = getInterface().getMethod(invocation.getMethodName(),invocation.getParameterTypes()).getReturnType();
    } catch (NoSuchMethodException e) {
      returnType = null;
    }
    
    // 开启多个线程执行
    Map<String, Future<Result>> results = new HashMap<String, Future<Result>>();
    for (final Invoker<T> invoker : invokers) {
      Future<Result> future = executor.submit(new Callable<Result>() {
        @Override
        public Result call() throws Exception {
          return invoker.invoke(new RpcInvocation(invocation, invoker));
        }
      });
      results.put(invoker.getUrl().getServiceKey(), future);
    }
    
    // 收集执行结果
    Object result = null;
    List<Result> resultList = new ArrayList<Result>(results.size());
    int timeout = getUrl().getMethodParameter(invocation.getMethodName(), Constants.TIMEOUT_KEY, Constants.DEFAULT_TIMEOUT);
    for (Map.Entry<String, Future<Result>> entry : results.entrySet()) {
      Future<Result> future = entry.getValue();
      Result r = future.get(timeout, TimeUnit.MILLISECONDS);
      if (r.hasException()) {
        //打印错误日志......
      } else {
        resultList.add(r);
      }
    }
    
    if (resultList.isEmpty()) {
      return new RpcResult((Object) null);
    } else if (resultList.size() == 1) {
      return resultList.iterator().next();
    }
    if (returnType == void.class) {
      return new RpcResult((Object) null);
    }
    
    if (merger.startsWith(".")) {
      //配置了在返回执行果需要执行的方法
      merger = merger.substring(1);
      Method method = returnType.getMethod(merger, returnType);
      if (!Modifier.isPublic(method.getModifiers())) {
        method.setAccessible(true);
      }
      result = resultList.remove(0).getValue();
      if (method.getReturnType() != void.class && method.getReturnType().isAssignableFrom(result.getClass())) {
        for (Result r : resultList) {
          result = method.invoke(result, r.getValue());
        }
      } else {
        for (Result r : resultList) {
          method.invoke(result, r.getValue());
        }
      }
    } else {
      Merger resultMerger;
      if (ConfigUtils.isDefault(merger)) {
        resultMerger = MergerFactory.getMerger(returnType);
      } else {
        resultMerger=ExtensionLoader.getExtensionLoader(Merger.class).getExtension(merger);
      }
      if (resultMerger != null) {
        List<Object> rets = new ArrayList<Object>(resultList.size());
        for (Result r : resultList) {
          rets.add(r.getValue());
        }
        result = resultMerger.merge(rets.toArray((Object[]) Array.newInstance(returnType, 0)));
      } else {
        throw new RpcException("There is no merger to merge result.");
      }
    }
    return new RpcResult(result);
  }
  
  //......
}
RegistryAwareCluster

RegistryAwareCluster,先通过"default"key查询来自本地注册的invoker,若没有查到,则返回第一个可用的invoker,若都不可用,抛出异常。

public class RegistryAwareCluster implements Cluster {

    public final static String NAME = "registryaware";

    @Override
    public <T> Invoker<T> join(Directory<T> directory) throws RpcException {
        return new RegistryAwareClusterInvoker<T>(directory);
    }

}

public class RegistryAwareClusterInvoker<T> extends AbstractClusterInvoker<T> {
  
  public RegistryAwareClusterInvoker(Directory<T> directory) {
    super(directory);
  }

  @Override
  @SuppressWarnings({"unchecked", "rawtypes"})
  public Result doInvoke(Invocation invocation, final List<Invoker<T>> invokers, LoadBalance loadbalance) throws RpcException {
    for (Invoker<T> invoker : invokers) {
      if (invoker.isAvailable() && invoker.getUrl().getParameter(Constants.REGISTRY_KEY + "." + Constants.DEFAULT_KEY, false)) {
        return invoker.invoke(invocation);
      }
    }
    
    for (Invoker<T> invoker : invokers) {
      if (invoker.isAvailable()) {
        return invoker.invoke(invocation);
      }
    }
    throw new RpcException("No provider available in " + invokers);
  }
}
2.2 dubbo消费端启动及调用过程引用集群

消费端启动过程会调用RegistryProtocol的refer方法来引用远程服务,源码如下

public <T> Invoker<T> refer(Class<T> type, URL url) throws RpcException {
  url = URLBuilder.from(url)
    .setProtocol(url.getParameter(REGISTRY_KEY, DEFAULT_REGISTRY))
    .removeParameter(REGISTRY_KEY)
    .build();
  Registry registry = registryFactory.getRegistry(url);
  if (RegistryService.class.equals(type)) {
    return proxyFactory.getInvoker((T) registry, type, url);
  }

  // 若配置了group="a,b" or group="*",则使用MergeableCluster
  Map<String, String> qs = StringUtils.parseQueryString(url.getParameterAndDecoded(REFER_KEY));
  String group = qs.get(Constants.GROUP_KEY);
  if (group != null && group.length() > 0) {
    if ((COMMA_SPLIT_PATTERN.split(group)).length > 1 || "*".equals(group)) {
      return doRefer(getMergeableCluster(), registry, type, url);
    }
  }
  return doRefer(cluster, registry, type, url);
}


private <T> Invoker<T> doRefer(Cluster cluster, Registry registry, Class<T> type, URL url) {
  RegistryDirectory<T> directory = new RegistryDirectory<T>(type, url);
  directory.setRegistry(registry);
  directory.setProtocol(protocol);
  
  // all attributes of REFER_KEY
  Map<String, String> parameters = new HashMap<String, String>(directory.getUrl().getParameters());
  URL subscribeUrl = new URL(CONSUMER_PROTOCOL, parameters.remove(REGISTER_IP_KEY), 0, type.getName(), parameters);
  if (!ANY_VALUE.equals(url.getServiceInterface()) && url.getParameter(REGISTER_KEY, true)){
    directory.setRegisteredConsumerUrl(getRegisteredConsumerUrl(subscribeUrl, url));
    registry.register(directory.getRegisteredConsumerUrl());
  }
  
  directory.buildRouterChain(subscribeUrl);
  directory.subscribe(subscribeUrl.addParameter(CATEGORY_KEY,PROVIDERS_CATEGORY + "," + CONFIGURATORS_CATEGORY + "," + ROUTERS_CATEGORY));
  
  // 返回 MergeableClusterInvoker,有调用时调用MergeableClusterInvoker的doInvoke方法
  Invoker invoker = cluster.join(directory);
  ProviderConsumerRegTable.registerConsumer(invoker, url, subscribeUrl, directory);
  return invoker;
}
2.3 AbstractClusterInvoker的select方法

上面的容错机制中都会对Invoker列表进行select,最终都会调用AbstractClusterInvoker的select方法,这里解析一下AbstractClusterInvoker的select方法,如下:

protected Invoker<T> select(LoadBalance loadbalance, Invocation invocation, List<Invoker<T>> invokers, List<Invoker<T>> selected) throws RpcException {
  if (CollectionUtils.isEmpty(invokers)) {
    return null;
  }
  
  String methodName = invocation == null ? StringUtils.EMPTY : invocation.getMethodName();
  
  // 获取"sticky"值,默认false。粘滞连接,用于有状态服务,尽可能让客户端总是向同一提供者发起调用,除非该提供者挂了,再连另一台
  boolean sticky = invokers.get(0).getUrl().getMethodParameter(methodName, Constants.CLUSTER_STICKY_KEY, Constants.DEFAULT_CLUSTER_STICKY);
  
  if (stickyInvoker != null && !invokers.contains(stickyInvoker)) {
    stickyInvoker = null;
  }

  // 开启粘滞连接 && 粘滞连接Invoker不为空 && (已经选择过的Invoker为空 || 已经选择过的Invoker不包含粘滞连接Invoker)
  if (sticky && stickyInvoker != null && (selected == null || !selected.contains(stickyInvoker))) {
    // 粘滞连接Invoker可用,直接返回粘滞连接Invoker
    if (availablecheck && stickyInvoker.isAvailable()) {
      return stickyInvoker;
    }
  }
  
  Invoker<T> invoker = doSelect(loadbalance, invocation, invokers, selected);
  
  // 开启粘滞连接,将Invoker设置为stickyInvoker
  if (sticky) {
    stickyInvoker = invoker;
  }
  return invoker;
}

private Invoker<T> doSelect(LoadBalance loadbalance, Invocation invocation, List<Invoker<T>> invokers, List<Invoker<T>> selected) throws RpcException {
  if (CollectionUtils.isEmpty(invokers)) {
    return null;
  }
  // 只有一个,直接返回
  if (invokers.size() == 1) {
    return invokers.get(0);
  }
  // 根据具体的负载均衡策略选择一个
  Invoker<T> invoker = loadbalance.select(invokers, getUrl(), invocation);
  
  // 若已经选择的Invoker列表中包含了上面选择的Invoker,或者 invoker 不可用,则重新选择
  if ((selected != null && selected.contains(invoker)) || (!invoker.isAvailable() && getUrl() != null && availablecheck)) {
    Invoker<T> rInvoker = reselect(loadbalance, invocation, invokers, selected, availablecheck);
    if (rInvoker != null) {
      invoker = rInvoker;
    } else {
      // 重新选择的Invoker为空,选择上面根据具体的负载均衡策略选择的invoker的下一个
      int index = invokers.indexOf(invoker);
      invoker = invokers.get((index + 1) % invokers.size());
    }
  }
  return invoker;
}

private Invoker<T> reselect(LoadBalance loadbalance, Invocation invocation, List<Invoker<T>> invokers, List<Invoker<T>> selected, boolean availablecheck) throws RpcException {
  List<Invoker<T>> reselectInvokers = new ArrayList<>(invokers.size() > 1 ? (invokers.size() - 1) : invokers.size());
  
  // 过滤出已经没有选择过的Invoker,放到reselectInvokers列表中
  for (Invoker<T> invoker : invokers) {
    if (availablecheck && !invoker.isAvailable()) {
      continue;
    }
    if (selected == null || !selected.contains(invoker)) {
      reselectInvokers.add(invoker);
    }
  }
  
  if (!reselectInvokers.isEmpty()) {
    // 重新根据具体的负载均衡策略选择一个
    return loadbalance.select(reselectInvokers, getUrl(), invocation);
  }
  
  // 若所有的Invoker都已经被选择过,则把reselectInvokers不包含的invoker加入到reselectInvokers中
  if (selected != null) {
    for (Invoker<T> invoker : selected) {
      if ((invoker.isAvailable()) && !reselectInvokers.contains(invoker)) {
        reselectInvokers.add(invoker);
      }
    }
  }
  
  if (!reselectInvokers.isEmpty()) {
    // 重新根据具体的负载均衡策略选择一个
    return loadbalance.select(reselectInvokers, getUrl(), invocation);
  }
  return null;
}