高可用之——隔离术

​

一、总览

隔离主要有：

• 线程池隔离
• 进程隔离
• 集群隔离
• 机房隔离
• 读写隔离
• 动静隔离
• 爬虫隔离
• 热点隔离
• 资源隔离
• 环境隔离
• 压测隔离
• AB测试
• 缓存隔离
• 查询隔离

爬虫隔离可以在Nginx中进行如下配置。

set $flag "0";
if ($http_user_agent ~* "spider") {
  set $flag "1";
}
if ($flag == "0") {
  //代理到正常集群
}
if ($flag == "1") {
  //代理到爬虫集群
}

二、使用Hystrix实现隔离

1.线程池隔离

编写代码如下：

public class QueryOrderIdCommand extends HystrixCommand<Integer> {
    private final static Logger logger = LoggerFactory.getLogger(QueryOrderIdCommand.class);
    private OrderServiceProvider orderServiceProvider;
 
    public QueryOrderIdCommand(OrderServiceProvider orderServiceProvider) {
        super(Setter.withGroupKey(HystrixCommandGroupKey.Factory.asKey("orderService"))
                .andCommandKey(HystrixCommandKey.Factory.asKey("queryByOrderId"))
                .andCommandPropertiesDefaults(HystrixCommandProperties.Setter()
                        .withCircuitBreakerRequestVolumeThreshold(10)//至少有10个请求，熔断器才进行错误率的计算
                        .withCircuitBreakerSleepWindowInMilliseconds(5000)//熔断器中断请求5秒后会进入半打开状态,放部分流量过去重试
                        .withCircuitBreakerErrorThresholdPercentage(50)//错误率达到50开启熔断保护
                        .withExecutionTimeoutEnabled(true))
                .andThreadPoolPropertiesDefaults(HystrixThreadPoolProperties
                        .Setter().withCoreSize(10)));
        this.orderServiceProvider = orderServiceProvider;
    }
 
    @Override
    protected Integer run() {
        return orderServiceProvider.queryByOrderId();
    }
 
    @Override
    protected Integer getFallback() {
        return -1;
    }
}

调用如下：

@Test
public void testQueryByOrderIdCommand() {
  Integer r = new QueryOrderIdCommand(orderServiceProvider).execute();
  logger.info("result:{}", r);
}

2.信号量隔离

编写代码如下：

public class QueryByOrderIdCommandSemaphore extends HystrixCommand<Integer> {
    private final static Logger logger = LoggerFactory.getLogger(QueryByOrderIdCommandSemaphore.class);
    private OrderServiceProvider orderServiceProvider;
 
    public QueryByOrderIdCommandSemaphore(OrderServiceProvider orderServiceProvider) {
        super(Setter.withGroupKey(HystrixCommandGroupKey.Factory.asKey("orderService"))
                .andCommandKey(HystrixCommandKey.Factory.asKey("queryByOrderId"))
                .andCommandPropertiesDefaults(HystrixCommandProperties.Setter()
                        .withCircuitBreakerRequestVolumeThreshold(10)至少有10个请求，熔断器才进行错误率的计算
                        .withCircuitBreakerSleepWindowInMilliseconds(5000)//熔断器中断请求5秒后会进入半打开状态,放部分流量过去重试
                        .withCircuitBreakerErrorThresholdPercentage(50)//错误率达到50开启熔断保护
                        .withExecutionIsolationStrategy(HystrixCommandProperties.ExecutionIsolationStrategy.SEMAPHORE)
                        .withExecutionIsolationSemaphoreMaxConcurrentRequests(10)));//最大并发请求量
        this.orderServiceProvider = orderServiceProvider;
    }
 
    @Override
    protected Integer run() {
        return orderServiceProvider.queryByOrderId();
    }
 
    @Override
    protected Integer getFallback() {
        return -1;
    }
}

调用如下：

@Test
public void testQueryByOrderIdCommand() {
  Integer r = new QueryOrderIdCommand(orderServiceProvider).execute();
  logger.info("result:{}", r);
}

三、使用Servlet 3实现请求隔离

需要使用Tomcat7.x以上版本。对Tomcat的优化可以参见《​​Tomcat之——运行模式​​》 和《​​Tomcat之——并发优化​​》

关于Servlet 3的相关知识，可以参考《Servlet3.1规范（中文最终版）》 。

下面以一个实例说明使用Servlet 3实现请求隔离。

1.接口层调用

@RestController
@RequestMapping("/app")
public class NIOCtrl {
    @Autowired
    private LocalNewsAsyncContext localNewsAsyncContext;
    @Autowired
    private NewsService newsService;

    @RequestMapping("/news")
    public void getNews(HttpServletRequest request,@RequestParam(value = "type",required = false) String type){
        if("1".equals(type)){
            localNewsAsyncContext.submit(request, () -> newsService.getNews());
            return;
        }
        localNewsAsyncContext.submit(request, () -> newsService.getNewsMap());
    }
}

2.将请求丢进指定线程池

@Service
public class LocalNewsAsyncContext {
    private final static Long timeOutSeconds= 5L;
    @Autowired
    private CustomAsyncListener asyncListener;
    @Autowired
    private ThreadPoolExecutor executor;

    public void submit(final HttpServletRequest request,final Callable<Object> task){
        final String uri= request.getRequestURI();
        final Map<String,String[]> params= request.getParameterMap();
        //开启异步上下文
        final AsyncContext asyncContext= request.startAsync();
        asyncContext.getRequest().setAttribute(Constant.URI,uri);
        asyncContext.getRequest().setAttribute(Constant.PARAMS, params);
        asyncContext.setTimeout(timeOutSeconds * 1000);
        if(asyncContext!=null){
            asyncContext.addListener(asyncListener);
        }
        executor.submit(new CustomCallable(asyncContext, task));

    }
}

3.自定义线程处理

public class CustomCallable implements Callable{
    private static final Logger LOG = LoggerFactory.getLogger(CustomCallable.class);

    public AsyncContext asyncContext;
    private Callable<Object> task;
    private String uri;
    private  Map<String,String[]> params;

    public CustomCallable(AsyncContext asyncContext, Callable<Object> task){
        this.asyncContext= asyncContext;
        this.task= task;
        this.uri= (String) asyncContext.getRequest().getAttribute(Constant.URI);
        this.params= (Map<String, String[]>) asyncContext.getRequest().getAttribute(Constant.PARAMS);
    }
    @Override public Object call() throws Exception {
        Object o= task.call();
        if(o==null){
            callback(asyncContext,o);
        }else if(o instanceof String){
            callback(asyncContext, o);
        }else if(o instanceof CompletableFuture){
            CompletableFuture<Object> future= (CompletableFuture<Object>) o;
            future.thenAccept(o1 -> callback(asyncContext, o1))
                    .exceptionally(throwable -> {
                        callback(asyncContext,"");
                        return null;
                    });
        }else {
            callback(asyncContext, o);
        }
        return null;
    }

    private void callback(AsyncContext asyncContext,Object result){
        HttpServletResponse response= (HttpServletResponse) asyncContext.getResponse();
        try{
            if(result instanceof String){
                write(response, (String) result);
            }else {
                write(response, JSON.toJSONString(result));
            }
        }catch (Exception e){
            response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
            e.printStackTrace();
            try {
                LOG.error("get info error for uri:{}, params:{}",uri,JSON.toJSONString(params),e);
            }catch (Exception e1){}
        }finally {
            asyncContext.complete();
        }
    }

    private void write(HttpServletResponse response,String result) throws IOException {
        response.getOutputStream().write(result.getBytes());
    }
}

4.定义业务线程池

@Configuration
public class LocalNewsPoolConfig {
    private final static Logger LOG= LoggerFactory.getLogger(LocalNewsPoolConfig.class);

    @Bean
    public ThreadPoolExecutor init(){
        int corePoolSize= 10;
        int maximumPoolSize= 100;
        int queueCapacity= 200;
        LinkedBlockingDeque<Runnable> queue= new LinkedBlockingDeque<>(queueCapacity);
        ThreadPoolExecutor executor= new ThreadPoolExecutor(corePoolSize,maximumPoolSize,60L, TimeUnit.SECONDS,queue);
        executor.allowCoreThreadTimeOut(true);
        executor.setRejectedExecutionHandler((r, executor1) -> {
            if(r instanceof CustomCallable){
                CustomCallable call= (CustomCallable) r;
                AsyncContext asyncContext= call.asyncContext;
                if(asyncContext!=null){
                    handler(asyncContext);
                }
            }
        });
        return executor;
    }

    private static void handler(AsyncContext asyncContext){
        try{
            ServletRequest req= asyncContext.getRequest();
            String uri= (String) req.getAttribute(Constant.URI);
            Map params= (Map) req.getAttribute(Constant.PARAMS);
            LOG.error("async req rejected. uri :{},params:{}",uri, JSON.toJSONString(params));
        }catch (Exception e){
            e.printStackTrace();
            try{
                HttpServletResponse response= (HttpServletResponse) asyncContext.getResponse();
                response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
            }catch (Exception e1){}
        }finally {
            asyncContext.complete();
        }
    }    
}

5.定义异步请求监听

@Component
public class CustomAsyncListener implements AsyncListener {
    private Logger LOG= LoggerFactory.getLogger(CustomAsyncListener.class);
    @Override public void onComplete(AsyncEvent asyncEvent) throws IOException {

    }

    @Override public void onTimeout(AsyncEvent asyncEvent) throws IOException {
        AsyncContext asyncContext= asyncEvent.getAsyncContext();
        try{
            ServletRequest req= asyncContext.getRequest();
            String uri= (String) req.getAttribute(Constant.URI);
            Map params= (Map) req.getAttribute(Constant.PARAMS);
            LOG.error("async req timeOut. uri :{},params:{}",uri, JSON.toJSONString(params));
        }catch (Exception e){
            e.printStackTrace();
        }finally {
            try{
                HttpServletResponse response= (HttpServletResponse) asyncContext.getResponse();
                response.setStatus(HttpServletResponse.SC_REQUEST_TIMEOUT);
            }catch (Exception e1){}
            asyncContext.complete();
        }
    }

    @Override public void onError(AsyncEvent asyncEvent) throws IOException {
        AsyncContext asyncContext= asyncEvent.getAsyncContext();
        try{
            ServletRequest req= asyncContext.getRequest();
            String uri= (String) req.getAttribute(Constant.URI);
            Map params= (Map) req.getAttribute(Constant.PARAMS);
            LOG.error("async req error. uri :{},params:{}",uri, JSON.toJSONString(params));
        }catch (Exception e){
            e.printStackTrace();
            try{
                HttpServletResponse response= (HttpServletResponse) asyncContext.getResponse();
                response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
            }catch (Exception e1){}
        }finally {
            asyncContext.complete();
        }
    }

    @Override public void onStartAsync(AsyncEvent asyncEvent) throws IOException {

    }
}

6.业务处理和常量类

@Service
public class NewsService {
    public String getNews(){
        return "servlet3 nio test.";
    }
    public StringBuilder getNewsMap(){
        return new StringBuilder("I do and i understand.");
    }
}
public class Constant {
    public static final String URI= "uri";
    public static final String PARAMS= "params";
}