在接口服务中,如果每一次都进行数据库查询,那么必然会给数据库造成很大的并发压力。所以需要为接口添加缓存,缓存技术选用Redis,并且使用Redis的集群,Api使用Spring-Data-Redis。Docker安装入门请参考之前博文:Docker基础及安装入门
1、使用Docker搭建Redis集群
#拉取镜像
docker pull redis:5.0.2
#创建容器
docker create --name redis-node01 -v /data/redis-data/node01:/data -p 6379:6379
redis:5.0.2 --cluster-enabled yes --cluster-config-file nodes-node-01.conf
docker create --name redis-node02 -v /data/redis-data/node02:/data -p 6380:6379
redis:5.0.2 --cluster-enabled yes --cluster-config-file nodes-node-02.conf
docker create --name redis-node03 -v /data/redis-data/node03:/data -p 6381:6379
redis:5.0.2 --cluster-enabled yes --cluster-config-file nodes-node-03.conf
#启动容器
docker start redis-node01 redis-node02 redis-node03
#开始组建集群
#进入redis-node01进行操作
docker exec -it redis-node01 /bin/bash
#组建集群
redis-cli --cluster create 172.17.0.1:6379 172.17.0.1:6380 172.17.0.1:6381 --
cluster-replicas 0
出现连接不到redis节点的问题:
尝试使用容器的ip地址(172.17.0.1这个地址是docker容器分配给主机的地址)重新搭建集群:
#查看容器的ip地址
docker inspect redis-node01 -> 172.17.0.4
docker inspect redis-node02 -> 172.17.0.5
docker inspect redis-node03 -> 172.17.0.6
#删除容器
docker stop redis-node01 redis-node02 redis-node03
docker rm redis-node01 redis-node02 redis-node03
rm -rf /data/redis-data
#进入redis-node01进行操作
docker exec -it redis-node01 /bin/bash
#组建集群(注意端口的变化)
redis-cli --cluster create 172.17.0.4:6379 172.17.0.5:6379 172.17.0.6:6379 --
cluster-replicas 0
搭建成功,查看集群信息:
可以看到,集群中节点的ip地址是docker分配的地址,那么在客户端(spring-data-redis)是没有办法访问的?如何解决?
2、docker的网络类型
docker的网络类型有:
- None:不为容器配置任何网络功能,没有网络 --net=none
- Container:与另一个运行中的容器共享Network Namespace,–net=container:containerID
- Host:与主机共享Network Namespace,–net=host
- Bridge:Docker设计的NAT网络模型(默认类型)
重点关注下Host类型:
host模式创建的容器没有自己独立的网络命名空间,是和物理机共享一个Network Namespace,并且共享物理机的所有端口与IP。但是它将容器直接暴露在公共网络中,是有安全隐患的。更详细的可以查看下篇博文:Docker网络及Docker Compose并编排微服务
3、使用host网络进行搭建集群
#创建容器
docker create --name redis-node01 --net host -v /data/redis-data/node01:/data
redis:5.0.2 --cluster-enabled yes --cluster-config-file nodes-node-01.conf --port
6379
docker create --name redis-node02 --net host -v /data/redis-data/node02:/data
redis:5.0.2 --cluster-enabled yes --cluster-config-file nodes-node-02.conf --port
6380
docker create --name redis-node03 --net host -v /data/redis-data/node03:/data
redis:5.0.2 --cluster-enabled yes --cluster-config-file nodes-node-03.conf --port
6381
#启动容器
docker start redis-node01 redis-node02 redis-node03
#进入redis-node01容器进行操作
docker exec -it redis-node01 /bin/bash
#172.16.55.185是主机的ip地址
redis-cli --cluster create 172.16.55.185:6379 172.16.55.185:6380 172.16.55.185:6381
--cluster-replicas 0
搭建成功,查看集群信息:
root@qqxhb:/data# redis-cli
127.0.0.1:6379> CLUSTER NODES
46e5582cd2d96a506955cc08e7b08343037c91d9 172.16.55.185:6380@16380 master - 0
1543766975796 2 connected 5461-10922
b42d6ccc544094f1d8f35fa7a6d08b0962a6ac4a 172.16.55.185:6381@16381 master - 0
1543766974789 3 connected 10923-16383
4c60f45d1722f771831c64c66c141354f0e28d18 172.16.55.185:6379@16379 myself,master - 0
1543766974000 1 connected 0-5460
4、编写代码进行测试集群
4.1、导入依赖
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>2.9.0</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.6</version>
</dependency>
4.2、编写配置文件
# redis集群配置
spring.redis.jedis.pool.max-wait = 5000
spring.redis.jedis.pool.max-Idle = 100
spring.redis.jedis.pool.min-Idle = 10
spring.redis.timeout = 10
spring.redis.cluster.nodes = 172.16.55.185:6379,172.16.55.185:6380,172.16.55.185:6381
spring.redis.cluster.max-redirects=5
4.3、编写配置类
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;
import java.util.List;
@Component
@ConfigurationProperties(prefix = "spring.redis.cluster")
public class ClusterConfigurationProperties {
private List<String> nodes;
private Integer maxRedirects;
public List<String> getNodes() {
return nodes;
}
public void setNodes(List<String> nodes) {
this.nodes = nodes;
}
public Integer getMaxRedirects() {
return maxRedirects;
}
public void setMaxRedirects(Integer maxRedirects) {
this.maxRedirects = maxRedirects;
}
}
4.4、注册Redis连接工厂
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisClusterConfiguration;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.connection.jedis.JedisConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.StringRedisSerializer;
@Configuration
public class RedisClusterConfig {
@Autowired
private ClusterConfigurationProperties clusterProperties;
@Bean
public RedisConnectionFactory connectionFactory() {
RedisClusterConfiguration configuration = new
RedisClusterConfiguration(clusterProperties.getNodes());
configuration.setMaxRedirects(clusterProperties.getMaxRedirects());
return new JedisConnectionFactory(configuration);
}
@Bean
public RedisTemplate<String, String> redisTemplate(RedisConnectionFactory
redisConnectionfactory) {
RedisTemplate<String, String> redisTemplate = new RedisTemplate<>();
redisTemplate.setConnectionFactory(redisConnectionfactory);
redisTemplate.setKeySerializer(new StringRedisSerializer());
redisTemplate.setValueSerializer(new StringRedisSerializer());
redisTemplate.afterPropertiesSet();
return redisTemplate;
}
}
4.5、编写测试用例
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.test.context.junit4.SpringRunner;
import java.util.Set;
@RunWith(SpringRunner.class)
@SpringBootTest
public class TestRedis {
@Autowired
private RedisTemplate<String,String> redisTemplate;
@Test
public void testSave(){
for (int i = 0; i < 100; i++) {
this.redisTemplate.opsForValue().set("key_" + i, "value_"+i);
}
Set<String> keys = this.redisTemplate.keys("key_*");
for (String key : keys) {
String value = this.redisTemplate.opsForValue().get(key);
System.out.println(value);
this.redisTemplate.delete(key);
}
}
}
5、接口统一添加缓存逻辑
5.1、采用拦截器进行缓存命中
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.commons.codec.digest.DigestUtils;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.stereotype.Component;
import org.springframework.web.servlet.HandlerInterceptor;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.util.Map;
@Component
public class RedisCacheInterceptor implements HandlerInterceptor {
@Autowired
private RedisTemplate<String, String> redisTemplate;
private static ObjectMapper mapper = new ObjectMapper();
@Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse
response, Object handler) throws Exception {
if (!StringUtils.equalsIgnoreCase(request.getMethod(), "get")) {
// 非get请求,如果不是graphql请求,放行
if (!StringUtils.equalsIgnoreCase(request.getRequestURI(), "/graphql"))
{
return true;
}
}
String data =
this.redisTemplate.opsForValue().get(createRedisKey(request));
if (StringUtils.isEmpty(data)) {
// 缓存未命中
return true;
}
response.setCharacterEncoding("UTF-8");
response.setContentType("application/json; charset=utf-8");
response.getWriter().write(data);
return false;
}
public static String createRedisKey(HttpServletRequest request) throws
Exception {
String paramStr = request.getRequestURI();
Map<String, String[]> parameterMap = request.getParameterMap();
if (parameterMap.isEmpty()) {
paramStr += IOUtils.toString(request.getInputStream(), "UTF-8");
} else {
paramStr += mapper.writeValueAsString(request.getParameterMap());
}
String redisKey = "WEB_DATA_" + DigestUtils.md5Hex(paramStr);
return redisKey;
}
}
注册拦截器到Spring容器:
import cn.qqxhb.dubbo.api.interceptor.RedisCacheInterceptor;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.servlet.config.annotation.InterceptorRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
@Configuration
public class WebConfig implements WebMvcConfigurer {
@Autowired
private RedisCacheInterceptor redisCacheInterceptor;
@Override
public void addInterceptors(InterceptorRegistry registry) {
registry.addInterceptor(this.redisCacheInterceptor).addPathPatterns("/**");
}
}
5.2、测试拦截器
错误分析:由于在拦截器中读取了输入流的数据,在request中的输入流只能读取一次,请求进去Controller时,输入流中已经没有数据了,导致获取不到数据。
5.3、通过包装request解决读取不到数据的错误
import org.apache.commons.io.IOUtils;
import javax.servlet.ReadListener;
import javax.servlet.ServletInputStream;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletRequestWrapper;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
/**
* 包装HttpServletRequest
*/
public class MyServletRequestWrapper extends HttpServletRequestWrapper {
private final byte[] body;
/**
* Construct a wrapper for the specified request.
*
* @param request The request to be wrapped
* */
public MyServletRequestWrapper(HttpServletRequest request) throws IOException
{
super(request);
body = IOUtils.toByteArray(super.getInputStream());
}
@Override
public BufferedReader getReader() throws IOException {
return new BufferedReader(new InputStreamReader(getInputStream()));
}
@Override
public ServletInputStream getInputStream() throws IOException {
return new RequestBodyCachingInputStream(body);
}
private class RequestBodyCachingInputStream extends ServletInputStream {
private byte[] body;
private int lastIndexRetrieved = -1;
private ReadListener listener;
public RequestBodyCachingInputStream(byte[] body) {
this.body = body;
}
@Override
public int read() throws IOException {
if (isFinished()) {
return -1;
}
int i = body[lastIndexRetrieved + 1];
lastIndexRetrieved++;
if (isFinished() && listener != null) {
try {
listener.onAllDataRead();
} catch (IOException e) {
listener.onError(e);
throw e;
}
}
return i;
}
@Override
public boolean isFinished() {
return lastIndexRetrieved == body.length - 1;
}
@Override
public boolean isReady() {
return isFinished();
}
@Override
public void setReadListener(ReadListener listener) {
if (listener == null) {
throw new IllegalArgumentException("listener cann not be null");
}
if (this.listener != null) {
throw new IllegalArgumentException("listener has been set");
}
this.listener = listener;
if (!isFinished()) {
try {
listener.onAllDataRead();
} catch (IOException e) {
listener.onError(e);
}
} else {
try {
listener.onAllDataRead();
} catch (IOException e) {
listener.onError(e);
}
}
}
@Override
public int available() throws IOException {
return body.length - lastIndexRetrieved - 1;
}
@Override
public void close() throws IOException {
lastIndexRetrieved = body.length - 1;
body = null;
}
}
}
通过过滤器进行包装request对象:
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;
import javax.servlet.FilterChain;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
/**
* 替换Request对象
*/
@Component
public class RequestReplaceFilter extends OncePerRequestFilter {
@Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse
response, FilterChain filterChain) throws ServletException, IOException {
if (!(request instanceof MyServletRequestWrapper)) {
request = new MyServletRequestWrapper(request);
}
filterChain.doFilter(request, response);
}
}
5.4、测试
6、响应结果写入到缓存
通过ResponseBodyAdvice进行实现。ResponseBodyAdvice是Spring提供的高级用法,会在结果被处理前进行拦截,拦截的逻辑自己实现,这样就可以实现拿到结果数据进行写入缓存的操作了。
import cn.qqxhb.dubbo.api.controller.GraphQLController;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.core.MethodParameter;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.http.MediaType;
import org.springframework.http.server.ServerHttpRequest;
import org.springframework.http.server.ServerHttpResponse;
import org.springframework.http.server.ServletServerHttpRequest;
import org.springframework.web.bind.annotation.ControllerAdvice;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.servlet.mvc.method.annotation.ResponseBodyAdvice;
import java.time.Duration;
@ControllerAdvice
public class MyResponseBodyAdvice implements ResponseBodyAdvice {
@Autowired
private RedisTemplate<String, String> redisTemplate;
private ObjectMapper mapper = new ObjectMapper();
@Override
public boolean supports(MethodParameter returnType, Class converterType) {
if (returnType.hasMethodAnnotation(GetMapping.class)) {
return true;
}
if (returnType.hasMethodAnnotation(PostMapping.class) &&
StringUtils.equals(GraphQLController.class.getName(),
returnType.getExecutable().getDeclaringClass().getName())) {
return true;
}
return false;
}
@Override
public Object beforeBodyWrite(Object body, MethodParameter returnType,
MediaType selectedContentType, Class selectedConverterType, ServerHttpRequest
request, ServerHttpResponse response) {
try {
String redisKey =
RedisCacheInterceptor.createRedisKey(((ServletServerHttpRequest)
request).getServletRequest());
String redisValue;
if(body instanceof String){
redisValue = (String)body;
}else{
redisValue = mapper.writeValueAsString(body);
}
this.redisTemplate.opsForValue().set(redisKey,redisValue ,
Duration.ofHours(1));
} catch (Exception e) {
e.printStackTrace();
}
return body;
}
}
7、增加CORS的支持
整合前端系统测试会发现,前面实现的拦截器中并没有对跨域进行支持,需要对CORS跨域支持:可以根据自己的项目设置合适的允许跨域的源及头信息。
@Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse
response, Object handler) throws Exception {
if(StringUtils.equalsIgnoreCase(request.getMethod(), "OPTIONS")){
return true;
}
if (!StringUtils.equalsIgnoreCase(request.getMethod(), "get")) {
// 非get请求,如果不是graphql请求,放行
if (!StringUtils.equalsIgnoreCase(request.getRequestURI(), "/graphql"))
{
return true;
}
}
String data =
this.redisTemplate.opsForValue().get(createRedisKey(request));
if (StringUtils.isEmpty(data)) {
// 缓存未命中
return true;
}
response.setCharacterEncoding("UTF-8");
response.setContentType("application/json; charset=utf-8");
// 支持跨域
response.setHeader("Access-Control-Allow-Origin", "*");
response.setHeader("Access-Control-Allow-Methods",
"GET,POST,PUT,DELETE,OPTIONS");
response.setHeader("Access-Control-Allow-Credentials", "true");
response.setHeader("Access-Control-Allow-Headers", "Content-Type,X-Token");
response.setHeader("Access-Control-Allow-Credentials", "true");
response.getWriter().write(data);
return false;
}