Boot Spring 集成redis spring boot redis stream

本文不讨论那些专业的消息队列，只对Redis的两种消息队列的应用进行论述

1、集成Redis-PubSub发布订阅

Redis提供一种基于“发布/订阅”的消息机制，也称“广播模式”，发布者往指定的频道（channel）中发送消息，订阅了此频道的在线的消费者就都能收到这条消息。发布者发出消息之后就不会再管这条消息，Redis本身也不提供消息的持久化，所以消息一经发出，不管有没有消费者消息都会消失。这也是发布订阅模式的一个缺点，话不多说，直接撸代码。
以下版本从低到高代表着需求与设计上的逐步优化，不算多复杂，下面逐步展示。

1.1、版本1

1.1.1、发布

发布很简单，使用spring框架提供的方法即可。

RedisTemplate.convertAndSend(String channel, Object message);

1.1.2、订阅

消费者需要订阅指定的频道来接收消息，所以这个订阅的动作需要消费者自己去完成。Redis缓存相关的功能被封装在基础依赖中，功能设计时也要保证通用性，与业务解耦，能够达到拿来即用的程度。
先假定一个消息对象

import cn.hutool.json.JSONObject;
import cn.hutool.json.JSONUtil;
import lombok.Getter;
import lombok.Setter;

import java.io.Serializable;

/**
 * @desc redis发布订阅消息体
 */
@Getter
@Setter
public class RedisPubSubMessage implements Serializable {

    private static final long serialVersionUID = 1479841433987843107L;

    /**
     * 数据id
     */
    private Long dataId;

    /**
     * 数据所属人
     */
    private Long userId;

    /**
     * 附加额外数据
     */
    private JSONObject extInfo;

    @Override
    public String toString() {
        return JSONUtil.toJsonStr(this);
    }
}

1.1.2.1、自定义注解

最开始的想法是使用自定义注解，消费者将自定义注解加载启动类上实现开启订阅，但是开启订阅的同时需要知道订阅的频道和消息的消费处理器（后面简称“handler”），那就只能使用嵌套注解在开启的注解中指定频道和handler，需要先设计子注解（命名为RedisListener），RedisListener中要定义出频道和handler的类型，由于和业务解耦，所以我们提供一个接口暴露给消费者，消费者实现这个接口完成消息处理，暴露的消息处理接口：

import cn.hutool.core.bean.BeanUtil;
import cn.hutool.core.util.StrUtil;
import cn.hutool.json.JSONUtil;

/**
 * redis订阅消息处理器
 */
public interface IRedisMessageReceiver {

    /**
     * 接收消息对象
     * @param message
     */
    void receiveMessage(RedisPubSubMessage message);
}

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

/**
 * @desc 消息监听配置注解
 */
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
public @interface RedisListener {

    /**
     * 监听的频道名称
     *
     * @return
     */
    String channelTopic();

    /**
     * 消息处理器
     *
     * @return
     */
    Class<? extends IRedisMessageReceiver> receive();
}

接着完成开启监听的注解，支持多个频道的话所以用了数组：

import org.springframework.context.annotation.Import;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

/**
 * @desc 开启redis订阅监听
 */
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
public @interface EnableRedisListener {

    /**
     * 消息监听配置注解
     * @return
     */
    RedisListener[] listeners() default {};
}

1.1.2.2、注册订阅

注解完成了，下面需要考虑的是如何通过注解来完成订阅的动作，订阅需要的参数都在注解中，所以我们需要在订阅之前解析注解中的参数。与注解经常搭配的是@Import，所以我们如下设计订阅的类：

import cn.hutool.core.map.MapUtil;
import cn.hutool.core.util.ArrayUtil;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.BeansException;
import org.springframework.beans.MutablePropertyValues;
import org.springframework.beans.factory.BeanFactory;
import org.springframework.beans.factory.BeanFactoryAware;
import org.springframework.beans.factory.config.ConstructorArgumentValues;
import org.springframework.beans.factory.support.BeanDefinitionRegistry;
import org.springframework.beans.factory.support.BeanNameGenerator;
import org.springframework.beans.factory.support.RootBeanDefinition;
import org.springframework.context.annotation.ImportBeanDefinitionRegistrar;
import org.springframework.core.annotation.AnnotationAttributes;
import org.springframework.core.type.AnnotationMetadata;
import org.springframework.data.redis.listener.ChannelTopic;
import org.springframework.data.redis.listener.RedisMessageListenerContainer;
import org.springframework.data.redis.listener.adapter.MessageListenerAdapter;

import java.util.HashMap;
import java.util.Map;

/**
 * @desc 消息监听配置自动解析注册
 */
@Slf4j
public class RedisListenerRegistrar implements ImportBeanDefinitionRegistrar, BeanFactoryAware {

    private BeanFactory beanFactory;

    /**
     * 根据注解动态注册redis发布订阅配置
     * @param importingClassMetadata annotation metadata of the importing class
     * @param registry current bean definition registry
     * @param importBeanNameGenerator the bean name generator strategy for imported beans:
     * {@link ConfigurationClassPostProcessor#IMPORT_BEAN_NAME_GENERATOR} by default, or a
     * user-provided one if {@link ConfigurationClassPostProcessor#setBeanNameGenerator}
     * has been set. In the latter case, the passed-in strategy will be the same used for
     * component scanning in the containing application context (otherwise, the default
     * component-scan naming strategy is {@link AnnotationBeanNameGenerator#INSTANCE}).
     */
    @Override
    public void registerBeanDefinitions(AnnotationMetadata importingClassMetadata, BeanDefinitionRegistry registry, BeanNameGenerator importBeanNameGenerator) {
        final Map<String, Object> attributes = importingClassMetadata.getAnnotationAttributes(EnableRedisListener.class.getName());
        if (MapUtil.isEmpty(attributes)) {
            return;
        }
        final AnnotationAttributes[] listeners = (AnnotationAttributes[]) attributes.get("listeners");
        if (ArrayUtil.isEmpty(listeners)) {
            return;
        }
        String channelTopic;
        Class<? extends IRedisMessageReceiver> receiver;
        IRedisMessageReceiver bean;
        RootBeanDefinition rootBeanDefinition;
        ConstructorArgumentValues argumentValues;
        MessageListenerAdapter adapter;
        Map<String, MessageListenerAdapter> listenerAdapters = new HashMap<>(listeners.length);
        for (AnnotationAttributes listener : listeners) {
            //订阅通道
            channelTopic = listener.getString("channelTopic");
            //通道消息处理器
            receiver = listener.getClass("receive");
            log.info("\n---------建立redis订阅通道：{}，消息处理器：{}", channelTopic, receiver.getName());
            //先注册消息处理器
            bean = getReceiver(receiver, registry);
            if (bean == null) {
                continue;
            }
            //在注册监听器
            rootBeanDefinition = new RootBeanDefinition();
            rootBeanDefinition.setBeanClass(MessageListenerAdapter.class);
            //类构造器参数
            argumentValues = new ConstructorArgumentValues();
            argumentValues.addIndexedArgumentValue(0, bean);
            argumentValues.addIndexedArgumentValue(1, "receiveMessage");
            rootBeanDefinition.setConstructorArgumentValues(argumentValues);
            rootBeanDefinition.setSynthetic(true);
            //绑定通道和消息处理器，用组和名注册消息处理器，防止多个频道使用相同的消息处理器而重复注册
            final String listenerAdapterName = channelTopic + "_" + MessageListenerAdapter.class.getName();
            //注册bean
            registry.registerBeanDefinition(listenerAdapterName, rootBeanDefinition);
            adapter = (MessageListenerAdapter) this.beanFactory.getBean(listenerAdapterName);
            //为每个监听器指定序列化器，能够在反射监听方法（receiveMessage）时，根据方法入参的对象类型自动进行反序列化
            adapter.setSerializer(RedisConfig.jackson2JsonRedisSerializer());
            listenerAdapters.put(channelTopic, adapter);
        }
        //最后注册监听容器RedisMessageListenerContainer，将监听器放入容器
        rootBeanDefinition = new RootBeanDefinition();
        rootBeanDefinition.setBeanClass(RedisMessageListenerContainer.class);
        MutablePropertyValues propertyValues = new MutablePropertyValues();
        //绑定redis连接
        propertyValues.add("connectionFactory", this.beanFactory.getBean("syncRedisFactory"));
        rootBeanDefinition.setPropertyValues(propertyValues);
        registry.registerBeanDefinition(RedisMessageListenerContainer.class.getName(), rootBeanDefinition);
        RedisMessageListenerContainer container = (RedisMessageListenerContainer) this.beanFactory.getBean(RedisMessageListenerContainer.class.getName());
        for (String channelTopicKey : listenerAdapters.keySet()) {
            //加入监听器
            container.addMessageListener(listenerAdapters.get(channelTopicKey), new ChannelTopic(channelTopicKey));
        }
    }

    /**
     * 消息处理器不存在就注册
     * @param receiver
     * @param registry
     * @return
     */
    private IRedisMessageReceiver getReceiver(Class<? extends IRedisMessageReceiver> receiver, BeanDefinitionRegistry registry) {
        if (!this.beanFactory.containsBean(receiver.getName())) {
            RootBeanDefinition rootBeanDefinition = new RootBeanDefinition();
            rootBeanDefinition.setBeanClass(receiver);
            registry.registerBeanDefinition(receiver.getName(), rootBeanDefinition);
        }
        return this.beanFactory.getBean(receiver);
    }

    @Override
    public void setBeanFactory(BeanFactory beanFactory) throws BeansException {
        this.beanFactory = beanFactory;
    }
}

在这个类中我们拿到注解中的数据，再向bean工厂中注册订阅，相关类请看代码。
现在我们将这个类导入到开启注解上，这样消费者在使用此注解开启订阅后，会自动导入订阅类RedisListenerRegistrar来完成注册订阅：

import org.springframework.context.annotation.Import;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

/**
 * @desc 开启redis订阅监听
 */
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Import({RedisListenerConfig.class, RedisListenerRegistrar.class})
public @interface EnableRedisListener {

    /**
     * 消息监听配置注解
     * @return
     */
    RedisListener[] listeners() default {};
}

RedisListenerConfig为redis连接配置，不在多说，贴一份代码：

import io.lettuce.core.resource.ClientResources;
import lombok.Getter;
import lombok.Setter;
import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.autoconfigure.AutoConfigureAfter;
import org.springframework.boot.context.properties.bind.Binder;
import org.springframework.context.EnvironmentAware;
import org.springframework.context.annotation.Bean;
import org.springframework.core.env.Environment;
import org.springframework.data.redis.connection.RedisStandaloneConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceClientConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.connection.lettuce.LettucePoolingClientConfiguration;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.RedisSerializer;

import java.time.Duration;

/**
 * @desc redis发布订阅配置
 */
public class RedisListenerConfig {
    /**
     * 注册连接
     *
     * @param properties
     * @param clientResources
     * @return
     */
    @Bean(destroyMethod = "destroy")
    public LettuceConnectionFactory syncRedisFactory(RedisListenerProperties properties, ClientResources clientResources) {
        RedisStandaloneConfiguration redisConfiguration = new RedisStandaloneConfiguration(properties.getHost(), properties.getPort());
        redisConfiguration.setDatabase(properties.getDatabase());
        redisConfiguration.setUsername(properties.getUsername());
        redisConfiguration.setPassword(properties.getPassword());
        GenericObjectPoolConfig<String> genericObjectPoolConfig = new GenericObjectPoolConfig();
        genericObjectPoolConfig.setMaxIdle(properties.getMaxIdle());
        genericObjectPoolConfig.setMaxWait(Duration.ofMillis((long) properties.getMaxWait()));
        LettucePoolingClientConfiguration.LettucePoolingClientConfigurationBuilder builder = LettucePoolingClientConfiguration.builder().clientResources(clientResources).commandTimeout(Duration.ofMillis((long) properties.getMaxWait()));
        builder.shutdownTimeout(Duration.ofMillis(properties.getMaxWait()));
        builder.poolConfig(genericObjectPoolConfig);
        LettuceClientConfiguration lettuceClientConfiguration = builder.build();
        LettuceConnectionFactory lettuceConnectionFactory = new LettuceConnectionFactory(redisConfiguration, lettuceClientConfiguration);
        lettuceConnectionFactory.afterPropertiesSet();
        return lettuceConnectionFactory;
    }

    @Bean
    public ObjectRedisService syncObjectRedisService(@Qualifier("syncRedisFactory") LettuceConnectionFactory lettuceConnectionFactory) {
        RedisTemplate<String, Object> redisTemplate = new RedisTemplate();
        redisTemplate.setConnectionFactory(lettuceConnectionFactory);
        redisTemplate.setKeySerializer(RedisSerializer.string());
        redisTemplate.setValueSerializer(RedisSerializer.string());
        redisTemplate.setHashKeySerializer(RedisSerializer.string());
        redisTemplate.setHashValueSerializer(RedisSerializer.string());
        redisTemplate.afterPropertiesSet();
        return new ObjectRedisService(redisTemplate);
    }
}

import cn.hutool.core.collection.CollUtil;
import lombok.Getter;
import lombok.Setter;
import org.springframework.beans.factory.InitializingBean;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.util.Assert;

import java.util.List;

/**
 * @desc redis监听连接配置
 */
@Getter
@Setter
@ConfigurationProperties(prefix = RedisListenerProperties.PREFIX)
public class RedisListenerProperties implements InitializingBean {

    public static final String PREFIX = "hb.sync-redis";
    /**
     * 订阅的redis主机
     */
    private String host;
    /**
     * 端口
     */
    private int port;
    /**
     * 用户名
     */
    private String username;
    /**
     * 密码
     */
    private String password;
    /**
     * 连接库
     */
    private int database;
    /**
     * 连接超时时间(豪秒)
     */
    private int timeout;
    /**
     * 最大连接数
     */
    private int maxIdle;
    /**
     * 最大等待时间(豪秒)
     */
    private int maxWait;
    /**
     * 连接耗尽时是否阻塞, false报异常,true阻塞直到超时, 默认true
     */
    private Boolean blockWhenExhausted;
    /**
     * 是否启用pool的jmx管理功能, 默认true
     */
    private Boolean jmxEnabled;
    /**
     * 缓存过期时间
     */
    private long expireTime;
}

1.1.2.3、消费者使用

消费者所在项目引入基础依赖，实现依赖中暴露的消息处理接口完成消息的处理：

/**
 * @desc redis消息处理器
 */
public class RedisMessageReceiver implements IRedisMessageReceiver {
    @Override
    public void receiveMessage(RedisPubSubMessage message) {
        System.out.println("RedisMessageReceiver接收数据：" + message);
    }
}

配置文件：

hb:
  sync-redis:
    host: 127.0.0.1
    port: 6379
    database: 0
    # 连接超时时间(豪秒)
    timeout: 10000
    # 最大连接数
    maxIdle: 10
    # 最大等待时间(豪秒)
    maxWait: 10000
    # 连接耗尽时是否阻塞, false报异常,true阻塞直到超时, 默认true
    blockWhenExhausted: true
    # 是否启用pool的jmx管理功能, 默认true
    jmxEnabled: true
    # 缓存过期时间
    expireTime: 100000

将自定义注解加在消费者启动类上，并配置频道和handler，服务启动即可完成注册订阅，支持多频道订阅：

import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.scheduling.annotation.EnableScheduling;

@Slf4j
@EnableScheduling
@EnableRedisListener(listeners = {
        @RedisListener(channelTopic = "sync", receive = RedisMessageReceiver.class)
})
@SpringBootApplication
public class BpApplication {

    public static void main(String[] args) {
        ConfigurableApplicationContext context = SpringApplication.run(BpApplication.class, args);
        int beanDefinitionCount = context.getBeanDefinitionCount();
        log.info("BpApplication 启动成功, 加载bean数量: {}", beanDefinitionCount);
    }
}

1.2、版本2

1.2.1、订阅

版本1使用自定义注解方式实现开启订阅，做到了拿来即用，但是还不够灵活，最灵活的方式当然是放在配置中，使用spi根据配置参数自动配置，不用改代码就可以动态的调整，也更不容易引起代码冲突，是一个更优的解决方案。所以版本2中将移除自定义注解，将频道和handler相关配置放在配置文件中，首先调整配置文件：

import cn.hutool.core.collection.CollUtil;
import lombok.Getter;
import lombok.Setter;
import org.springframework.boot.context.properties.ConfigurationProperties;

import java.util.List;

/**
 * @desc redis监听连接配置
 */
@Getter
@Setter
@ConfigurationProperties(prefix = RedisListenerProperties.PREFIX)
public class RedisListenerProperties {

    public static final String PREFIX = "hb.sync-redis";
    /**
     * 总开关，是否开启redis监听
     */
    private Boolean enabledSub;
    /**
     * 订阅的redis主机
     */
    private String host;
    /**
     * 端口
     */
    private int port;
    /**
     * 用户名
     */
    private String username;
    /**
     * 密码
     */
    private String password;
    /**
     * 连接库
     */
    private int database;
    /**
     * 连接超时时间(豪秒)
     */
    private int timeout;
    /**
     * 最大连接数
     */
    private int maxIdle;
    /**
     * 最大等待时间(豪秒)
     */
    private int maxWait;
    /**
     * 连接耗尽时是否阻塞, false报异常,true阻塞直到超时, 默认true
     */
    private Boolean blockWhenExhausted;
    /**
     * 是否启用pool的jmx管理功能, 默认true
     */
    private Boolean jmxEnabled;
    /**
     * 缓存过期时间
     */
    private long expireTime;
    /**
     * 数据同步：频道和监听器配置
     */
    private List<SyncListenProperties> listenList;
    /**
     * 数据同步频道和监听器配置
     */
    @Getter
    @Setter
    public static class SyncListenProperties {
        /**
         * 监听的频道名称
         */
        private String channelTopic;
        /**
         * 消息处理器
         */
        private Class<? extends IRedisMessageReceiver> receive;
    }
}

1.2.1.1、调整注册订阅类

主要调整内容为从注解中获取配置数据改为从配置文件中获取

import cn.hutool.core.collection.CollUtil;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.MutablePropertyValues;
import org.springframework.beans.factory.config.ConstructorArgumentValues;
import org.springframework.beans.factory.support.RootBeanDefinition;
import org.springframework.context.support.GenericApplicationContext;
import org.springframework.data.redis.listener.ChannelTopic;
import org.springframework.data.redis.listener.RedisMessageListenerContainer;
import org.springframework.data.redis.listener.adapter.MessageListenerAdapter;

import javax.annotation.PostConstruct;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

/**
 * @desc 消息监听配置自动解析注册
 */
@Slf4j
@RequiredArgsConstructor
public class RedisListenerRegistrar {

    private final RedisListenerProperties properties;
    private final GenericApplicationContext context;

    /**
     * 根据配置动态注册redis发布订阅配置
     */
    @PostConstruct
    public void register() {
        final List<RedisListenerProperties.SyncListenProperties> listenList;
        //再次判断
        if (!properties.getEnableSub() || CollUtil.isEmpty(listenList = properties.getListenList())) {
            return;
        }
        String channelTopic;
        Class<? extends IRedisMessageReceiver> receiver;
        IRedisMessageReceiver bean;
        RootBeanDefinition rootBeanDefinition;
        ConstructorArgumentValues argumentValues;
        MessageListenerAdapter adapter;
        Map<String, MessageListenerAdapter> listenerAdapters = new HashMap<>(listenList.size());
        for (RedisListenerProperties.SyncListenProperties syncListenProperties : listenList) {
            receiver = syncListenProperties.getReceive();
            channelTopic = syncListenProperties.getChannelTopic();
            bean = this.getReceiver(receiver);
            if (bean == null) {
                continue;
            }
            log.info("\n---------建立redis订阅通道：{}，消息处理器：{}", channelTopic, receiver.getName());
            //在注册监听器
            rootBeanDefinition = new RootBeanDefinition();
            rootBeanDefinition.setBeanClass(MessageListenerAdapter.class);
            //类构造器参数
            argumentValues = new ConstructorArgumentValues();
            argumentValues.addIndexedArgumentValue(0, bean);
            argumentValues.addIndexedArgumentValue(1, "receiveMessage");
            rootBeanDefinition.setConstructorArgumentValues(argumentValues);
            rootBeanDefinition.setSynthetic(true);
            //绑定通道和消息处理器，用组和名注册消息处理器，防止多个频道使用相同的消息处理器而重复注册
            final String listenerAdapterName = channelTopic + "_" + MessageListenerAdapter.class.getName();
            //注册bean
            this.context.registerBeanDefinition(listenerAdapterName, rootBeanDefinition);
            adapter = (MessageListenerAdapter) this.context.getBean(listenerAdapterName);
            //为每个监听器指定序列化器，能够在反射监听方法（receiveMessage）时，根据方法入参的对象类型自动进行反序列化
            adapter.setSerializer(RedisConfig.jackson2JsonRedisSerializer());
            listenerAdapters.put(channelTopic, adapter);
        }
        //最后注册监听容器RedisMessageListenerContainer，将监听器放入容器
        rootBeanDefinition = new RootBeanDefinition();
        rootBeanDefinition.setBeanClass(RedisMessageListenerContainer.class);
        MutablePropertyValues propertyValues = new MutablePropertyValues();
        //绑定redis连接
        propertyValues.add("connectionFactory", this.context.getBean("syncRedisFactory"));
        rootBeanDefinition.setPropertyValues(propertyValues);
        this.context.registerBeanDefinition(RedisMessageListenerContainer.class.getName(), rootBeanDefinition);
        RedisMessageListenerContainer container = (RedisMessageListenerContainer) this.context.getBean(RedisMessageListenerContainer.class.getName());
        for (String channelTopicKey : listenerAdapters.keySet()) {
            //加入监听器
            container.addMessageListener(listenerAdapters.get(channelTopicKey), new ChannelTopic(channelTopicKey));
        }
    }

    /**
     * 注册/获取bean
     *
     * @param receiver
     * @return
     */
    private IRedisMessageReceiver getReceiver(Class<? extends IRedisMessageReceiver> receiver) {
        if (!this.context.containsBean(receiver.getName())) {
            this.context.registerBean(receiver);
        }
        return this.context.getBean(receiver);
    }
}

1.2.1.2、增加spi自动配置类

增加配置类，只要引入了相关依赖就能通过spi自动配置，在配置类上进行条件判断和相关类的导入

import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Import;

/**
 * @desc redis监听自定配置类，抽离出这个对象使能按条件决定是否开启监听
 */
@EnableConfigurationProperties(value = RedisListenerProperties.class)
@ConditionalOnProperty(prefix = RedisListenerProperties.PREFIX, name = "enabled-sub", havingValue = "true", matchIfMissing = false)
@Import({RedisListenerConfig.class, RedisListenerRegistrar.class})
public class RedisListenerAutoConfiguration {
}

1.2.1.3、加入spi文件

| -- resources
	| -- META-INF
    	| -- spring.factories
//内容:
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
org.xxx.RedisListenerAutoConfiguration

1.2.1.4、消费者使用

移除启动类上的自定义注解，在配置文件中加入配置：

hb:
	sync-redis:
  	enabled-sub: true
  	# 监听的频道与消息处理器
    listen-list: [
      { channel-topic: "sync", receive: org.xxx.RedisMessageReceiver }
    ]

启动项目即可完成自动订阅。

2、集成Redis-Stream消息队列

为了解决redis发布订阅模式消息丢失的问题，决定升级到redis的Stream消息队列，支持消息持久化，消息确认机制，是Redis5.0开始提供的一种轻量级的消息队列。
发布订阅的功能已经写好，那就让依赖同时支持发布订阅和Stream两种方式，决定权交给消费者。

2.1、最终版本

2.1.1、发布

增加Stream的消息发布方法：

final RedisTemplate<String, Object> redisTemplate;
	/**
     * 添加对象消息
     *
     * @param key
     * @param obj
     * @return
     */
    public RecordId xadd(String key, Object obj) {
        final ObjectRecord<String, Object> record = StreamRecords.objectBacked(obj).withStreamKey(key);
        return this.redisTemplate.opsForStream().add(record);
    }

    /**
     * 添加map消息
     *
     * @param key
     * @param message
     * @return
     */
    public RecordId addStream(String key, Map<String, Object> message) {
        return this.redisTemplate.opsForStream().add(key, message);
    }

    /**
     * 添加分组
     *
     * @param key
     * @param groupName
     */
    public void addGroup(String key, String groupName) {
        this.redisTemplate.opsForStream().createGroup(key, groupName);
    }

    /**
     * 分组是否存在
     *
     * @param key
     * @param groupName
     * @return
     */
    public Boolean groupExists(String key, String groupName) {
        final StreamInfo.XInfoGroups groups = this.redisTemplate.opsForStream().groups(key);
        if (groups == null) {
            return false;
        }
        return groups.stream().anyMatch(b -> Objects.equals(groupName, b.groupName()));
    }

    /**
     * 消费确认
     *
     * @param key
     * @param group
     * @param ids
     * @return
     */
    public Long ack(String key, String group, RecordId... ids) {
        return this.redisTemplate.opsForStream().acknowledge(key, group, ids);
    }

    /**
     * 删除指定id的消息
     *
     * @param key
     * @param ids
     * @return
     */
    public Long delField(String key, RecordId... ids) {
        if (ArrayUtil.isEmpty(ids)) {
            return 0L;
        }
        return this.redisTemplate.opsForStream().delete(key, ids);
    }

    /**
     * 获取未ack的消息列表
     *
     * @param key
     * @param group
     * @param consumer
     * @return
     */
    public PendingMessages pendingMessages(String key, String group, String consumer) {
        return this.redisTemplate.opsForStream().pending(key, Consumer.from(group, consumer));
    }

    /**
     * 消息是否确认，判断pending列表中有没有该消息
     * @param key
     * @param group
     * @param consumer
     * @param id
     * @return true:已确认
     */
    public Boolean msgIsAck(String key, String group, String consumer, RecordId id) {
        final PendingMessages pending = this.redisTemplate.opsForStream().pending(key, Consumer.from(group, consumer), Range.rightOpen(id.toString(), id.toString()), -1);
        return null == pending || pending.isEmpty();
    }

    /**
     * 消息是否确认，判断pending列表中有没有该消息
     * @param key
     * @param group
     * @param consumer
     * @param ids
     * @return true:已确认
     */
    public Boolean msgIsAck(String key, String group, String consumer, RecordId... ids) {
        final PendingMessages pendingMessages = this.pendingMessages(key, group, consumer);
        if (pendingMessages == null || pendingMessages.isEmpty()) {
            return true;
        }
        final ArrayList<RecordId> recordIds = CollUtil.toList(ids);
        return !pendingMessages.stream().anyMatch(b -> CollUtil.contains(recordIds, b.getId()));
    }

    /**
     * 获取消息
     *
     * @param key
     * @param range
     * @return
     */
    public List<MapRecord<String, Object, Object>> rangeMap(String key, Range<String> range) {
        return this.redisTemplate.opsForStream().range(key, range, RedisZSetCommands.Limit.unlimited());
    }

    /**
     * 获取消息
     *
     * @param cla
     * @param key
     * @param range
     * @param <T>
     * @return
     */
    public <T> List<ObjectRecord<String, T>> rangeObj(Class<T> cla, String key, Range<String> range) {
        return this.redisTemplate.opsForStream().range(cla, key, range, RedisZSetCommands.Limit.unlimited());
    }

2.1.2、消费

2.1.2.1、更改配置

调整配置文件，增加Stream相关配置：

import lombok.Getter;
import lombok.Setter;
import org.springframework.beans.factory.InitializingBean;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.context.properties.NestedConfigurationProperty;
import org.springframework.util.Assert;

/**
 * @desc redis监听连接配置
 */
@Getter
@Setter
@ConfigurationProperties(prefix = RedisListenerProperties.PREFIX)
public class RedisListenerProperties {

    public static final String PREFIX = "hb.sync-redis";

    /**
     * 订阅的redis主机
     */
    private String host;

    /**
     * 端口
     */
    private int port;

    /**
     * 用户名
     */
    private String username;

    /**
     * 密码
     */
    private String password;

    /**
     * 连接库
     */
    private int database;

    /**
     * 连接超时时间(豪秒)
     */
    private int timeout;

    /**
     * 最大连接数
     */
    private int maxIdle;

    /**
     * 最大等待时间(豪秒)
     */
    private int maxWait;

    /**
     * 连接耗尽时是否阻塞, false报异常,true阻塞直到超时, 默认true
     */
    private Boolean blockWhenExhausted;

    /**
     * 是否启用pool的jmx管理功能, 默认true
     */
    private Boolean jmxEnabled;

    /**
     * 缓存过期时间
     */
    private long expireTime;

    /**
     * 是否开启redis消息队列
     */
    private Boolean enableMq;

    /**
     * 发布订阅配置
     */
    @NestedConfigurationProperty
    private PubSubConfigProperties pubsub = new PubSubConfigProperties();

    /**
     * Stream消息队列配置
     */
    @NestedConfigurationProperty
    private StreamConfigProperties stream = new StreamConfigProperties();

}

独立出发布订阅和Stream的配置文件：

import cn.hutool.core.collection.CollUtil;
import lombok.Getter;
import lombok.Setter;
import java.util.List;

/**
 * @desc 发布订阅配置
 */
@Getter
@Setter
public class PubSubConfigProperties {
    
    /**
     * 数据同步：频道和监听器配置
     */
    private List<RedisPubSubConfigProperties> pubSubConfigProperties;

    public boolean isEmpty() {
        return CollUtil.isEmpty(pubSubConfigProperties);
    }

    /**
     * 数据同步频道和监听器配置
     */
    @Getter
    @Setter
    public static class RedisPubSubConfigProperties {

        /**
         * 监听的频道名称
         */
        private String channelTopic;

        /**
         * 消息处理器
         */
        private Class<? extends IRedisMessageReceiver> receive;
    }
}

import cn.hutool.core.collection.CollUtil;
import lombok.Data;
import lombok.Getter;
import lombok.Setter;
import java.util.List;

/**
 * @desc Stream消息队列配置
 */
@Getter
@Setter
public class StreamConfigProperties {
    
    /**
     * stream消息队列
     */
    private List<RedisStreamConfigProperties> streamConfigProperties;

    public boolean isEmpty() {
        return CollUtil.isEmpty(streamConfigProperties);
    }

    /**
     * Stream消息队列配置
     */
    @Getter
    @Setter
    public static class RedisStreamConfigProperties {
        /**
         * 消费者分组
         */
        private String group = "sync_group";
        /**
         * 消息队列key
         */
        private String key = "sync";
        /**
         * 消费者名称
         */
        private String consumerName = "consumer_";
        /**
         * 自动消息确认，默认关闭
         */
        private boolean autoAck;
        /**
         * 处理器
         */
        private Class<? extends AbstractSyncStreamListener> listener;
    }
}

2.1.2.2、独立两种方式的spi自动配置类

发布订阅配置：

import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Conditional;
import org.springframework.context.annotation.Import;

/**
 * @desc redis监听自定配置类，抽离出这个对象使能按条件决定是否开启监听
 */
@EnableConfigurationProperties(value = RedisListenerProperties.class)
@Conditional(PubSubCondition.class)
@Import({RedisListenerConfig.class, RedisPubSubRegistrar.class})
public class RedisPubSubConfiguration {

    /**
 * @desc redis发布订阅模式开启条件类
 */
static class PubSubCondition implements Condition {
    @Override
    public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) {
        //无法直接获取配置参数对象，使用binder API获取
        final RedisListenerProperties property = Binder.get(context.getEnvironment()).bind(RedisListenerProperties.PREFIX, RedisListenerProperties.class).get();
        return property.getEnabledMq() && !property.getPubsub().isEmpty();
    }
}
}

RedisPubSubRegistrar注册订阅类不变。
增加Stream配置类：

import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.MutablePropertyValues;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.support.RootBeanDefinition;
import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Conditional;
import org.springframework.context.annotation.Import;
import org.springframework.context.support.GenericApplicationContext;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.connection.stream.*;
import org.springframework.data.redis.hash.ObjectHashMapper;
import org.springframework.data.redis.stream.StreamListener;
import org.springframework.data.redis.stream.StreamMessageListenerContainer;

import java.time.Duration;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.LinkedBlockingDeque;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;

/**
 * @desc Stream自动配置类
 */
@Slf4j
@EnableConfigurationProperties(RedisListenerProperties.class)
@Conditional(StreamCondition.class)
@Import({RedisListenerConfig.class})
public class RedisStreamConfiguration {
    private final GenericApplicationContext context;
    private final ObjectRedisService syncObjectRedisService;
    private List<StreamConfigProperties.RedisStreamConfigProperties> streamConfigProperties;

    public RedisStreamConfiguration(GenericApplicationContext context,
                                    @Qualifier("syncObjectRedisService") ObjectRedisService syncObjectRedisService,
                                    RedisListenerProperties properties) {
        this.context = context;
        this.syncObjectRedisService = syncObjectRedisService;
        this.streamConfigProperties = properties.getStream().getStreamConfigProperties();
    }

    /**
     * Stream消息队列监听容器
     * @param lettuceConnectionFactory
     * @return
     */
    @Bean(initMethod = "start", destroyMethod = "stop")
    @ConditionalOnClass(RedisListenerProperties.class)
    public StreamMessageListenerContainer<String, ObjectRecord<String, RedisPubSubMessage>> streamMessageListenerContainer(@Qualifier("syncRedisFactory") LettuceConnectionFactory lettuceConnectionFactory) {
        AtomicInteger index = new AtomicInteger(1);
        int processors = Runtime.getRuntime().availableProcessors();
        ThreadPoolExecutor executor = new ThreadPoolExecutor(processors, processors, 0, TimeUnit.SECONDS,
                new LinkedBlockingDeque<>(), r -> {
            Thread thread = new Thread(r);
            thread.setName("async-stream-consumer-" + index.getAndIncrement());
            thread.setDaemon(true);
            return thread;
        });
        StreamMessageListenerContainer.StreamMessageListenerContainerOptions<String, ObjectRecord<String, RedisPubSubMessage>> options =
                StreamMessageListenerContainer.StreamMessageListenerContainerOptions
                        .builder()
                        // 一次最多获取多少条消息
                        .batchSize(10)
                        // 运行 Stream 的 poll task
                        .executor(executor)
//                        // Stream Key 的序列化形式
//                        .keySerializer(RedisSerializer.string())
//                        // Stream field 的序列化形式
//                        .hashKeySerializer(RedisSerializer.string())
//                        // Stream value 的序列化形式
//                        .hashValueSerializer(RedisConfig.jackson2JsonRedisSerializer())
                        // Stream 中没有消息时，阻塞多长时间，必须要比 `spring.redis.timeout` 小
                        .pollTimeout(Duration.ofSeconds(1))
                        // ObjectRecord 时，将 对象的 filed 和 value 转换成一个 Map 比方：将Book对象转换成map
                        .objectMapper(new ObjectHashMapper())
                        // 异常处理
//                        .errorHandler(new CustomErrorHandler())
                        // 将发送到Stream中的Record转换成ObjectRecord，转换成具体的类型
                        .targetType(RedisPubSubMessage.class)
                        .build();
        StreamMessageListenerContainer<String, ObjectRecord<String, RedisPubSubMessage>> streamMessageListenerContainer =
                StreamMessageListenerContainer.create(lettuceConnectionFactory, options);
        StreamConfigProperties.RedisStreamConfigProperties property;
        for (int i = 0; i < streamConfigProperties.size(); i++) {
            property = streamConfigProperties.get(i);
            this.initStream(property.getKey(), property.getGroup());
            // 注册消费者
            streamMessageListenerContainer.register(
                    StreamMessageListenerContainer.StreamReadRequest.builder(StreamOffset.create(property.getKey(), ReadOffset.lastConsumed()))
                            .consumer(Consumer.from(property.getGroup(), property.getConsumerName()))
                            .autoAcknowledge(property.isAutoAck())
                            .build(),
                    this.getListener(property, i));
            log.info("\n----------------------->建立redis Stream 消息队列，key:{}, group:{}, consumer:{}, autoAck:{}, 消息处理器：{}", property.getKey(), property.getGroup(), property.getConsumerName(), property.isAutoAck(), property.getListener());
        }
        return streamMessageListenerContainer;
    }

    /**
     * 注册/获取bean
     *
     * @param property 处理器配置
     * @param index 下标，相同类型的多个处理器，在注册时加上下标防止重复
     * @return
     */
    private StreamListener getListener(StreamConfigProperties.RedisStreamConfigProperties property, int index) {
        final Class<? extends AbstractSyncStreamListener> listener = property.getListener();
        final String beanName = listener.getName() + index;
        if (!this.context.containsBean(beanName)) {
            //注册监听器
            final RootBeanDefinition rootBeanDefinition = new RootBeanDefinition(listener);
            MutablePropertyValues propertyValues = new MutablePropertyValues();
            propertyValues.add("streamKey", property.getKey());
            propertyValues.add("group", property.getGroup());
            propertyValues.add("autoAck", property.isAutoAck());
            propertyValues.add("consumer", property.getConsumerName());
            rootBeanDefinition.setPropertyValues(propertyValues);
            this.context.registerBeanDefinition(beanName, rootBeanDefinition);
        }
        return this.context.getBean(beanName, listener);
    }

    /**
     * 初始化分组
     *
     * @param key
     * @param group
     */
    private void initStream(String key, String group) {
        //判断key是否存在，如果不存在则创建
        boolean hasKey = this.syncObjectRedisService.hasKey(key);
        if (!hasKey) {
            Map<String, Object> map = new HashMap<>();
            map.put("field", "value");
            RecordId recordId = this.syncObjectRedisService.addStream(key, map);
            this.syncObjectRedisService.addGroup(key, group);
            //将初始化的值删除掉
            this.syncObjectRedisService.delField(key, recordId);
            log.info("stream:{}-group:{} initialize success", key, group);
        } else if (!this.syncObjectRedisService.groupExists(key, group)) {
            this.syncObjectRedisService.addGroup(key, group);
        }
    }

    /**
 * @desc redis消息队列Stream开启条件类
 */
static class StreamCondition implements Condition {
    @Override
    public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) {
        //无法直接获取配置参数对象，使用binder API获取
        final RedisListenerProperties property = Binder.get(context.getEnvironment()).bind(RedisListenerProperties.PREFIX, RedisListenerProperties.class).get();
        return property.getEnabledMq() && !property.getStream().isEmpty();
    }
}
}

抽象出Stream消息处理的抽象类，完成对消息的确认消费的处理，消费者继承这个抽象类完成实际业务的处理：

import lombok.Getter;
import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.data.redis.connection.stream.ObjectRecord;
import org.springframework.data.redis.connection.stream.RecordId;
import org.springframework.data.redis.stream.StreamListener;

/**
 * @desc 数据同步抽象监听类
 */
@Slf4j
@Getter
@Setter
public abstract class AbstractSyncStreamListener implements StreamListener<String, ObjectRecord<String, RedisPubSubMessage>> {

    private final ObjectRedisService redisService;

    public AbstractSyncStreamListener(@Qualifier("syncObjectRedisService") ObjectRedisService redisService) {
        this.redisService = redisService;
    }

    /**
     * 消息队列的key
     */
    private String streamKey;

    /**
     * 分组
     */
    private String group;

    /**
     * 消费者名称
     */
    private String consumer;

    /**
     * 消息自动确认
     */
    private boolean autoAck;

    /**
     * 处理消息数据
     * @param message 消息内容
     */
    public abstract void handleMessage(RedisPubSubMessage message);

    /**
     * 接收stream队列消息
     * @param message never {@literal null}.
     */
    @Override
    public void onMessage(ObjectRecord<String, RedisPubSubMessage> message) {
        log.info("\n---------------------{}接收数据：{}", this.getClass().getName(), message.toString());
        final RecordId id = message.getId();
        final String key = message.getStream();
        try {
            this.handleMessage(message.getValue());
            if (!this.autoAck) {
                //手动确认
                this.redisService.ack(key, this.group, id);
            }
            if (this.redisService.msgIsAck(key, this.group, this.consumer, id)) {
                //删除消息
                this.redisService.delField(key, id);
            }
        } catch (Exception e) {
            log.error(e.getMessage(), e);
        }
    }
}

修改spi文件内容：

org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
com.xxx.RedisPubSubConfiguration,\
com.xxx.RedisStreamConfiguration

2.1.2.3、使用

引入了依赖的消费者根据需要决定使用哪一个或是两个都用，配置如下：

hb:
  sync-redis:
    host: 127.0.0.1
    port: 6379
    database: 0
    # 连接超时时间(豪秒)
    timeout: 10000
    # 最大连接数
    maxIdle: 10
    # 最大等待时间(豪秒)
    maxWait: 10000
    # 连接耗尽时是否阻塞, false报异常,true阻塞直到超时, 默认true
    blockWhenExhausted: true
    # 是否启用pool的jmx管理功能, 默认true
    jmxEnabled: true
    # 缓存过期时间
    expireTime: 100000
    # 是否开启redis消息队列
    enabled-mq: true
    # 监听的频道与消息处理器
    pubsub: [
      { channel-topic: "sync", receive: org.xxx.RedisMessageReceiver }
    ]
    stream: [
      { group: "sync_group",
        auto-ack: false,
        key: "sync",
        consumer-name: "consumer",
        listener: org.xxx.RedisStreamMessageReceiver }
    ]

总结：
Stream中未被消费的消息仍然会在等待队列中，需要消费者自己再去从等待队列中处理消息。Stream模式虽然比发布订阅更安全，也仍然可能会出现消息丢失的情况，需要消费者做好消息丢失的补偿措施，开启redis的持久化，消费端建立异常表等，这里就不在讨论了，感谢观看！