Rocketmq延迟服务无法正常消费数据

原创

sqtce 2020-03-18 20:30:26 博主文章分类：RocketMQ ©著作权

文章标签 rocketmq 文章分类 运维

©著作权归作者所有：来自51CTO博客作者sqtce的原创作品，请联系作者获取转载授权，否则将追究法律责任

背景

架构部署

32C 128G 1.8T 4主4从架构部署

rocketmq延迟服务生产者生产数据，消费者无法正常消费到数据，查看日志有重大发现，所有的主节点均有这报错

#vim /data2/logs/rocketmqlogs/storeerror.log

020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:57 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:57 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:57 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:57 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1

发现消费的位点全部错乱，导致消费者无法常消费到数据

延迟消息存储时被替换为“SCHEDULE_TOPIC_XXXX”主题，broker会为每个等级的建立定时任务进行调度，将各个等级到时间的消息替换为原来的主题，从而消费者可以消费到该消息。以上日志加上消息查询结果已存储在“SCHEDULE_TOPIC_XXXX”，可以断定调度环节出问题了，由于offset远远超过最大offset而报错；没能将目标主题成功替换。

解决方法

先将broker节点的流量清除，2表示只写权限，4表示只读权限，6表示读写权限

#bin/mqadmin updateBrokerConfig -b brokerip:10911 -n nameserver地址:9876 -k brokerPermission -v 4

查看broker节点流量是否为0,需要等待broker完全没有流量了才可以进行关闭broker节点操作，以防止丢数据

#bin/mqadmin clusterList -n nameserver:9876

关闭broker节点进程

#bin/mqshutdown broker

将delayOffset.json 和delayOffset.json.bak文件拷走此文件用于记录消费位点的，拷走后，重启应用进程会自动生产

#mv /data1/rocketmq/store/config/delayOffset.json* /data2/

将SCHEDULE_TOPIC_XXXX 文件拷走，用于调度任务，拷走后，重启应用进程会自动生产

#mv /data1/rocketmq/store/consumequeue/SCHEDULE_TOPIC_XXXX/ /data2/

将rocketmq的日志文件全部删除