文章基于rocket-mq4.0 代码分析
主要分析消息拉取流程
Client端启动入口
以Push模式为例
org.apache.rocketmq.client.consumer.DefaultMQPushConsumer#start
-->org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl#start
------------------------------------------------------
public void start() throws MQClientException {
System.out.println(this.getSubscriptionInner());
switch (this.serviceState) {
case CREATE_JUST:
//....省略代码....
mQClientFactory.start();
log.info("the consumer [{}] start OK.", this.defaultMQPushConsumer.getConsumerGroup());
this.serviceState = ServiceState.RUNNING;
break;
case RUNNING:
case START_FAILED:
case SHUTDOWN_ALREADY:
throw new MQClientException("The PushConsumer service state not OK, maybe started once, "//
+ this.serviceState//
+ FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
null);
default:
break;
}
this.updateTopicSubscribeInfoWhenSubscriptionChanged();
this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
this.mQClientFactory.rebalanceImmediately();
}
主要是下面标注代码
mQClientFactory 是 MQClientInstance的实例,主要通过
this.pullMessageService = new PullMessageService(this);
this.rebalanceService = new RebalanceService(this);
两个service配合完成消息的拉取
PullMessageService 和 RebalanceService 都继承了 ServiceThread
PullMessageService 启动后会阻塞在
PullRequest pullRequest = this.pullRequestQueue.take();
获取PullRequest对象,再根据这个对象请求broker获得消息体
而PullRequest对象是RebalanceService启动后根据topic定时到broker查询该topic是否变更了(定时调 org.apache.rocketmq.client.impl.consumer.RebalanceImpl#doRebalance方法
),如果变更了才会生成该对象并放入到
org.apache.rocketmq.client.impl.consumer.PullMessageService#pullRequestQueue 中去
RebalanceService
在run方法启动后,通过一些列调用(类相互关系复杂);以push模式为例
MQClientInstance.doRebalance()
-->DefaultMQPushConsumerImpl.doRebalance()
-->RebalanceImpl.doRebalance(final boolean isOrder)
-->RebalanceImpl.rebalanceByTopic(final String topic, final boolean isOrder)
最后更消息模式[BROADCASTING,CLUSTERING]去获取是否改拉取消息了
以 CLUSTERING 为例
case CLUSTERING: {
//mqSet 为该topic在broker端逻辑队列comsumequeue集合的映射,该值会有单独线程定时更新;
//例如:org.apache.rocketmq.client.impl.factory.MQClientInstance#startScheduledTask,还有其他地方也会更新该值
Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
List<String> cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);
//省略代码.....
if (mqSet != null && cidAll != null) {
List<MessageQueue> mqAll = new ArrayList<MessageQueue>();
mqAll.addAll(mqSet);
Collections.sort(mqAll);
Collections.sort(cidAll);
//通过消费端负载均衡策略,client获得自己的MessageQueue
AllocateMessageQueueStrategy strategy = this.allocateMessageQueueStrategy;
List<MessageQueue> allocateResult = null;
try {
allocateResult = strategy.allocate(
this.consumerGroup,
this.mQClientFactory.getClientId(),
mqAll,
cidAll);
} catch (Throwable e) {
log.error("AllocateMessageQueueStrategy.allocate Exception. allocateMessageQueueStrategyName={}", strategy.getName(),
e);
return;
}
Set<MessageQueue> allocateResultSet = new HashSet<MessageQueue>();
if (allocateResult != null) {
allocateResultSet.addAll(allocateResult);
}
//判断改topic是否有新消息可以拉取了
boolean changed = this.updateProcessQueueTableInRebalance(topic, allocateResultSet, isOrder);
if (changed) {
//省略代码.....
//组装PullMessageService拉取消息所需的PullRequest对象并放入队列中
this.messageQueueChanged(topic, mqSet, allocateResultSet);
}
}
break;
}
PullMessageService
PullMessageService 获取到 PullRequest后最终执行方法到:
org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl#pullMessage
然后是一个很长的 PullCallback,会在拉取消息成功后调用 ConsumeMessageService执行(说实话,这代码不是一般的绕)
只有结果为 pullResult.getPullStatus()为FOUND才会执行 submitConsumeRequest 请求(用户自定义注册监听执行逻辑),其他的貌似都会放入重试
最终是我们注册的监听执行了自定义逻辑(DefaultMQPushConsumer 即push模式下我们需要实现 MessageListenerOrderly 或者 MessageListenerConcurrently )
消费进度更新
ConsumeMessageService 目前框架提供了两种实现,
ConsumeMessageOrderlyService 和 ConsumeMessageConcurrentlyService,框架会根据我们实现的自定义监听类型注入对应的实现类;
我理解的ConsumeMessageConcurrentlyService 和 ConsumeMessageOrderlyService关键区别在于
其内部类ConsumeRequest在执行run方法时后者通过加锁的方式,旨在保证获得锁的时候框架才处理对应的消息;
如果从comsumequeue里获取到了消息,
会执行用户自定义监听:
这里会根据自定义程序返回的status做一个处理,如果监听程序返回的是null,则会被赋值成 :ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT
continueConsume = ConsumeMessageOrderlyService.this.processConsumeResult(msgs, status, context, this);
这个方法会根据status的值做不同的逻辑处理,如果是 SUSPEND_CURRENT_QUEUE_A_MOMENT,则会在校验是否可以重复消费后,另外提交一个线程池任务处理该请求
是否可重复消费校验:
private boolean checkReconsumeTimes(List<MessageExt> msgs) {
boolean suspend = false;
if (msgs != null && !msgs.isEmpty()) {
for (MessageExt msg : msgs) {
//********这里的 getMaxReconsumeTimes() 默认是 Integer.MAX_VALUE
if (msg.getReconsumeTimes() >= getMaxReconsumeTimes()) {
MessageAccessor.setReconsumeTime(msg, String.valueOf(msg.getReconsumeTimes()));
//*******如果客户端重试次数大于了设置的次数,则构建 RETRY 消息重新发送至broker端
if (!sendMessageBack(msg)) {
suspend = true;
msg.setReconsumeTimes(msg.getReconsumeTimes() + 1);
}
} else {
suspend = true;
msg.setReconsumeTimes(msg.getReconsumeTimes() + 1);
}
}
}
return suspend;
}
提交线程池新任务:
private void submitConsumeRequestLater(
final ProcessQueue processQueue,
final MessageQueue messageQueue,
final long suspendTimeMillis
) {
long timeMillis = suspendTimeMillis;
if (timeMillis == -1) {
timeMillis = this.defaultMQPushConsumer.getSuspendCurrentQueueTimeMillis();
}
if (timeMillis < 10) {
timeMillis = 10;
} else if (timeMillis > 30000) {
timeMillis = 30000;
}
//*****重新提交线程池处理*****
this.scheduledExecutorService.schedule(new Runnable() {
@Override
public void run() {
ConsumeMessageOrderlyService.this.submitConsumeRequest(null, processQueue, messageQueue, true);
}
}, timeMillis, TimeUnit.MILLISECONDS);
}
在执行完自定义逻辑后就会向broker发起更新消费offset的请求
****to continue
附客户端代码:
public class Consumer {
public static void main(String[] args) throws InterruptedException, MQClientException {
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("ConsumerGroupNamecc4");
consumer.setNamesrvAddr("127.0.0.1:9876");
consumer.subscribe("topicaaa","TagA");
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
System.out.printf(Thread.currentThread().getName() + " Receive1 New Messages: " + msgs + "%n");
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
consumer.start();
System.out.printf("Consumer Started.%n");
}
}