MDC

这个问题是我当初写项目时遇到的,因为用RabbitMQ做削峰处理,高并发情况下,channel数到达了限制,所以不能继续创建,相信大家也遇到过。

正常来说,这个错误还是比较少见的,只不过项目需要保证消息的可靠性,所以采取了发送确认和消费手动确认机制,导致并发性能下降,从而出现这个问题。、

这里先上结论,方便着急的小伙伴们改bug。

 结论:RabbitMQ java客户端在创建连接时,会向服务端发送一个请求,这个请求会获取到服务端的channelMax值,java客户端会自己进行一个处理,两者都不为0时,会选择一个小的值,如果你没有在rabbitmq.conf文件中修改channel_Max的值,那么java客户端会采用默认的2047或更小,这就会导致你明明在客户端连接上配置了channelMax(比如你配置了4095),但依旧会报错,而且web管理页面最大值依旧是2047

第一次修改配置不生效

出现这种情况经常伴随着消息丢失,而且消息丢失情况非常严重,达到了百分之二十的丢失率,这个丢失率也会因为并发量、每次消费数量等等配置的不同而变化。

由于项目是基于SpringBoot2.2的,yml暂时无法配置RequestChannelMax的值,这里只能采用直接通过set的方式放入值。

@Configuration
@Slf4j
public class RabbitMQConfig {


@Bean
public RabbitTemplate rabbitTemplate(final ConnectionFactory connectionFactory) {
CachingConnectionFactory cachingConnectionFactory = (CachingConnectionFactory) connectionFactory;
//这里我明明设置了4095,但是项目运行之后,压测之后,还是会报异常,而且报异常的时候,RabbitMQ //web管理页面上的channel数依旧是2047,不得已只能分析源码了
cachingConnectionFactory.getRabbitConnectionFactory().setRequestedChannelMax(4095);
final RabbitTemplate rabbitTemplate = new RabbitTemplate(cachingConnectionFactory);
rabbitTemplate.setMessageConverter(jackson2MessageConverter());
rabbitTemplate.setMandatory(true);
rabbitTemplate.setConfirmCallback((correlationData, b, s) -> {
if(!b){
log.error("confirmCallBack 发送失败的数据:{}",correlationData);
log.error("confirmCallBack 确认情况:{}",b);
log.error("confirmCallBack 发送失败的原因:{}",s);
}
});
rabbitTemplate.setReturnCallback((message, i, s, s1, s2) -> {
log.error("returnCallBack 消息:{}",message);
log.error("returnCallBack 回应码:{}",i);
log.error("returnCallBack 回应信息:{}",s);
log.error("returnCallBack 交换机:{}",s1);
log.error("returnCallBack 路由键:{}",s2);
});
return rabbitTemplate;
}

@Bean
public Jackson2JsonMessageConverter jackson2MessageConverter() {
return new Jackson2JsonMessageConverter();
}
}

分析源码

首先是模拟出报错的场景,然后进入报异常的类。

发现是this.delegate.createChannel();方法返回的是一个空channel对象,进入这个方法看一下。

RabbitMQ:The channelMax limit is reached. Try later._jar

 

 

 

发现有一个ChannelManager对象,顾名思义,就是一个channel管理器,由它负责创建channel,那么看一下这个对象都有什么值呢?

com.rabbitmq.client.impl.ChannelManager#createChannel(com.rabbitmq.client.impl.AMQConnection)
public Channel createChannel() throws IOException {
this.ensureIsOpen();
ChannelManager cm = this._channelManager;
if (cm == null) {
return null;
} else {
Channel channel = cm.createChannel(this);
this.metricsCollector.newChannel(channel);
return channel;
}
}

只截取了部分代码,首先可以看到有一个int类型的channelMax,这个值就是channel的最大值,还有一个构造器,很明显,这个值是通过构造器传进来的,通过容器初始化时打断点进行跟踪,发现此时的channelMax依旧是2047,这也进一步证明了,值的覆盖或者处理发生在这个类调用之前。

com.rabbitmq.client.impl.ChannelManager
public class ChannelManager {
private static final Logger LOGGER = LoggerFactory.getLogger(ChannelManager.class);
private final Object monitor;
private final Map<Integer, ChannelN> _channelMap;
private final IntAllocator channelNumberAllocator;
private final ConsumerWorkService workService;
private final Set<CountDownLatch> shutdownSet;
private final int _channelMax;
private ExecutorService shutdownExecutor;
private final ThreadFactory threadFactory;
private int channelShutdownTimeout;
protected final MetricsCollector metricsCollector;

public ChannelManager(ConsumerWorkService workService, int channelMax, ThreadFactory threadFactory, MetricsCollector metricsCollector) {
this.monitor = new Object();
this._channelMap = new HashMap();
this.shutdownSet = new HashSet();
this.channelShutdownTimeout = 63000;
if (channelMax == 0) {
channelMax = 65535;
}

this._channelMax = channelMax;
this.channelNumberAllocator = new IntAllocator(1, channelMax);
this.workService = workService;
this.threadFactory = threadFactory;
this.metricsCollector = metricsCollector;
}
}

 

进一步跟踪之后,发现在AMQConnection类里的instantiateChannelManager()方法调用了构造器,继续往上追踪。

com.rabbitmq.client.impl.AMQConnection#instantiateChannelManager
protected ChannelManager instantiateChannelManager(int channelMax, ThreadFactory threadFactory) {
ChannelManager result = new ChannelManager(this._workService, channelMax, threadFactory, this.metricsCollector);
this.configureChannelManager(result);
return result;
}

在AMQConnetion类的start()方法中最终发现了值改变的地方。

this.requestedChannelMax值是我在配置类中配置的4095

connTune.getChannelMax()是2047

也就是说,negotiateChannelMax()方法对这两个值进行了处理,最终选择了2047

int channelMax = this.negotiateChannelMax(this.requestedChannelMax, connTune.getChannelMax());
this._channelManager = this.instantiateChannelManager(channelMax, this.threadFactory);

最终发现这么一段处理逻辑,如果两个数字都不为0,那么就取最小的,反之取最大的,看到这里是明白做了什么处理,但是还是有一处不明白,2047的值究竟从何处来的?

com.rabbitmq.client.impl.AMQConnection#negotiateChannelMax
/**
* Private API, allows for easier simulation of bogus clients.
*/
protected int negotiateChannelMax(int requestedChannelMax, int serverMax) {
return negotiatedMaxValue(requestedChannelMax, serverMax);
}


private static int negotiatedMaxValue(int clientValue, int serverValue) {
return (clientValue == 0 || serverValue == 0) ?
Math.max(clientValue, serverValue) :
Math.min(clientValue, serverValue);
}

 

通过对connTune的追寻,发现了这段处理,debug也证明了确实在这里获取的2047这个值,

其实不管从方法名rpc()还是变量名serverResponse来看,这个都是做了一个请求,那么向谁请求其实很显而易见了,这里向RabbitMQ端做了一个请求,用来索取MQ端的channelMax、frameMax、heartBeat值等等

Tune connTune = null;

try {
......
try {
Method serverResponse = this._channel0.rpc((Method)method, this.handshakeTimeout / 2).getMethod();
if (serverResponse instanceof Tune) {
connTune = (Tune)serverResponse;
}
......

到现在其实就很明确了,我们只在客户端修改边界值配置是无效的,必须同步修改MQ服务端的配置,也就是rabbitmq.conf文件

## Set the max permissible number of channels per connection.
## 0 means "no limit".
##在配置文件中,输入以下参数和自己想要设置的值即可,如果用不到2047,那就不用配置
# channel_max = 128

其实问题并不大,主要还是不了解MQ的一个客户端连接过程,导致耗费了大量时间。

 

这里还是推荐大家,先用百度搜索,第一页看不到正确解决方案,那就去StackOverflow网站,还不行的话,那就使用终极大法,要么官网逐行看文档,要么走一波源码,也是锻炼自己解决问题的思路和能力。


https://github.com/spring-projects/spring-amqp/issues/853

2020-09-09 14:02:36.549  INFO 6 --- [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#10-60623] c.r.c.impl.AbstractMetricsCollector      : Error while computing metrics in newChannel: null
2020-09-09 14:02:36.549 ERROR 6 --- [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#10-60623] o.s.a.r.l.SimpleMessageListenerContainer : Failed to check/redeclare auto-delete queue(s).

org.springframework.amqp.AmqpResourceNotAvailableException: The channelMax limit is reached. Try later.
at org.springframework.amqp.rabbit.connection.SimpleConnection.createChannel(SimpleConnection.java:60) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createBareChannel(CachingConnectionFactory.java:1434) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.access$200(CachingConnectionFactory.java:1420) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.doCreateBareChannel(CachingConnectionFactory.java:723) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:706) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getCachedChannelProxy(CachingConnectionFactory.java:676) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getChannel(CachingConnectionFactory.java:567) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$1600(CachingConnectionFactory.java:102) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createChannel(CachingConnectionFactory.java:1439) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:2095) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2062) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2042) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.core.RabbitAdmin.getQueueInfo(RabbitAdmin.java:407) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.core.RabbitAdmin.getQueueProperties(RabbitAdmin.java:391) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.attemptDeclarations(AbstractMessageListenerContainer.java:1842) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.redeclareElementsIfNecessary(AbstractMessageListenerContainer.java:1823) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.initialize(SimpleMessageListenerContainer.java:1349)[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1195) [spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]

2020-09-09 14:02:36.549 INFO 6 --- [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#10-60623] c.r.c.impl.AbstractMetricsCollector : Error while computing metrics in newChannel: null