mongo too many open files怎么解决

转载

梦断蓝桥魂 2024-07-09 07:47:21

文章标签 MongoDB GridFS spring mongodb java 文章分类 架构后端开发

背景

在开发中可能遇到这样的场景：比如使用MongoDB存储文件，但是同时又想支持MinIO 方式。在代码层面做了多种方式的接入。但就在做接入的时候遇到这样的问题：
在application.properties 中配置了MongoDB的连接：

file.storage.mode=MONGODB
spring.data.mongodb.host=MONGO_DB_HOST
spring.data.mongodb.database=MONGO_DB_DATABASE

熟悉SpringBoot 的小伙伴应该都知道，这样写是对应了SpingBoot 中的一个配置类的属性文件：org.springframework.boot.autoconfigure.mongo.MongoProperties

由于此处是用 MongoDB 来存储文件，另又认为MongoDB默认连接超时的时间过长。因此重新进行相关的类进行配置：

@Configuration
@ConditionalOnProperty(name = "file.storage.mode",havingValue = "MONGODB")
@EnableConfigurationProperties(MongoProperties.class)
public class MongoDBConfiguration {

    @Autowired
    private MongoProperties properties;

    @Bean
    public MongoClient mongoClient() {
        MongoClientOptions.Builder builder = MongoClientOptions.builder().
        connectTimeout(8000).serverSelectionTimeout(8000);
        return new MongoClient(properties.getHost(), builder.build());
    }

    @Bean
    public GridFSBucket gridFSBucket(MongoClient mongoClient) {
        MongoDatabase database = mongoClient.getDatabase(properties.getDatabase());
        GridFSBucket bucket = GridFSBuckets.create(database);
        return bucket;
    }

}

现象

将MongoDB的配置写好后，启动系统测试上传文件，并没有问题，可以正常存进数据库中。但是将如果将application.properties 中的属性：file.storage.mode 改为：MinIO （ps: 此时系统不再需要配置MONGO_DB_HOST的值，直接空字符串就好。）但在系统启动时，会提示MongoDB连接失败，并且连接的主机默认为：127.0.0.1。虽然并不影响系统后续的运行，但是总是看着这个报错不太舒服，再想想以后这个代码还要上到生产环境的就更不舒畅了。具体报错:

INFO 21364 --- [           main] org.mongodb.driver.cluster               : Cluster created with settings {hosts=[127.0.0.1:27017], mode=SINGLE, requiredCluste
rType=UNKNOWN, serverSelectionTimeout='8000 ms', maxWaitQueueSize=500}
INFO 21364 --- [127.0.0.1:27017] org.mongodb.driver.cluster               : Exception in monitor thread while connecting to server 127.0.0.1:27017

com.mongodb.MongoSocketOpenException: Exception opening socket
        at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70)
        at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:128)
        at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:117)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused: connect
        at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
        at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:606)
        at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:64)
        at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:79)
        at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65)
        ... 3 common frames omitted

于是开始找解决的办法，也有同样的网友给出的答案是：将下面两个类的自动配置排除在Spring 容器外：

org.springframework.boot.autoconfigure.mongo.MongoAutoConfiguration

org.springframework.boot.autoconfigure.data.mongo.MongoDataAutoConfiguration

// 具体写法：在SpringBoot 启动类上加上注解 @SpringBootApplication
@SpringBootApplication(exclude = { MongoAutoConfiguration.class,MongoDataAutoConfiguration.class })

解决方式——暂且一试

在启动类上加上注解：@SpringBootApplication(exclude = { MongoAutoConfiguration.class,MongoDataAutoConfiguration.class }) 此时是没有启用MongoDB的情况，启动后也没有报错信息了，故事是按想象的发展。
打开MongoDB的配置，将 application.properties 中的属性：file.storage.mode 改为：MONGODB ，重新启动后，在启动过程中报错：找不到Bean：GridFsTemplate 。具体信息如下：

dependency expressed through field 'fileService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException:
  Error creating bean with name 'gridFSService': Unsatisfied dependency expressed through field 'gridFsTemplate';
   nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: 
   No qualifying bean of type 'org.springframework.data.mongodb.gridfs.GridFsTemplate' available: expected at least 1 bean which qualifies as autowire candidate.
    Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}

这个类是用来存储文件或者查询文件的，总之是不可或缺的。猜测应该是去掉了SpringBoot 与MongoDB 整合的配置类导致的原因（因为如果不排除自动配置类，系统就可以正常启动）。

尝试中

总结上面的问题和现象，如果用MongoDB 少不了配置类，不用MongoDB 则不需要配置类，但是如何让 @SpringBootApplication 能够动态根据自己的配置进行是否排除自动配置类呢？嗯，这东西是SpringBoot的东西，改它是不可能的了。此种方式行不通。换种方式：既然提示找不到Bean：GridFsTemplate ，那是不是可以注入一个？然后打开注解类：

@Configuration(proxyBeanMethods = false)
@ConditionalOnClass(MongoClient.class)
@EnableConfigurationProperties(MongoProperties.class)
@ConditionalOnMissingBean(type = "org.springframework.data.mongodb.MongoDbFactory")
public class MongoAutoConfiguration {

	@Bean
	@ConditionalOnMissingBean(type = { "com.mongodb.MongoClient", "com.mongodb.client.MongoClient" })
	public MongoClient mongo(MongoProperties properties, ObjectProvider<MongoClientOptions> options,
			Environment environment) {
		return new MongoClientFactory(properties, environment).createMongoClient(options.getIfAvailable());
	}

}

这个类和我自己的配置类差别不大，就是配置了一个 MongoClient ，再看下一个：

@Configuration(proxyBeanMethods = false)
@ConditionalOnClass({ MongoClient.class, com.mongodb.client.MongoClient.class, MongoTemplate.class })
@EnableConfigurationProperties(MongoProperties.class)
@Import({ MongoDataConfiguration.class, MongoDbFactoryConfiguration.class, MongoDbFactoryDependentConfiguration.class })
@AutoConfigureAfter(MongoAutoConfiguration.class)
public class MongoDataAutoConfiguration {

}

这个类导入了三个配置类：MongoDataConfiguration，MongoDbFactoryConfiguration，MongoDbFactoryDependentConfiguration.
在最后一个类中，找到了我们需要的GridFsTemplate :

@Bean
@ConditionalOnMissingBean(GridFsOperations.class)
GridFsTemplate gridFsTemplate(MongoDbFactory mongoDbFactory, MongoTemplate mongoTemplate) {
	return new GridFsTemplate(new GridFsMongoDbFactory(mongoDbFactory, this.properties),
			mongoTemplate.getConverter());
}

尝试了一番，在这里获取不到 MongoDbFactory 类。好吧，找同事讨论一下。

柳暗花明又一村

得同事启发，在spring-boot-autoconfigure 中的 spring.factories 文件中找到：

# Auto Configure
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
org.springframework.boot.autoconfigure.data.mongo.MongoDataAutoConfiguration,\
org.springframework.boot.autoconfigure.data.mongo.MongoReactiveDataAutoConfiguration,\
org.springframework.boot.autoconfigure.data.mongo.MongoReactiveRepositoriesAutoConfiguration,\
org.springframework.boot.autoconfigure.data.mongo.MongoRepositoriesAutoConfiguration

我们要找的就是 MongoDataAutoConfiguration 类了。查看源码：

package org.springframework.boot.autoconfigure.mongo;

import com.mongodb.MongoClient;
import com.mongodb.MongoClientOptions;

import org.springframework.beans.factory.ObjectProvider;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.env.Environment;

@Configuration(proxyBeanMethods = false)
@ConditionalOnClass(MongoClient.class)
@EnableConfigurationProperties(MongoProperties.class)
@ConditionalOnMissingBean(type = "org.springframework.data.mongodb.MongoDbFactory")
public class MongoAutoConfiguration {

	@Bean
	@ConditionalOnMissingBean(type = { "com.mongodb.MongoClient", "com.mongodb.client.MongoClient" })
	public MongoClient mongo(MongoProperties properties, ObjectProvider<MongoClientOptions> options,
			Environment environment) {
		return new MongoClientFactory(properties, environment).createMongoClient(options.getIfAvailable());
	}

}

其它都没有问题，就是类上的注解 @ConditionalOnClass(MongoClient.class) 条件不符合这里的场景需求。
那既然SpringBoot 也可以这样写，不妨在我们自己的项目中新建一个重名的类： MongoAutoConfiguration 。类名和包名都和原有的保持一致，只是将其存放在自己的项目中。于是将类改造为如下：

package org.springframework.boot.autoconfigure.mongo;

import com.mongodb.MongoClient;
import com.mongodb.MongoClientOptions;
import com.mongodb.client.MongoDatabase;
import com.mongodb.client.gridfs.GridFSBucket;
import com.mongodb.client.gridfs.GridFSBuckets;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * 覆盖SpringBoot 自动配置类
 */
@Configuration
@ConditionalOnExpression(value = "'${spring.data.mongodb.host}'!=''")
@EnableConfigurationProperties(MongoProperties.class)
@ConditionalOnMissingBean(type = "org.springframework.data.mongodb.MongoDbFactory")
public class MongoAutoConfiguration {

    @Autowired
    private MongoProperties properties;

    @Bean
    @ConditionalOnMissingBean(type = {"com.mongodb.MongoClient", "com.mongodb.client.MongoClient"})
    public MongoClient mongoClient() {
        MongoClientOptions.Builder builder = MongoClientOptions.builder().connectTimeout(8000).serverSelectionTimeout(8000);
        return new MongoClient(properties.getHost(), builder.build());
    }

    @Bean
    public GridFSBucket gridFSBucket(MongoClient mongoClient) {
        MongoDatabase database = mongoClient.getDatabase(properties.getDatabase());
        GridFSBucket bucket = GridFSBuckets.create(database);
        return bucket;
    }

}

这样在系统自动配置MongoDB 时，如果没有配置MongoDB的host 就不注入MongoDB 相关的类，这也比较符合实际的使用。