springboot 集成log4j springboot 集成minio集群

转载

langrisser 2024-08-28 18:09:27

文章标签 springboot 集成log4j java linux docker 分布式 文章分类 架构后端开发

Minio7.x的部署(docker)+springboot2.x整合Minio7.x

一.为什么选择Minio

简介：分布式文件存储常见解决方案介绍

目前业界比较多这个解决方案，这边就挑选几个介绍下

MinIO

是在 Apache License v2.0 下发布的对象存储服务器,学习成本低，安装运维简单，主流语言的客户端整合都有, 号称最强的对象存储文件服务器，且可以和容器化技术docker/k8s等结合，社区活跃但不够成熟，业界参考资料较少

官网：https://docs.min.io/cn/

FastDFS

一个开源的轻量级分布式文件系统,比较少的客户端可以整合，目前主要是C和java客户端，在一些互联网创业公司中有应用比较多，没有官方文档,社区不怎么活跃.
架构+部署结构复杂，出问题定位比较难定位，可以说是fastdfs零件的组装过程，需要去理解fastDFS的架构设计，才能够正确的安装部署

云厂商

阿里云OSS
七牛云
腾讯云
亚马逊云
CDN最强：Akamai https://www.akamai.com/cn

选云厂商理由

优点：开发简单，功能强大，容易维护（不同网络下图片质量、水印、加密策略、扩容、加速）
缺点：要钱, 个性化处理，未来转移比较复杂，不排除有些厂商会提供一键迁移工具

选开源MinIO的理由

优点：功能强大、可以根据业务做二次的定制，新一代分布式文件存储系统，容器化结合强大，更重要的是免费（购买磁盘、内存、带宽）
缺点：自己需要有专门的团队进行维护、扩容等

注：更多理论知识需自行补充

二.Minio在Centos7.x中骨架搭建

所需配置不高：
CPU&内存1 核1GiB
操作系统 CentOS 7.8 64位
实例规格 ecs.t5-lc1m1.small(性能约束实例)
实例规格族 ecs.t5
云盘1
当前使用带宽 1Mbps

1.安装docker

#依次运行以下命令添加yum源
 yum update
 yum install epel-release -y
 yum clean all
 yum list#安装并运行Docker。
 yum install docker-io -y
 systemctl start docker#检查安装结果。
 docker info#启动使用Docker
 systemctl start docker #运行Docker守护进程
 systemctl stop docker #停止Docker守护进程
 systemctl restart docker #重启Docker守护进程#修改镜像仓库
 vim /etc/docker/daemon.json
 #改为下面内容，然后重启docker
 {
 “debug”:true,“experimental”:true,
 “registry-mirrors”:[“https://xxx.mirror.aliyuncs.com”,“https://hub-mirror.c.163.com”,“https://docker.mirrors.ustc.edu.cn”]
 }#查看信息
 docker info

2.部署minio单节点（docker部署）

docker run -p 9000:9000 
 –name minio_x 
 -v /Users/xx/Desktop/test:/data 
 -e “MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE” 
 -e “MINIO_ROOT_PASSWORD=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY” 
 minio/minio server /data

3.安装docker-compose

安装docker-compose

yum install -y python-pip
 curl -L https://github.com/docker/compose/releases/download/1.25.1/run.sh > /usr/local/bin/docker-compose
 chmod +x /usr/local/bin/docker-compose
 docker-compose --version

4.伪分布式部署minio集群（docker-compose部署）

vim docker-compose.yml

# starts 4 docker containers running minio server instances. Each

# minio server's web interface will be accessible on the host at port

# 9001 through 9004.

services:

  minio1:

    image: minio/minio

    volumes:

      - data1-1:/data1

      - data1-2:/data2

    ports:

      - "9001:9000"

    environment:

      MINIO_ACCESS_KEY: minioadmin

      MINIO_SECRET_KEY: minioadmin

    command: server http://minio{1...4}/data{1...2}

    healthcheck:

      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]

      interval: 30s

      timeout: 20s

      retries: 3

 

  minio2:

    image: minio/minio

    volumes:

      - data2-1:/data1

      - data2-2:/data2

    ports:

      - "9002:9000"

    environment:

      MINIO_ACCESS_KEY: minioadmin

      MINIO_SECRET_KEY: minioadmin

    command: server http://minio{1...4}/data{1...2}

    healthcheck:

      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]

      interval: 30s

      timeout: 20s

      retries: 3

 

  minio3:

    image: minio/minio

    volumes:

      - data3-1:/data1

      - data3-2:/data2

    ports:

      - "9003:9000"

    environment:

      MINIO_ACCESS_KEY: minioadmin

      MINIO_SECRET_KEY: minioadmin

    command: server http://minio{1...4}/data{1...2}

    healthcheck:

      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]

      interval: 30s

      timeout: 20s

      retries: 3

 

  minio4:

    image: minio/minio

    volumes:

      - data4-1:/data1

      - data4-2:/data2

    ports:

      - "9004:9000"

    environment:

      MINIO_ACCESS_KEY: minioadmin

      MINIO_SECRET_KEY: minioadmin

    command: server http://minio{1...4}/data{1...2}

    healthcheck:

      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]

      interval: 30s

      timeout: 20s

      retries: 3

 

## By default this config uses default local driver,

## For custom volumes replace with volume driver configuration.

volumes:

  data1-1:

  data1-2:

  data2-1:

  data2-2:

  data3-1:

  data3-2:

  data4-1:

  data4-2:

docker-compose up

5.多服务器部署minio集群（docker-compose部署）

说明：此项目只用了4个节点8个分区，根据需要可以自行扩展使用

修改hosts文件4个节点均需要修改 vi /etc/hosts

172.18.29.152 minio-node-1
 172.18.29.150 minio-node-2
 172.18.29.153 minio-node-3
 172.18.29.151 minio-node-4

vim docker-compose.yml
节点4：

version: '3.7'



# starts 4 docker containers running minio server instances. Each

# minio server's web interface will be accessible on the host at port

# 9001 through 9004.

services:

  minio-node:

    image: minio/minio

    network_mode: "host"

    volumes:

      - /root/data4-1:/data1   

      - /root/data4-2:/data2   

#    ports:

#      - "9001:9000"

    environment:

      MINIO_ACCESS_KEY: minioadmin

      MINIO_SECRET_KEY: minioadmin

    command: server http://minio-node-{1...4}/data{1...2}

    healthcheck:

      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]

      interval: 30s

      timeout: 20s

      retries: 3

节点3：

version: '3.7'



# starts 4 docker containers running minio server instances. Each

# minio server's web interface will be accessible on the host at port

# 9001 through 9004.

services:

  minio-node:

    image: minio/minio

    network_mode: "host"

    volumes:

      - /root/data3-1:/data1

      - /root/data3-2:/data2

#    ports:

#      - "9001:9000"

    environment:

      MINIO_ACCESS_KEY: minioadmin

      MINIO_SECRET_KEY: minioadmin

    command: server http://minio-node-{1...4}/data{1...2}

    healthcheck:

      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]

      interval: 30s

      timeout: 20s

      retries: 3

节点2：

version: '3.7'



# starts 4 docker containers running minio server instances. Each

# minio server's web interface will be accessible on the host at port

# 9001 through 9004.

services:

  minio-node:

    image: minio/minio

    network_mode: "host"

    volumes:

      - /root/data2-1:/data1

      - /root/data2-2:/data2

#    ports:

#      - "9001:9000"

    environment:

      MINIO_ACCESS_KEY: minioadmin

      MINIO_SECRET_KEY: minioadmin

    command: server http://minio-node-{1...4}/data{1...2}

    healthcheck:

      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]

      interval: 30s

      timeout: 20s

      retries: 3

节点1：

version: '3.7'



# starts 4 docker containers running minio server instances. Each

# minio server's web interface will be accessible on the host at port

# 9001 through 9004.

services:

  minio-node:

    image: minio/minio

    network_mode: "host"

    volumes:

      - /root/data1-1:/data1

      - /root/data1-2:/data2

#    ports:

#      - "9001:9000"

    environment:

      MINIO_ACCESS_KEY: minioadmin

      MINIO_SECRET_KEY: minioadmin

    command: server http://minio-node-{1...4}/data{1...2}

    healthcheck:

      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]

      interval: 30s

      timeout: 20s

      retries: 3

也可用过scp命令进行快速传送配置文件

scp docker-compose.yml root@ip:/root

注：第一次启动集群需要所有存储路径为空

通过docker-compose up启动4个节点

出现面内容说明启动成功

[root@iZ2ze8tuatzv94fm6fgl1mZ ~]# docker-compose up
WARNING: Found orphan containers (root_minio-node-4_1, root_minio3_1, root_minio4_1, root_minio1_1, root_minio2_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Recreating root_minio-node_1 ... done
Attaching to root_minio-node_1
minio-node_1  | Waiting for all MinIO sub-systems to be initialized.. lock acquired
minio-node_1  | All MinIO sub-systems initialized successfully
minio-node_1  | Status:         8 Online, 0 Offline. 
minio-node_1  | Endpoint: http://172.18.29.151:9000  http://172.17.0.1:9000  http://172.19.0.1:9000  http://127.0.0.1:9000           
minio-node_1  | 
minio-node_1  | Browser Access:
minio-node_1  |    http://172.18.29.151:9000  http://172.17.0.1:9000  http://172.19.0.1:9000  http://127.0.0.1:9000          
minio-node_1  | 
minio-node_1  | Object API (Amazon S3 compatible):
minio-node_1  |    Go:         https://docs.min.io/docs/golang-client-quickstart-guide
minio-node_1  |    Java:       https://docs.min.io/docs/java-client-quickstart-guide
minio-node_1  |    Python:     https://docs.min.io/docs/python-client-quickstart-guide
minio-node_1  |    JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
minio-node_1  |    .NET:       https://docs.min.io/docs/dotnet-client-quickstart-guide
minio-node_1  | Detected default credentials 'minioadmin:minioadmin', please change the credentials immediately using 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD'
minio-node_1  | Waiting for all MinIO IAM sub-system to be initialized.. lock acquired

三.nginx实现负载均衡

四.springboot整合Minio

1.添加Maven依赖

<properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
    </properties>

    <dependencies>
        <dependency>
            <groupId>xxx</groupId>
            <artifactId>xx-common</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>io.minio</groupId>
            <artifactId>minio</artifactId>
            <version>7.1.0</version>
        </dependency>
    </dependencies>

</project>

2.yml配置：（.properties类似）

#配置文件上传限制策略
  servlet:
    multipart:
      max-file-size: 200MB
      max-request-size: 200MB

#MinIO配置
minio:
  endpoint: http://8.131.119.1:xxxx
  access-key-id: AKIAIOSFODNN7EXAMPLE
  access-key-secret: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
  bucketname: aaa

3.配置类：

package net.xx.config;

import io.minio.MinioClient;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Data
@Configuration
@ConfigurationProperties(prefix = "minio")
public class MinioConfig {

    private String endpoint;

    private String accessKeyId;

    private String accessKeySecret;

    private String bucketname;

    @Bean
    public MinioClient minioClient(){
        //创建minio对象
        MinioClient minioClient = MinioClient.builder()
                .endpoint(endpoint)
                .credentials(accessKeyId,accessKeySecret).build();
        return minioClient;
    }
}

4上传文件：（根据需求自行修改）

@Component
@Slf4j
public class MinIOComponent {

    @Autowired
    private MinioConfig minioConfig;

    @Autowired
    private MinioClient minioClient;

    /**
     * 文件上传
     * @param file
     * @return
     */
    public String uploadFile(MultipartFile file){
        //获取相关配置
        String bucketname = minioConfig.getBucketname();
        String endpoint = minioConfig.getEndpoint();
        String accessKeyId = minioConfig.getAccessKeyId();
        String accesskeySecret = minioConfig.getAccessKeySecret();

        //获取原生文件名 xxx.jpg
        String originalFilename = file.getOriginalFilename();

        //JDK8的日期格式化
        LocalDateTime ldt = LocalDateTime.now();
        DateTimeFormatter dtf = DateTimeFormatter.ofPattern("yyyy/MM/dd");

        //拼装路径，上存储的路径 2021/2/14/xxxx.jpg
        //2021/2/5/
        String folder = dtf.format(ldt);
        //xxxx
        String fileName = CommonUtil.generateUUID();
        //.jpg
        String extension = originalFilename.substring(originalFilename.lastIndexOf("."));
        //新文件名
        String newFileName = "user/"+folder+"/"+fileName+extension;

        try (InputStream inputStream = file.getInputStream()){
            ObjectWriteResponse putObject = minioClient.putObject(PutObjectArgs.builder()
                    .bucket(bucketname)
                    .object(newFileName)
                    .stream(inputStream, file.getSize(), -1)
                    .contentType(file.getContentType())
                    .build());
            if(putObject!=null){
                //String imgUrl = minioClient.getObjectUrl(bucketname,newFileName);
                String imgUrl = minioClient.getPresignedObjectUrl(GetPresignedObjectUrlArgs.builder().method(Method.GET).bucket(bucketname).object(newFileName).build());
                return imgUrl;
            }
        } catch (Exception e) {
            log.error("文件上传失败:{}",e);
        }
        return null;
    }

5.删除文件

/**
     * 删除头像
     * @param attachmentNameInServer
     * @return
     */
    @Override
    public boolean delUserImg(String attachmentNameInServer) {
//        String[] strArr = url.substring(0,url.indexOf('?')).split("/");
//        String fileName = strArr[strArr.length-1];
        try {
            minioClient.removeObject(RemoveObjectArgs.builder().bucket(minioConfig.getBucketname()).object(attachmentNameInServer).build());
            return true;
        }catch (Exception e){
            log.warn("fileName 文件删除异常:{}",e);
            return false;
        }
    }

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。