环境如下

//前三个为K8s集群中的节点

主机名 IP 运行服务
docker-k8s01 192.168.171.151 docker+registry
docker-k8s02 192.168.171.150 docker
docker-k8s03 192.168.171.152 docker
Jenkins 192.168.171.153 Jenkins+gitlab+docker

Jenkins采用war包的方式部署,需要用到tomcat环境,自行参考博文,进行部署;

Docker 部署

Tomcat 的安装与优化

K8s 安装部署

一、部署registry私有仓库

//任意节点都可部署**

1、运行registry容器

[root@docker-k8s01 ~]# docker run -itd --name registry --restart=always -p 5000:5000 -v /data/registry:/var/lib/registry registry 

2、配置各个节点指向私有仓库

//选择其中一个节点操作
[root@docker-k8s01 ~]# vim /usr/lib/systemd/system/docker.service 
//修该下面的配置项,以便指定其私有仓库的监听地址
ExecStart=/usr/bin/dockerd -H unix:// --insecure-registry 192.168.171.151:5000
//将已经配置好的文件发送到各节点
[root@docker-k8s01 ~]# scp /usr/lib/systemd/system/docker.service docker-k8s02:/usr/lib/systemd/system/
[root@docker-k8s01 ~]# scp /usr/lib/systemd/system/docker.service docker-k8s03:/usr/lib/systemd/system/ 
[root@docker-k8s01 ~]# scp /usr/lib/systemd/system/docker.service jenkins:/usr/lib/systemd/system/

3、各节点重启docker服务,使之生效

//每个节点都需要执行
[root@docker-k8s01 ~]# systemctl daemon-reload 
[root@docker-k8s01 ~]# systemctl restart docker 

二、部署Jenkins服务

//部署Jenkins之前,自行部署Tomcat服务,参考开头链接即可

//进入tomcat的webapps目录,根据实际情况而定,这里我直接删除了
[root@jenkins ~]# cd /usr/local/tomcat/webapps/
[root@jenkins webapps]# rm -rf *
[root@jenkins webapps]# ls      //刚刚上传上来发现tomcat自动给我解压了
jenkins  jenkins.war
//接下来指定Jenkins的家目录
[root@jenkins webapps]# vim ../bin/catalina.sh 

#!/bin/sh
//在开头添加如下内容
export CATALINA_OPTS="-DJENKINS_HOME=/data/jenkins"
export JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Dhudson.ClassicPluginStrategy.noBytecodeTransformer=true"
[root@jenkins webapps]# cd ../bin/
[root@jenkins bin]# ./catalina.sh start   //启动tomcat
//确定端口已启动
[root@jenkins bin]# ss -anput | grep 8080
tcp    LISTEN     0      100    [::]:8080               [::]:*                   users:(("java",pid=12236,fd=49))

三、配置Jenkins服务

要想实现Jenkins的CI/CD,必须有GitLab、GitLab HOOK、Git Lab Authentication这三个插件,国内不太好下载这几个插件。

1、配置Jenkins web界面

//浏览器访问tomcat服务的IP+端口/Jenkins(访问前最好先断掉外网,否则在输入密码后,会让在线安装插件,比较慢,而且很有可能安装失败)

Jinkens+gitlab针对k8s集群实现CI/CD
image-20200917162719969

//查看密码
[root@jenkins ~]# cat 
/data/jenkins/secrets/initialAdminPassword 
3eb89452e2614173a548a9d4cfe9fcc3

//输入密码后稍等一会出现如下图
Jinkens+gitlab针对k8s集群实现CI/CD
//创建相应用户
Jinkens+gitlab针对k8s集群实现CI/CD
Jinkens+gitlab针对k8s集群实现CI/CD
Jinkens+gitlab针对k8s集群实现CI/CD
登录成功后,如下**
Jinkens+gitlab针对k8s集群实现CI/CD

2、配置所需插件

//关闭浏览器页面,对Jenkins添加插件

[root@jenkins ~]# rm -rf /data/jenkins/plugins/
[root@jenkins ~]# tar zxf plugins.tar.gz -C /data/jenkins/
//重启tomcat使之生效
[root@jenkins ~]# /usr/local/tomcat/bin/catalina.sh stop 
[root@jenkins ~]# /usr/local/tomcat/bin/catalina.sh start 

//再次访问Jenkins的web界面

Jinkens+gitlab针对k8s集群实现CI/CD
//可以看到已经变成了中文页面,说明插件配置生效

Jinkens+gitlab针对k8s集群实现CI/CD

四、部署gitlab

1、安装gitlab

//下载gitlab的rpm包
[root@jenkins ~]# wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-11.9.8-ce.0.el7.x86_64.rpm
[root@jenkins ~]# rpm -ivh gitlab-ce-11.9.8-ce.0.el7.x86_64.rpm 
[root@jenkins ~]# vim /etc/gitlab/gitlab.rb 
external_url 'http://192.168.171.153:90'      //将原本的域名更改为本机IP
unicorn['listen'] = '192.168.171.153'   //这两行在下方是有注释的,这里我直接添加了
unicorn['port'] = 3000
[root@jenkins ~]# gitlab-ctl reconfigure 

接下来的操作可参考文章开头链接中的Gitlab安装部署来接着部署,最后保证可以将远端的库克隆到本地即可
**

//可以看到本地已经访问到gitlab
[root@jenkins ~]# ls -d test1/
test1/

五、配置Jenkins创建一个任务

Jinkens+gitlab针对k8s集群实现CI/CD
Jinkens+gitlab针对k8s集群实现CI/CD
Jinkens+gitlab针对k8s集群实现CI/CD
Jinkens+gitlab针对k8s集群实现CI/CD

//脚本中的IP,192.168.171.151为私有仓库的IP,最后的192.168.171.151为k8s集群中master的IP地址
#!/bin/bash
backupcode="/data/backcode/$JOB_NAME/$BUILD_NUMBER"   #这里引用了Jenkins的默认变量
mkdir -p $backupcode
chmod 644 "$JENKINS_HOME"/workspace/"$JOB_NAME"/*
rsync -acP   "$JENKINS_HOME"/workspace/"$JOB_NAME"/*  $backupcode
echo From  192.168.171.151:5000/nginx > "$JENKINS_HOME"/workspace/Dockerfile
echo COPY ./"$JOB_NAME"/* /usr/share/nginx/html/ >> "$JENKINS_HOME"/workspace/Dockerfile
docker rmi 192.168.171.151:5000/nginx        
docker build -t 192.168.171.151:5000/nginx /"$JENKINS_HOME"/workspace/.
docker push 192.168.171.151:5000/nginx
ssh root@192.168.171.151 kubectl delete deployment nginx
ssh root@192.168.171.151 kubectl apply -f /root/nginx.yaml

Jinkens+gitlab针对k8s集群实现CI/CD
//脚本填写后,先别保存,根据以下操作复制一下Jenkins的地址

Jinkens+gitlab针对k8s集群实现CI/CD

六、开启Jenkins的匿名访问权限

Jinkens+gitlab针对k8s集群实现CI/CD
Jinkens+gitlab针对k8s集群实现CI/CD
Jinkens+gitlab针对k8s集群实现CI/CD
Jinkens+gitlab针对k8s集群实现CI/CD
Jinkens+gitlab针对k8s集群实现CI/CD
Jinkens+gitlab针对k8s集群实现CI/CD

七、gitlab上开启允许向自己发送web hook

//这是因为我是把Jenkins和gitlab装在了一台服务器上

Jinkens+gitlab针对k8s集群实现CI/CD
Jinkens+gitlab针对k8s集群实现CI/CD
Jinkens+gitlab针对k8s集群实现CI/CD
Jinkens+gitlab针对k8s集群实现CI/CD
Jinkens+gitlab针对k8s集群实现CI/CD
//保存后,下拉页面,即可看到新添加的web-hook,点击如下,进行测试

Jinkens+gitlab针对k8s集群实现CI/CD
//返回状态码200,则表示配置无误
Jinkens+gitlab针对k8s集群实现CI/CD

八、配置Jenkins免密登录k8s群集的master节点

//Jenkins配置如下

//下方IP为k8s群集master的IP
[root@jenkins ~]# ssh-copy-id root@192.168.171.151

九、测试持续CI/CD效果

1、k8s群集运行nginx资源对象

Jinkens+gitlab针对k8s集群实现CI/CD

2、客户端访问nginx

Jinkens+gitlab针对k8s集群实现CI/CD

3、在gitlab进行版本的更新迭代测试

[root@jenkins ~]# cd test1/
[root@jenkins test1]# git config --global user.name "test"
[root@jenkins test1]# git config --global user.email "test@test.com"
[root@jenkins test1]# echo "test--version==v1" > index.html
[root@jenkins test1]# git add *
[root@jenkins test1]# git commit -m "Test CI/CD"
[root@jenkins test1]# git push origin master 

//在进行上述操作后,即可在Jenkins新建的任务中,看到构建成功的信息
Jinkens+gitlab针对k8s集群实现CI/CD
//再次访问nginx首页,发现已经变成了我们在gitlab上提交的内容

Jinkens+gitlab针对k8s集群实现CI/CD
//历史版本内容存放路径如下

[root@jenkins data]# pwd
/data
//这个目录下记录的是历史版本的代码
[root@jenkins data]# tree backcode/
backcode/
└── test-01
    ├── 1
    │   └── README.md
    └── 2
        ├── index.html
        └── README.md

[root@jenkins workspace]# pwd
/data/jenkins/workspace
//此目录下是将新代码构建成镜像的关键所在,Dockersfile就在这个目录下
[root@jenkins workspace]# ls
Dockerfile  test-01