需要环境
- Git(GitLab)
- Harbor 私服
- kubernetes-plugin 使用
- Kubernetes
- jenkins
通过前面三篇我们已经了解了jenkins和 kubernetes-plugin的使用。还没过的小伙伴可以看下
- 基于K8S构建企业级Jenkins CI/CD平台实战(一) 之 环境搭建
- 基于K8S构建企业级Jenkins CI/CD平台实战(二) 之 kubernetes-plugin 插件使用
- Jenkins Pipeline构建流水线发布
接下来为大家带来Java 项目的基于K8S构建企业级Jenkins CI/CD过程
编写Pipeline脚本完成CI阶段
我们使用第二节中配置好的 podTemplate
模板jnlp-maven
。
再jnlp
容器中执行:
- 使用 git 拉取代码。使用码云上的springboot demo :
https://gitee.com/andanyoung/springboot-hello
,public 仓库,git_auth 就不需要了 - 使用maven 命令打包成 jar
- 使用docker 命令打包成镜像,并推送至私服
//项目git 地址
def git_address = "https://gitee.com/andanyoung/springboot-hello.git"
//项目git 分支地址
def git_branch = "master"
//git 证书ID
def git_auth = ""
//打包镜像名称
def docker_image_name = "andanyoung/springboot-hello:v1.0"
//docker registry 地址
def docker_registry = ""
def docker_registry_auth = "b0f6a24a-f587-4b91-bc1f-9de8b07c762f"
def docker_host = "tcp://192.168.0.192:2375"
podTemplate (inheritFrom: "jnlp-maven"){
node(POD_LABEL){
// 第一步
stage('拉取代码'){
git branch: "${git_branch}", credentialsId: "${git_auth}", url: "${git_address}"
}
// 第二步
stage('代码编译'){
sh "mvn clean package -Dmaven.test.skip=true"
}
// 第三步
stage('构建镜像'){
//Harbor镜像仓库登录验证,
withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
sh """
echo '
FROM openjdk:8-jdk-alpine
ADD target/app.jar app.jar
ENTRYPOINT [ "sh", "-c", "java -jar app.jar" ]
' > Dockerfile
export DOCKER_HOST="${docker_host}"
docker build -t ${docker_image_name} .
docker login -u ${username} -p '${password}' ${docker_registry}
docker push ${docker_image_name}
"""
}
}
}
}
pipeline script变量docker_registry_auth、git_auth、k8s_auth通过保存在jenkins凭据中相应的凭据ID。
各个阶段注释都是很清楚的。需要说明一点的是 export DOCKER_HOST="${docker_host}"
配置docker 守护进程(Daemon socket)host
如果你不写这个默认使用unix:///var/run/docker.sock
连接,就会报connect: permission denied
权限不足错误,因为宿主机docker使用root起的,而docker运行jenkins是使用 jenkins(id 1000) 用户的。
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/auth": dial unix /var/run/docker.sock: connect: permission denied
解决方案一:是在宿主机上执行chmod 777 /var/run/docker.sock
。但这个太low,宿主机一重启权限有不对了。有需要重新执行。所以还是使用方案二吧
解决方案二:docker -H tcp://0.0.0.0:2375 ps
或者export DOCKER_HOST="tcp://0.0.0.0:2375" && docker ps
。指定dockerd host。前提是dockerd启动需要开启tcp 连接:
sudo dockerd -H unix:///var/run/docker.sock -H tcp://192.168.59.106 -H tcp://10.10.10.2
或者修改vi /usr/lib/systemd/system/docker.service
:
配置docker私服 docker_registry_auth
账号密码
PS: git_auth 同理
我这边为了演示直接使用的docker官方仓库,生产上当然是需要使用docker私服。
使用docker官方仓库(类似于github) https://hub.docker.com/,先去注册账号密码。
- 点击保存生成凭据 复制配置id到我们的pipeline
点击执行,构建成功
查看docker私仓,push成功
运行再k8s 中部署
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-k8s
spec:
replicas: 3
selector:
matchLabels:
app: hello-k8s
template:
metadata:
labels:
app: hello-k8s
spec:
affinity:
# 反亲和性调度 使各个pod不在同个node里
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- hello-k8s
containers:
- name: pod-hello-k8s
image: andanyoung/springboot-hello:v1.0
imagePullPolicy: Always
resources:
limits:
memory: 2G
requests:
memory: 2G
如果是私服记得加上imagePullSecrets
,image
改为 私服地址host:ip/andanyoung/springboot-hello:v1.0
Jenkins在K8s中持续部署 CD 阶段
新增Cluster Role 集群权限
在第二节中已经添加了用户jenkins
,在这个里我们需要给他就是Deployment的一些部署的(Cluster Role)权限。
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cluster-jenkins-dev-ops
rules:
- verbs:
- create
- delete
- get
- list
- patch
- update
- watch
apiGroups:
- ''
resources:
- pods
- verbs:
- create
- delete
- get
- list
- patch
- update
- watch
apiGroups:
- ''
resources:
- pods/exec
- verbs:
- get
- list
- watch
apiGroups:
- ''
resources:
- pods/log
- verbs:
- watch
apiGroups:
- ''
resources:
- events
- verbs:
- get
apiGroups:
- ''
resources:
- secrets
- verbs:
- create
- delete
- deletecollection
- patch
- update
apiGroups:
- apps
resources:
- daemonsets
- deployments
- deployments/rollback
- deployments/scale
- replicasets
- replicasets/scale
- statefulsets
- statefulsets/scale
- verbs:
- get
- list
- watch
apiGroups:
- apps
resources:
- controllerrevisions
- daemonsets
- daemonsets/status
- deployments
- deployments/scale
- deployments/status
- replicasets
- replicasets/scale
- replicasets/status
- statefulsets
- statefulsets/scale
- statefulsets/status
Cluster Role Binding 绑定权限
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cluster-jenkins-dev-ops-binding
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-jenkins-dev-ops
CD 阶段
部署无非是执行重启,或者是更新版本,先来看看命令吧。
- 重启
kubectl rollout restart deployment hello-k8s
- 更新版本
kubectl set image deployments/hello-k8s pod-hello-k8s=andanyoung/springboot-hello:V2.0
其中deployments/hello-k8s 表示部署名称,有上面的命令得到 pod-hello-k8s=andanyoung/springboot-hello:V2.0 表示pod部署更新到哪个容器
- 其他命令
还可以执行其他kuctl命令,但要注意namespace
重启命令pipline:
//项目git 地址
def git_address = "https://gitee.com/andanyoung/springboot-hello.git"
//项目git 分支地址
def git_branch = "master"
//git 证书ID
def git_auth = ""
//打包镜像名称
def docker_image_name = "andanyoung/springboot-hello:v1.0"
//docker registry 地址
def docker_registry = ""
def docker_registry_auth = "b0f6a24a-f587-4b91-bc1f-9de8b07c762f"
def docker_host = "tcp://192.168.0.192:2375"
podTemplate (inheritFrom: "jnlp-maven"){
node(POD_LABEL){
// 第一步
stage('拉取代码'){
git branch: "${git_branch}", credentialsId: "${git_auth}", url: "${git_address}"
}
// 第二步
stage('代码编译'){
sh "mvn clean package -Dmaven.test.skip=true"
}
// 第三步
stage('构建镜像'){
//Harbor镜像仓库登录验证,
withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
sh """
echo '
FROM openjdk:8-jdk-alpine
ADD target/app.jar app.jar
ENTRYPOINT [ "sh", "-c", "java -jar app.jar" ]
' > Dockerfile
export DOCKER_HOST="${docker_host}"
docker build -t ${docker_image_name} .
docker login -u ${username} -p '${password}' ${docker_registry}
docker push ${docker_image_name}
"""
}
}
// 第四步
container('jnlp-kubectl'){
stage('部署到K8S平台'){
sh 'kubectl config view'
sh 'kubectl rollout restart deployment hello-k8s'
}
}
}
}
运行完成,查看重启结果
更新版本命令pipline:
//项目git 地址
def git_address = "https://gitee.com/andanyoung/springboot-hello.git"
//项目git 分支地址
def git_branch = "master"
//git 证书ID
def git_auth = ""
//打包镜像名称
def docker_image_name = "andanyoung/springboot-hello"
//打包镜像tag
def docker_image_tag = "v2.0"
//docker registry 地址
def docker_registry = ""
def docker_registry_auth = "b0f6a24a-f587-4b91-bc1f-9de8b07c762f"
def docker_host = "tcp://192.168.0.192:2375"
podTemplate (inheritFrom: "jnlp-maven"){
node(POD_LABEL){
// 第一步
stage('拉取代码'){
git branch: "${git_branch}", credentialsId: "${git_auth}", url: "${git_address}"
}
// 第二步
stage('代码编译'){
sh "mvn clean package -Dmaven.test.skip=true"
}
// 第三步
stage('构建镜像'){
//Harbor镜像仓库登录验证,
withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
sh """
echo '
FROM openjdk:8-jdk-alpine
ADD target/app.jar app.jar
ENTRYPOINT [ "sh", "-c", "java -jar app.jar" ]
' > Dockerfile
export DOCKER_HOST="${docker_host}"
docker build -t ${docker_image_name}:${docker_image_tag} .
docker login -u ${username} -p '${password}' ${docker_registry}
docker push ${docker_image_name}:${docker_image_tag}
"""
}
}
// 第四步
container('jnlp-kubectl'){
stage('部署到K8S平台'){
sh 'kubectl config view'
sh "kubectl set image deployments/hello-k8s pod-hello-k8s=andanyoung/springboot-hello:${docker_image_tag}"
}
}
}
}
运行完成,查看重启结果
k8s部署Spring cloud
上面展示了K8s如何部署 Spring boot 项目,Spring Cloud 其实可以看出基于Spring boot 的按业务功能划分为多个 Spring boot 项目而已。所有部署Spring Cloud就是部署多个Spring boot 项目。在这里就不多说了
更复杂的可以使用参数化构建