基于kubernetes的jenkins持续集成

发布流程设计

在kubernetes中部署jenkins

-参考：https://github.com/jenkinsci/kubernetes

  • yml文件
jenkins
├── deployment.yml     #jenkins 部署
├── rbac.yml           #权限准入文件
└── service.yml        #服务暴露文件
  • 部署jenkins
$ kubectl apply -f jenkins/
$ kubectl get pod  -n ops
NAME                      READY   STATUS    RESTARTS   AGE
jenkins-9cc69b868-zms8w   1/1     Running   0          22h
$ kubectl logs jenkins-9cc69b868-zms8w  -n ops   #从日志中找到默认的admin密码

  • 登陆http://NodeIP:30008

  • 使用日志中的临时密码解锁jenkins

  • 选择插件

  • 选择无

  • 创建admin与密码完成登陆

  • 配置插件源

默认从国外网络下载插件，会比较慢，建议修改国内源：

# 找到NFS服务器，修改卷中的数据
$ cd /ifs/kubernetes/ops-jenkins-pvc-8947582f-11d3-47ed-92c0-bfdbf8aae813/updates/
$ sed -i 's/http:\/\/updates.jenkins-ci.org\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g' default.json
$ sed -i 's/http:\/\/www.google.com/https:\/\/www.baidu.com/g' default.json
# 删除pod重建，pod名称改成你实际的
kubectl delete pod jenkins-dccd449c7-vx6sj -n ops
  • 安装插件
    • 管理Jenkins->系统配置-->管理插件-->分别搜索Git Parameter/Git/Pipeline/kubernetes/Config File Provider，选中点击安装。
      • Git Parameter：Git参数化构建
      • Git：拉取代码
      • Pipeline：流水线
      • kubernetes：连接Kubernetes动态创建Slave代理
      • Config File Provider：存储kubectl用于连接k8s集群的kubeconfig配置文件

添加kubernetes集群到jenkins

管理Jenkins->Manage Nodes and Clouds->configureClouds->Add

在kubernetes中构建动态slave

构建slave镜像

Kubernetes插件：Jenkins在Kubernetes集群中运行动态代理

插件介绍：https://github.com/jenkinsci/kubernetes-plugin

​ https://github.com/jenkinsci/docker-inbound-agent

$ tree jenkins-slave/
jenkins-slave/
├── Dockerfile
├── jenkins-slave
├── kubectl
├── settings.xml
└── slave.jar

课件目录里涉及四个文件：

  • Dockerfile 构建镜像
  • jenkins-slave shell脚本启动slave.jar
  • settings.xml 修改maven官方源为阿里云源
  • slave.jar agent程序，接受master下发的任务

构建并推送到镜像仓库：

$ cd jenkins-slave/
$ docker build -t hub.cropy.cn/library/jenkins-slave-jdk:1.8 .
$ docker push hub.cropy.cn/library/jenkins-slave-jdk:1.8

二进制部署的k8s需要kubeconfig文件

  • 如果是kubeadm搭建的集群则不需要考虑这一步

  • 签发CA证书

$ cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}
EOF

$ cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

$ cfssl gencert -ca=/data/kubernetes/ssl/ca.pem -ca-key=/data/kubernetes/ssl/ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
  • 生成kubeconfig授权文件
# kubeconfig 
kubectl config set-cluster kubernetes \
  --certificate-authority=/data/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://192.168.56.14:6443 \
  --kubeconfig=admin.kubeconfig

# 设置客户端认证
kubectl config set-credentials admin \
  --client-key=admin-key.pem \
  --client-certificate=admin.pem \
  --embed-certs=true \
  --kubeconfig=admin.kubeconfig

# 设置默认上下文
kubectl config set-context kubernetes \
  --cluster=kubernetes \
  --user=admin \
  --kubeconfig=admin.kubeconfig

# 设置当前使用配置
kubectl config use-context kubernetes --kubeconfig=admin.kubeconfig
  • 绑定权限到admin
$  vim admin.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: admin
  
$ kubectl apply -f admin.yml
$ kubectl get pod --kubeconfig=admin.kubeconfig -n ops    # 测试权限
$ cp admin.kubeconfig ~/.kube/config

jenkins PIPELINE

Jenkins Pipeline是一套插件，支持在Jenkins中实现集成和持续交付管道；

  • Pipeline通过特定语法对简单到复杂的传输管道进行建模；
  • 声明式：遵循与Groovy相同语法。pipeline { }
  • 脚本式：支持Groovy大部分功能，也是非常表达和灵活的工具。node { }
  • Jenkins Pipeline的定义被写入一个文本文件，称为Jenkinsfile。

PIpeline 流水线流程

  • 新建pipeline job

  • 选择kubernetes 声明式pipeline

pipline脚本内容如下：

pipeline {
    agent {
        kubernetes {
            label "jenkins-slave"
            yaml '''
apiVersion: v1
kind: Pod
metadata:
  name: jenkins-slave
spec:
  containers:
  - name: jnlp
    image: hub.cropy.cn/library/jenkins-slave-jdk:1.8
    command:
    - sleep
    args:
    - infinity
'''
            // Can also wrap individual steps:
            // container('shell') {
            //     sh 'hostname'
            // }
            defaultContainer 'shell'
        }
    }
    stages {
        stage('Main') {
            steps {
                sh 'hostname'
            }
        }
    }
}

jenkins常见的认证信息保存

保存git和harbor的凭据ID

1bbda277-a0e0-42c9-b4f3-be22e6344d66

6020966e-3469-44e7-baff-e9945fbaa4f3

k8s认证配置文件放到jenkins

  • 添加配置

  • 选择自定义的加入即可

拷贝好ID： 4f0526c1-18c8-48c7-bd22-eeb8b179a950

基于参数化构建

  • 创建pipelinejob
  • 选择参数
  • 选择参数示例使用

image-20200808220707453

  • 修改Git中的yml文件参数

  • pipeline脚本

// 公共
def registry = "hub.cropy.cn"
// 项目
def project = "dev"
def app_name = "java-demo"
def image_name = "${registry}/${project}/${app_name}:${BUILD_NUMBER}"
def git_address = "http://192.168.56.19:9999/root/java-demo.git"
// 认证
def secret_name = "registry-pull-secret"
def harbor_auth = "6020966e-3469-44e7-baff-e9945fbaa4f3"
def git_auth = "1bbda277-a0e0-42c9-b4f3-be22e6344d66"
def k8s_auth = "4f0526c1-18c8-48c7-bd22-eeb8b179a950"

pipeline {
  agent {
    kubernetes {
        label "jenkins-slave"
        yaml """
kind: Pod
metadata:
  name: jenkins-slave
spec:
  containers:
  - name: jnlp
    image: "${registry}/library/jenkins-slave-jdk:1.8"
    imagePullPolicy: Always
    volumeMounts:
      - name: docker-cmd
        mountPath: /usr/bin/docker
      - name: docker-sock
        mountPath: /var/run/docker.sock
      - name: maven-cache
        mountPath: /root/.m2
  volumes:
    - name: docker-cmd
      hostPath:
        path: /usr/bin/docker
    - name: docker-sock
      hostPath:
        path: /var/run/docker.sock
    - name: maven-cache
      hostPath:
        path: /tmp/m2
"""
        }
      
      }
    parameters {    
        gitParameter branch: '', branchFilter: '.*', defaultValue: 'master', description: '选择发布的分支', name: 'Branch', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH'
        choice (choices: ['1', '3', '5', '7'], description: '副本数', name: 'ReplicaCount')
        choice (choices: ['dev','test','prod'], description: '命名空间', name: 'Namespace')
    }
    stages {
        stage('拉取代码'){
            steps {
                checkout([$class: 'GitSCM', 
                branches: [[name: "${params.Branch}"]], 
                doGenerateSubmoduleConfigurations: false, 
                extensions: [], submoduleCfg: [], 
                userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]
                ])
            }
        }

        stage('代码编译'){
           steps {
             sh """
                mvn clean package -Dmaven.test.skip=true
                """ 
           }
        }

        stage('构建镜像'){
           steps {
                withCredentials([usernamePassword(credentialsId: "${harbor_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
                sh """
                  echo '
                    FROM ${registry}/base/tomcat:v1
                    LABEL maitainer wanghui
                    RUN rm -rf /usr/local/tomcat/webapps/*
                    ADD target/*.war /usr/local/tomcat/webapps/ROOT.war
                  ' > Dockerfile
                  docker build -t ${image_name} .
                  docker login -u ${username} -p '${password}' ${registry}
                  docker push ${image_name}
                """
                }
           } 
        }
        stage('部署到K8S平台'){
          steps {
              configFileProvider([configFile(fileId: "${k8s_auth}", targetLocation: "admin.kubeconfig")]){
                sh """
                  sed -i 's#IMAGE_NAME#${image_name}#' deploy.yaml
                  sed -i 's#NAMESPACE#${Namespace}#' deploy.yaml
                  sed -i 's#REPLICAS#${ReplicaCount}#' deploy.yaml
                  kubectl apply -f deploy.yaml -n ${Namespace} --kubeconfig=admin.kubeconfig
                """
              }
          }
        }
    }
}
  • 部署到k8s这里面的kubeconfig需要使用生成的对应namespace的配置文件
  • docker镜像仓库地址加入secret：docker-regsitry-auth
$ kubectl create namespace test
$ kubectl create secret docker-registry docker-regsitry-auth --docker-username=admin --docker-password=Harbor12345 --docker-server=hub.cropy.cn -n test
  • deploy中的需要替换的大写参数也需要替换，然后提交之后再做jenkins编译
$cat deploy.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: java-demo
  namespace: NAMESPACE
spec:
  replicas: REPLICAS
  selector:
    matchLabels:
      project: www
      app: java-demo
  template:
    metadata:
      labels:
        project: www
        app: java-demo
    spec:
      imagePullSecrets: 
      - name: "docker-regsitry-auth"
      containers:
      - image: IMAGE_NAME
        name: java-demo
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        resources:
          requests:
            cpu: 0.5
            memory: 1Gi
          limits:
            cpu: 1
            memory: 2Gi
        livenessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 20
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: java-demo
  name: java-demo
  namespace: NAMESPACE
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
    nodePort: 30018
  selector:
    app: java-demo
  type: NodePort