为什么要写企业级持续集成(jenkins + pipeline + k8s)？

目前网上自动化持续集成的资料很多，但基本上都是局限于jenkins自由风格的job，结合shell脚本来实现持续集成，这种方式的缺点也很明显：

　　1、构建出问题，排查困难

　　2、构建节点挂了，就不能完成构建任务

而当前主流技术是 “ k8s + 微服务 ” 等，我们完全可以利用k8s的优势来完成持续构建任务，每次构建时可以调度到任意节点，或者是具有指定标签的节点，这就实现了高可用，另外，结合pipeline，可以轻松简单实现持续集成，并且哪个阶段出问题了一目了然，排查起来也容易。

自动化测试框架

0基础到实现：java + testng + httpclient + allure

企业级持续集成技术栈

git + gitlab + jenkins + pipeline + maven + harbor + docker + k8s

可以整合python、java等各种自动化测试框架

流程：拉取代码--》mvn打包--》构建镜像--》新镜像发布到k8s--》拉取自动化测试代码--》自动化测试--》allure报告

环境：jenkins使用k8s作为构建环境，在某个节点执行测试

持续集成常用工具 持续集成的缺点_jenkins

环境规划

192.168.117.160：harbor、jenkins、maven

192.168.117.161：k8s-master

192.168.117.162：k8s-node01

192.168.117.163：k8s-node02

192.168.117.180：gitlab

每台虚拟机都安装了git

环境搭建(含必备基础)

git安装及使用：

gitlab安装及使用：

maven安装及使用：

docker安装及使用：

harbor安装以及使用：

k8s安装：(k8s操作：)

jenkins搭建及devops自动化平台相关配置：

allure报告：allure-commandline下载、安装、配置(linux或者docker)，

jenkins把自动化测试结果发送到钉钉群：

pipeline设计

pipeline常用功能：

使用Blue Ocean设计pipeline脚本：

说明：由于资源不足，暂未加入sonarqube等

持续集成常用工具 持续集成的缺点_docker_02

pipeline具体实现

结合上面在Blue Ocean中设计的pipeline骨架，我们来完善并实现整个过程

parameters

可以选择要构建的分支



parameters {
gitParameter branch: '', branchFilter: 'origin/(.*)', defaultValue: '', description: '构建的分支', name: 'BRANCH', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'GitParameterDefinition'
}



环境变量



environment {
  HARBOR_ADDRESS = "192.168.117.160"
  IMAGE_NAME = "gift"
  NAMESPACE = "gift"
}



agent

k8s作为构建环境



agent {
    kubernetes {
      cloud 'qzcsbj_kubernetes'
      yaml '''apiVersion: v1
kind: Pod
spec:
  containers: 
    - image: 'registry.cn-chengdu.aliyuncs.com/qzcsbj6/jnlp:alpine'      
      name: jnlp
      imagePullPolicy: IfNotPresent
      args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\']
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "localtime"
          readOnly: false
    - image: "registry.cn-chengdu.aliyuncs.com/qzcsbj6/kubectl"
      imagePullPolicy: "IfNotPresent"
      name: "kubectl"
      tty: true
      command:
        - "cat"
      env: 
        - name: "LANGUAGE"
          value: "en_US:en"
        - name: "LANG"
          value: "en_US.UTF-8"
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "localtime"
          readOnly: false
    - image: "registry.cn-chengdu.aliyuncs.com/qzcsbj6/docker"
      imagePullPolicy: "IfNotPresent"
      name: "docker"
      tty: true
      command:
        - "cat"
      env:
        - name: "LANGUAGE"
          value: "en_US:en"
        - name: "LANG"
          value: "en_US.UTF-8"
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "localtime"
          readOnly: false
        - mountPath: "/var/run/docker.sock"
          name: "dockersock"
          readOnly: false
    - image: "registry.cn-chengdu.aliyuncs.com/qzcsbj6/maven"
      imagePullPolicy: "IfNotPresent"
      name: "maven"
      tty: true
      command:
        - "cat"
      env:
        - name: "LANGUAGE"
          value: "en_US:en"
        - name: "LANG"
          value: "en_US.UTF-8"
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "localtime"
        - mountPath: "/root/.m2/"
          name: "m2dir"
          readOnly: false
    - image: "registry.cn-chengdu.aliyuncs.com/allure-commandline"
      imagePullPolicy: "IfNotPresent"
      name: "allure"
      tty: true
      command:
        - "cat"
      env:
        - name: "LANGUAGE"
          value: "en_US:en"
        - name: "LANG"
          value: "en_US.UTF-8"
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "localtime"
          readOnly: false
  restartPolicy: "Never"
  volumes:
    - name: "dockersock"
      hostPath:
        path: "/var/run/docker.sock"
    - name: "localtime"
      hostPath:
        path: "/usr/share/zoneinfo/Asia/Shanghai"
    - name: "m2dir"
      hostPath: 
        path: "/opt/m2"
''' 
    }
  }



pull project code



stage('pull project code') {
      parallel {
        stage('ui build') {
          when {
            expression {
              env.gitlabBranch == null
            }
          }
          steps {
            sh """
              echo '=================开始拉取项目代码'
            """

            git(url: 'git@192.168.117.180:qzcsbj/gift.git', branch: "${BRANCH}", credentialsId: 'qzcsbj_gitlab')
            script {
                COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
                TAG = BUILD_TAG + '-' + COMMIT_ID 
                println "Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
            }            
          }
          post {
            failure {
              dingtalk (
                robot:'dd01', 
                type:'MARKDOWN',
                atAll: true,
                title: "notice: 拉取项目代码失败",
                text: ["#### '${JOB_NAME}'项目代码拉取失败\n - 构建：第'${BUILD_NUMBER}'次\n - 状态：'${currentBuild.result}'\n - [查看本次构建详情](${BUILD_URL})"]
              )
            }
          }
        }
        stage('trigger build') {
          steps {
            sh """
              echo "待更新。。。"
            """
          }
        }
      }
    }



mvn package



stage('mvn package') {
      steps {
        container(name: 'maven') {
          sh """
            echo '=================mvn开始打包'
            mvn clean package -Dmaven.test.skip=true
            echo '=================mvn打包完成'
          """ 
        }
      }
      post {
        failure {
          dingtalk (
            robot:'dd01', 
            type:'MARKDOWN',
            atAll: true,
            title: "notice: mvn package失败",
            text: ["#### '${JOB_NAME}'项目mvn package失败\n - 构建：第'${BUILD_NUMBER}'次\n - 状态：'${currentBuild.result}'\n - [查看本次构建详情](${BUILD_URL})"]
          )
        }
      }
    }



build and push image



stage('build and push image') {
      environment {
        HARBOR_USER = credentials('qzcsbj_harbor')
      }
      steps {
        container(name: 'docker') {
          sh """
            echo '=================开始构建镜像'
            docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .
            docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS}
            docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}
            echo '=================镜像推送完成'
          """ 
        }
      }
      post {
        failure {
          dingtalk (
            robot:'dd01', 
            type:'MARKDOWN',
            atAll: true,
            title: "notice: 构建或者推送镜像失败",
            text: ["#### '${JOB_NAME}'项目构建或者推送镜像失败\n - 构建：第'${BUILD_NUMBER}'次\n - 状态：'${currentBuild.result}'\n - [查看本次构建详情](${BUILD_URL})"]
          )
        }
      }
    }



deploy to k8s



stage('deploy to k8s') {
      environment {
        MY_KUBECONFIG = credentials('qzcsbj_k8s')
      }
      steps {
        container(name: 'kubectl'){
          sh """
            echo '=================开始部署到k8s'
            /usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE
            echo '=================部署到k8s完成'
          """
        }
      }
      post {
        success {
          dingtalk (
            robot:'dd01',
            type:'MARKDOWN',
            atAll: true,
            title: "notice: 自动化部署到k8s完成",
            text: ["#### '${JOB_NAME}'项目自动化部署到k8s完成\n - 构建：第'${BUILD_NUMBER}'次\n - 状态：'${currentBuild.result}'\n - [查看本次构建详情](${BUILD_URL})"]
          )
        }
        failure {
          dingtalk (
            robot:'dd01',
            type:'MARKDOWN',
            atAll: true,
            title: "notice: 自动化部署到k8s失败",
            text: ["#### '${JOB_NAME}'项目自动化部署到k8s失败\n - 构建：第'${BUILD_NUMBER}'次\n - 状态：'${currentBuild.result}'\n - [查看本次构建详情](${BUILD_URL})"]
          )
        }
      }
    }



pull autotest code



stage("pull autotest code"){
      steps{
        sh """
          echo '=================开始拉取自动化测试代码'
        """
        git(
          credentialsId: 'qzcsbj_gitlab',
          url: 'git@192.168.117.180:root/apiautotest.git'
        )
      }
      post {
        failure {
          dingtalk (
            robot:'dd01',
            type:'MARKDOWN',
            atAll: true,
            title: "notice: 拉取自动化测试代码失败",
            text: ["#### 拉取自动化测试代码失败\n - 构建：第'${BUILD_NUMBER}'次\n - 状态：'${currentBuild.result}'\n - [查看本次构建详情](${BUILD_URL})"]
          )
        }
      }
    }



run autotest



stage("run autotest"){
      steps{
        sh """
          mvn clean test -Dsurefire.suiteXmlFiles=testngXML/testng.xml
        """
      }
      post {
        success {
          dingtalk (
            robot:'dd01',
            type:'MARKDOWN',
            atAll: true,
            title: "notice: 自动化测试完成",
            text: ["#### '${JOB_NAME}'项目自动化测试完成\n - 构建：第'${BUILD_NUMBER}'次\n - 状态：'${currentBuild.result}'\n - [查看本次构建详情](${BUILD_URL}console)"]
          )
        }
        failure {
          dingtalk (
            robot:'dd01',
            type:'MARKDOWN',
            atAll: true,
            title: "notice: 自动化测试运行出错",
            text: ["#### '${JOB_NAME}'项目自动化测试运行出错\n - 构建：第'${BUILD_NUMBER}'次\n - 状态：'${currentBuild.result}'\n - [查看本次构建详情](${BUILD_URL})"]
          )
        }
      }
    }



allure report



stage("allure report"){
      steps{
        sh """
          echo '=================开始生成测试报告'
        """
        allure(
          includeProperties: false,
          jdk: '',
          results: [[path: 'target/allure-results']]
        )
      }
      post {
        success {
          dingtalk (
            robot:'dd01', 
            type:'MARKDOWN',
            atAll: true,
            title: "notice: 自动化测试报告已生成，全部通过",
            text: ["#### '${JOB_NAME}'项目自动化测试报告已生成，全部通过\n - 构建：第'${BUILD_NUMBER}'次\n - 状态：'${currentBuild.result}'\n - [查看本次测试报告](${BUILD_URL}allure)"]
          )
        }
        unstable {
          dingtalk (
            robot:'dd01', 
            type:'MARKDOWN',
            atAll: true,
            title: "notice: 自动化测试报告已生成，有未通过的",
            text: ["#### '${JOB_NAME}'项目自动化测试报告已生成，有未通过的\n - 构建：第'${BUILD_NUMBER}'次\n - 状态：'${currentBuild.result}'\n - [查看本次测试报告](${BUILD_URL}allure)"]
          )
        }
        failure {
          dingtalk (
            robot:'dd01', 
            type:'MARKDOWN',
            atAll: true,
            title: "notice: 自动化测试报告生成出错",
            text: ["#### '${JOB_NAME}'项目自动化测试报告生成出错\n - 构建：第'${BUILD_NUMBER}'次\n - 状态：'${currentBuild.result}'\n - [查看本次构建详情](${BUILD_URL})"]
          )
        }
      }
    }



构建及效果展示

pipeline脚本可以放在流水线项目的脚本框中

持续集成常用工具 持续集成的缺点_docker_03

也可以放gitlab上，选择SCM

持续集成常用工具 持续集成的缺点_devops_04

持续集成常用工具 持续集成的缺点_ci_05

持续集成常用工具 持续集成的缺点_docker_06

pipeline脚本放到Jenkinsfile中，Jenkinsfile在gift项目master分支下

持续集成常用工具 持续集成的缺点_docker_07

第一次构建，点击“Build Now”，会失败，因为我们使用了参数化构建，这次构建获取不到参数值

持续集成常用工具 持续集成的缺点_ci_08

刷新页面，就可以看到参数化构建了，就可以选择要构建的分支

持续集成常用工具 持续集成的缺点_devops_09

部署到k8s的通知

持续集成常用工具 持续集成的缺点_docker_10

消息

持续集成常用工具 持续集成的缺点_jenkins_11

运行自动化测试的通知

持续集成常用工具 持续集成的缺点_jenkins_12

消息

持续集成常用工具 持续集成的缺点_devops_13

生成allure测试报告的通知

持续集成常用工具 持续集成的缺点_docker_14

消息

持续集成常用工具 持续集成的缺点_devops_15

测试报告的通知可以优化，直接链接到allure测试报告

持续集成常用工具 持续集成的缺点_持续集成常用工具_16

测试报告

持续集成常用工具 持续集成的缺点_ci_17

持续集成常用工具 持续集成的缺点_jenkins_18