公司的CI/CD是使用Jenkins,开发、测试、预发的CI和CD都是在一起的,而生产环境的CI/CD我们是分开的

CI任务结束之后,开发可以选择发布哪个release版本。

可以看一下整体预览情况:

1.png

每个Job的Pipeline状态:

2.png

自定义发布机器、同时有发布及回滚功能:

3.png


我们都是基于maven的Java应用,进行编译打包其实比较简单,这里的CI较为简单,我这里只简单说明及其需要注意的点

  1. maven编译打包时,可以去掉单元测试

  2. 若Jenkins的机器配置比较高,可以可以开启maven的并发编译(3.3版本以上,默认支持并发)

  3. 适当调整maven的JVM


下面着重介绍CD部分的配置

以: 我们的msg-server服务为例

  1. 添加参数化构建

    a.添加PROJECT(字符参数):标识:应用名称

     image.png

      b.添加DEPLOY_TYPE(Active Choices Parameter ):标识:发布还是回滚

       image.png

      c.添加Version( Active Choices Reactive Parameter的):标识发布的版本

      image.png

    d.添加 HOTS( Active Choices Reactive Parameter),表示发布的主机清单

   image.png

 e.添加 DEPLOYED_HOSTS(Active Choices Reactive Parameter ),标识:已经发布过的主机清单

image.png


2.基础共用脚本部分

cat get_versions.sh


#!/bin/bash

#prod

packageDir="/data/jenkins/repos/master/"

#test

#packageDir="/data/jenkins/repos/test"

#dev

#packageDir="/data/jenkins/repos/dev"


project=$1

deploy_type=$2


if [[ "${deploy_type}" == "Rollback" ]] ; then 

  lasted_version=`ls -lt ${packageDir}/${project}|grep ${project}|awk '{print $9}'|awk -F '.' '{print $1}'`

  echo ${lasted_version}|awk -F " " '{for(i=1;i<=NF;i++) a[i,NR]=$i}END{for(i=1;i<=NF;i++) {for(j=1;j<=NR;j++) printf a[i,j] " ";print ""}}'

else

  lasted_version=`ls -lt ${packageDir}/${project}|grep ${project}|awk '{print $9}'|awk -F '.' '{print $1}'`

  echo ${lasted_version}|awk -F " " '{for(i=1;i<=NF;i++) a[i,NR]=$i}END{for(i=1;i<=NF;i++) {for(j=1;j<=NR;j++) printf a[i,j] " ";print ""}}'

fi


cat get_deploy_lists.sh


#!/bin/bash


PACKAGE_DIR=/data/jenkins/repos/release_package


project=$1

deploy_type=$2

version=$3


db_url='ops-db.xxxxxx.com'

db_user='xxxxx'

db_pwd="xxxxxxxx"

db_port=3306


format_version=`echo ${version}|awk -F '##' '{print $1}'`


#查询已经发布或回滚过的主机清单


query_sql="select host_ip from deploy_history where status=1 and deploy_type='${deploy_type}'  and version='${format_version}' and hostname like '%${project}%' "

result=`mysql -u${db_user} -p${db_pwd} -h${db_url} -P${db_port} -B jenkins  -e "${query_sql}" |awk 'NR>1'`


echo ${result}|awk -F " " '{for(i=1;i<=NF;i++) a[i,NR]=$i}END{for(i=1;i<=NF;i++) {for(j=1;j<=NR;j++) printf a[i,j] " ";print ""}}'


cat get_undeploy_lists.sh


#!/bin/bash


PACKAGE_DIR=/data/jenkins/repos/release_package


project=$1

deploy_type=$2

version=$3


db_url='ops-db.xxxxxxx.com'

db_user='xxxxx'

db_pwd="xxxxxxxx"

db_port=3306


format_version=`echo ${version}|awk -F '##' '{print $1}'`

#查询待发布的主机

#if [ "${deploy_type}" ==  "Deploy"  ] ; then

 

#query_sql="select host_ip from hosts where host_ip not in (select host_ip from deploy_history where status=1 and deploy_type='${deploy_type}'  and version='${format_version}' and hostname like '%${project}%' )  "

#result=`mysql -u${db_user} -p${db_pwd} -h${db_url} -P${db_port} -B jenkins  -e "${query_sql}" |awk 'NR>1'`

#echo ${result}|awk -F " " '{for(i=1;i<=NF;i++) a[i,NR]=$i}END{for(i=1;i<=NF;i++) {for(j=1;j<=NR;j++) printf a[i,j] " ";print ""}}'

#fi 

#if [ "${deploy_type}" ==  "Rollback"  ] ; then       

#if rollback ,

#        query_sql="select host_ip from hosts where host_ip not in (select host_ip from deploy_history where status=1 and deploy_type='Rollback'  and version='${format_version}' and hostname like '%${project}%' )  "

#        result=`mysql -u${db_user} -p${db_pwd} -h${db_url} -P${db_port} -B jenkins  -e "${query_sql}" |awk 'NR>1'`

#        echo ${result}|awk -F " " '{for(i=1;i<=NF;i++) a[i,NR]=$i}END{for(i=1;i<=NF;i++) {for(j=1;j<=NR;j++) printf a[i,j] " ";print ""}}'

#fi




        query_sql="select host_ip from hosts where host_ip not in (select host_ip from deploy_history where status=1 and deploy_type='${deploy_type}'  and version='${format_version}' and hostname like '%${project}%' ) and hostname like '%${project}%' "


        result=`mysql -u${db_user} -p${db_pwd} -h${db_url} -P${db_port} -B jenkins  -e "${query_sql}" |awk 'NR>1'`


        echo ${result}|awk -F " " '{for(i=1;i<=NF;i++) a[i,NR]=$i}END{for(i=1;i<=NF;i++) {for(j=1;j<=NR;j++) printf a[i,j] " ";print ""}}'


cat deploy_success_update.sh


#!/bin/bash


project=$1

host_ip=$2

deploy_type=$3

version=$4


db_url='ops-db.xxxxxxxx.com'

db_user='xxxxx'

db_pwd="xxxxxx"

db_port=3306


insert_sql="insert into  deploy_history(hostname,host_ip,deploy_type,status,version) values(\"${project}\",\"${host_ip}\",\"${deploy_type}\",1,\"${version}\" )"

echo ${insert_sql}


count_result=`mysql -u${db_user} -p${db_pwd} -h${db_url} -P${db_port} -B jenkins  -e "${insert_sql}" `


3.创建数据库及定义表结构


/*

Navicat MySQL Data Transfer


Source Server         : ops-db

Source Server Version : 50728

Source Host           : ops-db. xxxxx.com:3306

Source Database       : jenkins


Target Server Type    : MYSQL

Target Server Version : 50728

File Encoding         : 65001


Date: 2020-08-13 14:25:52

*/


Create Database: CREATE DATABASE `jenkins` /*!40100 DEFAULT CHARACTER SET utf8mb4 */


SET FOREIGN_KEY_CHECKS=0;


-- ----------------------------

-- Table structure for deploy_history

-- ----------------------------

CREATE TABLE `deploy_history` (

  `id` bigint(20) NOT NULL AUTO_INCREMENT,

  `hostname` varchar(255) NOT NULL COMMENT '发布的主机名',

  `host_ip` varchar(255) NOT NULL COMMENT '发布主机IP地址',

  `deploy_type` varchar(255) NOT NULL,

  `status` tinyint(1) NOT NULL COMMENT '0 发布失败; 1发布成功',

  `version` varchar(255) NOT NULL,

  `ctm_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

  PRIMARY KEY (`id`),

  KEY `udx_deploy_type_version` (`deploy_type`,`version`) USING BTREE,

  KEY `idx_host_ip` (`host_ip`) USING BTREE

) ENGINE=InnoDB AUTO_INCREMENT=128 DEFAULT CHARSET=utf8mb4;


-- ----------------------------

-- Table structure for hosts

-- ----------------------------

CREATE TABLE `hosts` (

  `id` int(11) NOT NULL AUTO_INCREMENT,

  `hostname` varchar(255) NOT NULL,

  `host_ip` varchar(255) NOT NULL,

  `comment` varchar(255) DEFAULT NULL,

  PRIMARY KEY (`id`),

  KEY `idx_host_ip` (`host_ip`) USING BTREE

) ENGINE=InnoDB AUTO_INCREMENT=19 DEFAULT CHARSET=utf8mb4;


-- ----------------------------

-- Table structure for hosts-test

-- ----------------------------

CREATE TABLE `hosts-test` (

  `id` int(11) NOT NULL AUTO_INCREMENT,

  `hostname` varchar(255) NOT NULL,

  `host_ip` varchar(255) NOT NULL,

  `comment` varchar(255) DEFAULT NULL,

  PRIMARY KEY (`id`),

  KEY `idx_host_ip` (`host_ip`) USING BTREE

) ENGINE=InnoDB AUTO_INCREMENT=19 DEFAULT CHARSET=utf8mb4;

#可以查看以往的历史发布记录:

6.png

  4.Pipelin流水线代码

   image.png  

#Groovy代码如下:

/**

 * Description:Master环境JavaSpringBoot的Pipeline脚本

 * Author: ledi

 * Date:2020-03-01

 */


pipeline {

    agent any


    environment {

        packageDir="/data/jenkins/repos/master"

        backupDir="/data/release_package"

        deployDir="/data/www"

        ansible_Dir="/etc/ansible/roles/masterv2"

        port="20881"



    }


    stages {



        stage("推包至远端"){

            steps{


                script{

    if ( "${DEPLOY_TYPE}" == 'Deploy' ){

                    sh '''

HOSTS=`echo ${HOSTS}|sed s/[[:space:]]//g`

                    source /etc/profile &> /dev/null

                    #/bin/mv  ${WORKSPACE}/${PROJECT}/target/${PROJECT}.jar ${packageDir}/${PROJECT}_${BUILD_NUMBER}.jar

                    old_version=`echo "${VERSION}"`

                    new_version=`echo "${VERSION}"|awk -F '##' '{print $1}'`

                    ansible-playbook -i ${ansible_Dir}/hosts ${ansible_Dir}/copy_jar.yaml -e "backupDir=${backupDir} packageDir=${packageDir} old_version=${old_version}  new_version=${new_version} project=${PROJECT}" --limit ${HOSTS}

                   '''

   }else{

echo "执行回滚操作,无需推包至远端"

}


                } // end script

            } // end steps

        } //end stage



        stage("停应用") {

            steps {


                script {

                    sh '''

HOSTS=`echo ${HOSTS}|sed s/[[:space:]]//g`

                    ansible-playbook -i ${ansible_Dir}/hosts ${ansible_Dir}/stop_app.yaml -e "project=${PROJECT} " --limit ${HOSTS}

                    '''

                }

            }

        }


        stage("启应用"){

            steps{


                script {

                    sh '''

HOSTS=`echo ${HOSTS}|sed s/[[:space:]]//g`

new_version=`echo "${VERSION}"|awk -F '##' '{print $1}'`

                    ansible-playbook -i ${ansible_Dir}/hosts ${ansible_Dir}/start_app.yaml -e "project=${PROJECT} new_version=${new_version} " --limit ${HOSTS}

                    '''

                }

            }

        }


        stage("健康检查"){

            steps{

                script{


                    sh '''

HOSTS=`echo ${HOSTS}|sed s/[[:space:]]//g`

                    hosts_ip=`echo ${HOSTS} |sed  "s/,/  /g" `

new_version=`echo "${VERSION}"|awk -F '##' '{print $1}'`

                    function check_health(){

                    for (( i=1;i<="$#";i++ )); do

                     ansible ${!i} -u www -i ${ansible_Dir}/hosts  -m shell -a "tail -1000 /data/logs/${PROJECT}/nohup.out"


                      for (( j=1;j<=60;j++ ))

                        do

                             echo "ip is ${!i}"

                             if  [[ `(echo "status -l ";sleep 1;exit)|telnet ${!i} ${port} |grep "server"| grep -o "OK"` == "OK" ]]; then

                                    echo  " ^_^^_^ IP ${!i} ${port} 端口检查成功 ^_^^_^"

sh /data/jenkins/scripts/deploy_success_update.sh ${PROJECT} ${!i}  ${DEPLOY_TYPE} ${new_version} 


                                    break 1;

                             else

                                    echo "==== IP ${!i} ${port} 端口异常,继续探测 ==== "

                                    sleep 3

                             fi

                        done

                    

                       if [[ `(echo "status -l ";sleep 1;exit)|telnet ${!i} ${port} |grep "server"| grep -o "OK"` != "OK" ]] ; then

                          echo  "==== IP ${!i} ${port} 端口异常,启动失败,请检查应用 === "

                          exit 1

                        fi

                    done

                    

                    }

                    

                    check_health ${hosts_ip}

                     '''


                }// end script


            } // end steps

            } // end stage heal





    } //end stages


}


说明:

这里也可以将此Groovy代码和应用放在一起,放入GitLab中,这样更容易维护及管理。

5.Ansible 公用部分

cat copy_jar.yaml 

---

- hosts: all

  gather_facts: False

  remote_user: root

  vars:

    backupDir: {backupDir}

    packageDir: {packageDir}

    project: {PROJECT}

    old_version: {old_version}

    new_version: {new_version}

  tasks:

   - name: Copy Jar to Remoute Machine

     copy: src={{ packageDir }}/{{project }}/{{ old_version }}.jar    dest={{ backupDir }}/{{ new_version }}.jar   owner=www group=www mode=0664 force=yes

cat stop_app.yaml

---

- hosts: all

  gather_facts: False

  remote_user: root

  vars:

    project: {PROJECT}

  tasks:

    - name: Stop Java App

      shell: sh /home/www/scripts/stop_springboot.sh {{ project }} owner=www group=www

cat start_app.yaml 

---

- hosts: all

  gather_facts: False

  remote_user: www

  vars:

    project: {PROJECT}

    new_version: {new_version}

  tasks:

    - name: Start Java App

      shell: /bin/bash  /home/www/scripts/start_springboot.sh  {{ project }} {{ new_version }} owner=www group=www


最后,开发们可以自定义发布主机,假设一个微服务有20台机器,可以先选择5台进行发布,通过后,再选择5台或以上,然后慢慢类似滚动发布。

存在不足的点:

  1. 若一个微服务有20台机器,若采用每5台发布,则开发需要手工进行4次操作,改进的地方:若第一次5台成功后,则自动执行后面的操作

  2. 在使用Ansible推包到阿里云ECS时,很慢;这个是我们的网络机房问题,因为打包机器在公司内部,和阿里云ECS通过虚拟隧道过去的,所以比较慢,改进的地方:

 一是将打包机器也迁移到云端;二是通过专线将公司网络与阿里云互通,形成物理线路;不过上面两个都是需要¥的哈,运维同学不单单考虑系统可用性、稳定性,也要为公司节约一定成本哈。(不知道领导看到了,会不会给我加个鸡腿,哈哈)