完成03 Jenkins master安装（在Kubernetes平台上）的Jenkins master安装后，我们现在可以使用China-jenkins-config配置Jenkins的Jenkins Kubernetes插件了。

Jenkins Kubernetes使用场景

当我们有很多不同的team，每个team的Jenkins任务依赖不同的环境，比如team1需要maven环境，team2是ruby环境，team3是nodejs环境。我们可以给每个team都做一个node出来，并预安装上不同的环境，但是这样的node不容易做到动态扩容，比如一个node同时支持5个job在运行，但是现在有6个同时运行的任务，那么必然有一个会在等待中。

我们可以利用Jenkins Kubernetes插件利用Kubernetes动态创建Pod作为node的特性动态创建出新node，Job完成后自动销毁（根据Retention Policy），这样的设计能够最大化利用资源，一不需要一个node时刻standby（根据Retention Policy）按需创建，二是支持大量的Job同时运行且job结束后自行销毁。




Jenkins k8s agent jenkins k8s agent pipeline_docker


动态Jenkins slave


Jenkins k8s agent jenkins k8s agent pipeline_Jenkins k8s agent_02


pipeline

先决条件

  • 完成Jenkins master的搭建，得到Jenkins的URL（jenkinsUrl）
  • 我们的Jenkins Kubernetes使用的k8s我们就用同一个k8s，这种类型是Local Cluster，为了能够让Jenkins master调度Local Cluster创建pod，我们需要给Jenkins master的deployment加上额外的权限，要不然配置的时候会出现如下报错：


Jenkins k8s agent jenkins k8s agent pipeline_docker_03


image.png

Error testing connection https://kubernetes.default: Failure executing: GET at: https://kubernetes.default/api/v1/namespaces/default/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:george:default" cannot list pods in the namespace "default".

添加的内容请参考：PR-2

diff --git a/deploy.yaml b/deploy.yaml
index 2244b72..33cc202 100644
--- a/deploy.yaml
+++ b/deploy.yaml
@@ -52,6 +52,7 @@ spec:
           volumeMounts:
             - name: dind-storage
               mountPath: /var/lib/docker
+      serviceAccount: jenkins-admin-george
       volumes:
         - name: disk-pvc
           persistentVolumeClaim:
diff --git a/rbac.yaml b/rbac.yaml
new file mode 100644
index 0000000..cd68ffd
--- /dev/null
+++ b/rbac.yaml
@@ -0,0 +1,24 @@
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  labels:
+    k8s-app: jenkins
+  name: jenkins-admin-george
+  namespace: George
+
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  labels:
+    kubernetes.io/bootstrapping: rbac-defaults
+  name: jenkins-admin-george
+subjects:
+  - kind: ServiceAccount
+    name: jenkins-admin-george
+    namespace: George
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: admin #TODO: This is too open


现在测试一下local k8s的连通性应该就是OK的了


Jenkins k8s agent jenkins k8s agent pipeline_docker_04


image.png

  • [可选]如果是Remote Cluster，那么请按照https://illya-chekrygin.com/2017/08/26/configuring-certificates-for-jenkins-kubernetes-plugin-0-12/的Remote Cluster部分来配置名字为cluster-jenkins-slave的credential
  • 配置名为docker-registry-id的credential，这个credential会被Jenkinsfile用来推container image到dockerhub中


Jenkins k8s agent jenkins k8s agent pipeline_Jenkins k8s agent_05


docker-registry-id.gif

  • 在k8s中创建node-team1, node-team1, node-team1 namespaces
georgehe@Sha-51664-Mbp  ~/Documents/george-sre/jenkins-master   master  kubectl create ns node-team1
namespace "node-team1" created
 georgehe@Sha-51664-Mbp  ~/Documents/george-sre/jenkins-master   master  kubectl create ns node-team2
namespace "node-team2" created
 georgehe@Sha-51664-Mbp  ~/Documents/george-sre/jenkins-master   master  kubectl create ns node-team3
namespace "node-team3" created


完成以上的先决条件后，我们现在可以配置以下部分了

接入jenkins-config来配置Jenkins自己

Jenkins-config repo中的Jenkinsfile如下：


pipeline {

    options { disableConcurrentBuilds() }

    environment {
        // TAG_NAME = "${env.BRANCH_NAME == 'master' ? 'stable' : 'test'}"
        TAG_NAME = 'latest'
        REGISTRY_CREDENTIALS_ID = 'docker-registry-id'
    }

    agent { label "do-not-use-master" }

    stages {

        stage('RAW CONFIG HANDLING') {
            steps {
                sh "ls -lrt && pwd"
                sh "mkdir -p ${JENKINS_HOME}/jenkins_config/"
                sh "cp -rf * ${JENKINS_HOME}/jenkins_config/"
            }
        }

        stage('CONSUL TEMPLATE') {
            steps { sh 'consul-template -vault-addr "$VAULT_ADDR" -config "jenkins_config.hcl" -once -vault-retry-attempts=1 -vault-renew-token=false' }
        }

        stage('RUN GROOVY on master') {
            steps {
                load("/var/jenkins_home/jenkins_config/src/kubernetes.groovy")
            }
        }

        stage('Build dockerfiles') {
            steps {
                script {
                    docker.withRegistry('', "${REGISTRY_CREDENTIALS_ID}") {
                        docker.build("georgesre/jnlp-node-team1:${TAG_NAME}", "-f dockerfiles/node-team1/Dockerfile dockerfiles/node-team1").push()
                    }
                    docker.withRegistry('', "${REGISTRY_CREDENTIALS_ID}") {
                        docker.build("georgesre/jnlp-node-team2:${TAG_NAME}", "-f dockerfiles/node-team2/Dockerfile dockerfiles/node-team2").push()
                    }
                    docker.withRegistry('', "${REGISTRY_CREDENTIALS_ID}") {
                        docker.build("georgesre/jnlp-node-team3:${TAG_NAME}", "-f dockerfiles/node-team3/Dockerfile dockerfiles/node-team3").push()
                    }
                }
            }
        }

        stage('Run Parallel Tests') {
            parallel {
                stage('Test On node-team1') {
                    agent { label "node-team1" }
                    steps { sh "echo I am good on node-team1" }
                }
                stage('Test On node-team2') {
                    agent { label "node-team2" }
                    steps { sh "echo I am good on node-team2" }
                }
                stage('Test On node-team3') {
                    agent { label "node-team3" }
                    steps { sh "echo I am good on node-team3" }
                }
            }
        }
    }
}


  1. 创建新GitHub Organizatin类型的Job，请参考05 GitHub Repo通过Jenkinsfile接入到Jenkins查看具体步骤与其他参数配置


Jenkins k8s agent jenkins k8s agent pipeline_Test_06


配置

  • 请务必使用Credential github_token_userpass_cred，且此token有所有repo的admin权限（拉取代码，GitHub Commit Status，设置webhook等）
  1. 可以通过Scan Organization Log查看Scan Organization Log
    当Scan Organization扫描到george-sre/jenkins-config的时候
Proposing jenkins-config
Examining george-sre/jenkins-config

  Checking branches...

  Getting remote branches...

    Checking branch master

  Getting remote pull requests...
      ‘Jenkinsfile’ found
    Met criteria

  1 branches were processed (query completed)

  1 branches were processed

Finished examining george-sre/jenkins-config


这个时候我们的jenkins-config repo就已经被接入到jenkins中，而且会自动的运行。

查看以上Jenkinsfile的配置结果

  • Jenkins的Kubernetes插件的配置已经被创建出来


Jenkins k8s agent jenkins k8s agent pipeline_Test_07


config.gif

  • 查看动态slave的创建过程
    当Jenkins job运行到Jenkinsfile的测试部分Jenkinsfile#L49
stage('Test On node-team1') {
                    agent { label "node-team1" }
                    steps { sh "echo I am good on node-team1" }
                }
                stage('Test On node-team2') {
                    agent { label "node-team2" }
                    steps { sh "echo I am good on node-team2" }
                }
                stage('Test On node-team3') {
                    agent { label "node-team3" }
                    steps { sh "echo I am good on node-team3" }
                }


这里会同时起三个不同的node，每个node都会被动态的创建到不同的namespace中，当pod状态为running的时候，开始有一些echo的输出，具体的动态创建过程可以从以下kubectl命令输出略知一二


georgehe@Sha-51664-Mbp  ~/Documents/george-sre/jenkins-master   master  k get pod -n node-team3
NAME               READY     STATUS              RESTARTS   AGE
node-team3-k206b   0/2       ContainerCreating   0          1m
 georgehe@Sha-51664-Mbp  ~/Documents/george-sre/jenkins-master   master  k get pod -n node-team1
NAME               READY     STATUS              RESTARTS   AGE
node-team1-p9cvw   0/2       ContainerCreating   0          1m
 georgehe@Sha-51664-Mbp  ~/Documents/george-sre/jenkins-master   master  k get pod -n node-team2
No resources found.
 georgehe@Sha-51664-Mbp  ~/Documents/george-sre/jenkins-master   master  k get pod -n node-team3
NAME               READY     STATUS              RESTARTS   AGE
node-team3-k206b   0/2       ContainerCreating   0          1m
 georgehe@Sha-51664-Mbp  ~/Documents/george-sre/jenkins-master   master  k get pod -n node-team1
NAME               READY     STATUS              RESTARTS   AGE
node-team1-p9cvw   0/2       ContainerCreating   0          1m
 georgehe@Sha-51664-Mbp  ~/Documents/george-sre/jenkins-master   master  k get pod -n node-team2
No resources found.
 georgehe@Sha-51664-Mbp  ~/Documents/george-sre/jenkins-master   master  k get pod -n node-team3
NAME               READY     STATUS        RESTARTS   AGE
node-team3-k206b   0/2       Terminating   0          1m
 georgehe@Sha-51664-Mbp  ~/Documents/george-sre/jenkins-master   master  k get pod -n node-team1
NAME               READY     STATUS        RESTARTS   AGE
node-team1-p9cvw   0/2       Terminating   0          1m
 georgehe@Sha-51664-Mbp  ~/Documents/george-sre/jenkins-master   master  k get pod -n node-team2
No resources found.
 georgehe@Sha-51664-Mbp  ~/Documents/george-sre/jenkins-master   master  k get pod -n node-team3
NAME               READY     STATUS    RESTARTS   AGE
node-team3-n4qt9   0/2       Pending   0          <invalid>


  • Jenkinsfile pipeline的结果


Jenkins k8s agent jenkins k8s agent pipeline_Jenkins k8s agent_02


pipeline

【测试任务】创建环境给node-team4使用

请先准备好新的namespace kubectl create ns node-team4

我们只需要提交PR到jenkins-config并merge到master分支即可，diff文件如下


diff --git a/Jenkinsfile b/Jenkinsfile
index 708f639..223f58e 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -42,6 +42,9 @@ pipeline {
                     docker.withRegistry('', "${REGISTRY_CREDENTIALS_ID}") {
                         docker.build("georgesre/jnlp-node-team3:${TAG_NAME}", "-f dockerfiles/node-team3/Dockerfile dockerfiles/node-team3").push()
                     }
+                    docker.withRegistry('', "${REGISTRY_CREDENTIALS_ID}") {
+                        docker.build("georgesre/jnlp-node-team4:${TAG_NAME}", "-f dockerfiles/node-team4/Dockerfile dockerfiles/node-team4").push()
+                    }
                 }
             }
         }
@@ -60,6 +63,10 @@ pipeline {
                     agent { label "node-team3" }
                     steps { sh "echo I am good on node-team3" }
                 }
+                stage('Test On node-team4') {
+                    agent { label "node-team4" }
+                    steps { sh "echo I am good on node-team4 && perl -v" }
+                }
             }
         }
     }
diff --git a/config/kubernetes.ctmpl b/config/kubernetes.ctmpl
index 518ae84..8adf831 100644
--- a/config/kubernetes.ctmpl
+++ b/config/kubernetes.ctmpl
@@ -34,6 +34,14 @@ kubernetes {
             rawYamlFilePath: '/var/jenkins_home/jenkins_config/config/node-team3.pod.yaml',
             exclusive: true,
             idleMinutes: 0
+        ],
+        [
+            name: 'node-team4',
+            nameSpace: 'node-team4',
+            label: 'node-team4',
+            rawYamlFilePath: '/var/jenkins_home/jenkins_config/config/node-team4.pod.yaml',
+            exclusive: true,
+            idleMinutes: 0
         ]

     ]
diff --git a/config/node-team4.pod.yaml.ctmpl b/config/node-team4.pod.yaml.ctmpl
new file mode 100644
index 0000000..bf976cf
--- /dev/null
+++ b/config/node-team4.pod.yaml.ctmpl
@@ -0,0 +1,49 @@
+metadata:
+  labels:
+    node-label: node-team4
+    class: KubernetesDeclarativeAgentTest
+spec:
+  imagePullSecrets:
+  - name: acrcred
+  containers:
+  - name: jnlp
+    env:
+    - name: TEAM_NAME
+      valueFrom:
+        fieldRef:
+          apiVersion: v1
+          fieldPath: metadata.namespace
+    - name: POD_IP
+      valueFrom:
+        fieldRef:
+          fieldPath: status.podIP
+    - name: DOCKER_HOST
+      value: tcp://localhost:2375
+    # envFrom:
+    # - configMapRef:
+    #     name: global-conf
+    # - secretRef:
+    #     name: node-team4.service-secrets
+    image: georgesre/jnlp-node-team4:latest
+    imagePullPolicy: Always
+    securityContext:
+      privileged: true
+    tty: true
+    # volumeMounts:
+    # - mountPath: /etc/service-secrets
+    #   name: service-secrets
+    #   readOnly: true
+  - name: dind
+    image: docker:dind
+    imagePullPolicy: Always
+    securityContext:
+      privileged: true
+    volumeMounts:
+      - name: dind-storage
+        mountPath: /var/lib/docker
+  volumes:
+  # - name: service-secrets
+  #   secret:
+  #     secretName: node-team4.service-secrets
+  - name: dind-storage
+    emptyDir: {}
diff --git a/dockerfiles/node-team4/Dockerfile b/dockerfiles/node-team4/Dockerfile
new file mode 100644
index 0000000..ee8b173
--- /dev/null
+++ b/dockerfiles/node-team4/Dockerfile
@@ -0,0 +1,13 @@
+FROM jenkins/jnlp-slave:3.27-1-alpine
+
+USER root
+RUN echo http://mirrors.aliyun.com/alpine/edge/community/ >> /etc/apk/repositories
+
+# 使用阿里镜像 https://mirrors.aliyun.com
+RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
+
+RUN apk update && 
+  apk add --upgrade docker sudo shadow git python py-pip curl libcurl nss perl && rm -rf /var/cache/apk/*
+RUN echo "jenkins ALL=NOPASSWD: ALL" >> /etc/sudoers
+RUN usermod -aG docker jenkins
+RUN usermod -aG root jenkins
diff --git a/dockerfiles/node-team4/README.md b/dockerfiles/node-team4/README.md
new file mode 100644
index 0000000..85f297b
--- /dev/null
+++ b/dockerfiles/node-team4/README.md
@@ -0,0 +1 @@
+# node-team4
diff --git a/jenkins_config.hcl b/jenkins_config.hcl
index f91faa9..2f7f30d 100644
--- a/jenkins_config.hcl
+++ b/jenkins_config.hcl
@@ -21,5 +21,10 @@
   "template": {
     "source": "/var/jenkins_home/jenkins_config/config/node-team3.pod.yaml.ctmpl",
     "destination": "/var/jenkins_home/jenkins_config/config/node-team3.pod.yaml"
+  },
+
+  "template": {
+    "source": "/var/jenkins_home/jenkins_config/config/node-team4.pod.yaml.ctmpl",
+    "destination": "/var/jenkins_home/jenkins_config/config/node-team4.pod.yaml"
   }
 }


提交到master后稍等片刻，就可以看到新的job被触发，继而到测试新node，新node上的测试结果如下


Jenkins k8s agent jenkins k8s agent pipeline_docker_09


测试结果

通知开发者jenkins环境已经准备完毕

开发者只需要在他们的repo中准备好Jenkinsfile文件，并设置好agent { label "<node-team-n>" }，然后按照05 GitHub Repo通过Jenkinsfile接入到Jenkins的说明接入到Jenkins中即可。

现在我们已经提供这些不同的环境给不同的开发团队了，我们甚至可以把这个jenkins-config repo推荐给开发团队他们自己，他们提交需求，SRE团队只需要review并approve即可。

参考

  • Configuring Certificates for Jenkins Kubernetes Plugin 0.12
  • kubernetes-plugin

更多

  • 回到Jenkins系列目录

云平台开发运维解决方案@george.sre
GitHub: https:// github.com/george-sre
简书: georgesre - 简书
欢迎交流～