11、集群外部、内部的jenkins如何在k8s集群上创建动态pod agent

Kubernetes插件

能够让Jenkins Master在Kubernetes集群上运行基于Pod的动态Agent
   ◼ 它会为启动的每个Agent创建一个Pod，并在运行完成后停止它
   ◼ 各Pod Agent以Inbound Agent形式运行，inbound-agent容器会自动连接到Jenkins Master
      ◆这意味着在每个Pod Agent中，始终有一个自动配置的名为jnlp的特殊容器
      ◆要运行的其它容器可由用户在Pod模板上自行按需添加和配置
      ◆默认情况下，命令都会在jnlp容器中执行
   ◼ inbound-agent容器会被注入一些以下几个环境变量，以便它了解Jenkins Master的信息
      ◆JENKINS_URL
      ◆JENKINS_SECRET
      ◆JENKINS_AGENT_NAME
      ◆JENKINS_NAME

该功能依赖的条件
   ◼ Kubernetes v1.14及以上版本
   ◼ 在Jenkins上安装了Kubernetes插件
   ◼ 在Kubernetes上为Jenkins配置了具有足够权限的ServiceAccount

在k8s集群上部署jenkins所需要的文件，以下文件在deploy目录下

使用了单独的名称空间
cat 01-namespace-jenkins.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: jenkins

作为jenkins的持久存储
cat 02-pvc-jenkins.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc
  namespace: jenkins
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: nfs-csi

定义SA叫Jenkins-master，授予了集群级别的一些权限
cat 03-rbac-jenkins.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-master
  namespace: jenkins

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jenkins-master
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins-master
roleRef:
  kind: ClusterRole
  name: jenkins-master
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: jenkins-master
  namespace: jenkins

部署jenkins，使用Jenkins的指定镜像文件、监听端口
cat 04-deploy-jenkins.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: jenkins 
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      serviceAccountName: jenkins-master
      volumes:
      - name: jenkins-store
        persistentVolumeClaim:
          claimName: jenkins-pvc
      containers:
      - name: jenkins
        image: jenkins/jenkins:jdk11
        #image: jenkins/jenkins:lts-jdk11
        volumeMounts:
        - name: jenkins-store
          mountPath: /var/jenkins_home/
        imagePullPolicy: IfNotPresent
        env:
        - name: JAVA_OPTS
          value: -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai -Djenkins.install.runSetupWizard=true
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        - containerPort: 50000
          name: agent
          protocol: TCP

使用jenkins的service开放两个端口8080、50000
cat 05-service-jenkins.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: jenkins 
  labels:
    app: jenkins
spec:
  selector:
    app: jenkins
  type: NodePort
  ports:
  - name: http
    port: 8080
    targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins-jnlp
  namespace: jenkins 
  labels:
    app: jenkins
spec:
  selector:
    app: jenkins
  ports:
  - name: agent
    port: 50000
    targetPort: 50000

基于动态pod做maven构建，对maven而言，每次创建pod，都需要下载maven模块，比较消耗时间，效率不高，定义pvc，专门存放maven的缓存，共享给多个pod使用
cat 06-pvc-maven-cache.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-maven-cache
  namespace: jenkins
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: nfs-csi
Footer

使用ingress controller把jenkins开放出去
cat 07-ingress-jenkins.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins
  namespace: jenkins
spec:
  ingressClassName: nginx
  rules:
  - host: cicd.magedu.com
    http:
      paths:
      - backend:
          service:
            name: jenkins
            port: 
              number: 8080 
        path: /
        pathType: Prefix
  - host: jenkins.magedu.com
    http:
      paths:
      - backend:
          service:
            name: jenkins
            port: 
              number: 8080 
        path: /
        pathType: Prefix

把deploy目录下的文件进行部署

kubectl apply -f deploy/

查看状态

使用172.29.6.200打开web服务，打开前对其进行域名解析

vim /etc/hosts
172.29.6.200 cicd.magedu.com

基于pod获取解锁密钥（在日志里面会有解锁密钥信息输出）

选择安装推荐的插件

在jenkins上安装kubernetes插件

如果jenkins在k8s集群外部，配置如下（选择kubernetes）

kubernetes地址kubeapi.magedu.com，为k8s集群的对外访问地址，以次来进行访问集群

在jenkins上对此地址进行解析

vim /etc/hosts
172.29.6.1 kubeapi.magedu.com

获取集群的证书，把证书粘贴到配置中

cd /etc/kubernetes/pki
cat ca.crt

将Service Account的Token保存为Jenkins上的认证凭据

创建新凭据

确保pod能解析jenkins.magedu.com:50000

(没有DNS服务器，需要给每个pod加上host记录)

查看coredns服务
kubectl get cm -n kube-system
coredns
把该文件原有配置取出来，做修改
kubectl get cm coredns -o yaml -n kube-system
vim configmap-coredns.yaml

kubectl apply -f configmap-coredns.yaml

添加pod模板

添加卷，做持久化保存(创建的任何一个pod下载的maven的缓存都放在卷上，任何pod只要基于该模板创建的，都能加载共享的内容)

在k8s上查看pvc
kubectl get pvc -n jenkins
pvc-maven-cache  Bound

定义pod label，表明此jenkins为外部的jemkins，不是k8s集群部署的jenkins

执行作业，限制到k8s的节点上去构建

集群内部的jenkins在k8s集群上创建动态的pod agent

安装kubernetes插件

此时jenkins本身就是以pod形式在运行，能直接认证到k8s的api server上，不需要做认证

其pod被赋予了SA在运行，有token，绑定在serviceAccountName上。

其token信息在/var/run/secrets/kubernetes.io/serviceaccount

kubectl get pods -n jenkins
kubectl get pods jenkins-d454fc966-kxzqp -o yaml -n jenkins

set up clund

jenkins地址与jenkins通道使用集群内部地址

添加pod label

添加pod template

添加卷

执行作业，限制到k8s的节点上去构建