jenkins部署k8s 应用 jenkins整合k8s

K8S部署jenkins

1、环境配置

• NFS服务器：1xx.xxx.xxx.229
• K8Smaster：1xx.xxx.xxx.221
• k8Snode1： 1xx.xxx.xxx.228 （宿主机）

2、安装NFS

• 需要将jenkins的配置目录挂载到NFS中，如果没安装可以参考之前的文档。

3、编写jenkins yaml文件

• 注意：jenkins2.372默认安装jdk11版本，没有mvn和jdk8，如果有需求就需要将宿主机的mvn、jdk、环境变量、都挂载到容器内。
• jenkins的版本需要固定，否则当容器重启后，jenkins版本会变。

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      name: jenkins 
  template:
    metadata:
      name: jenkins
      labels:
        name: jenkins
    spec:
      serviceAccountName: jenkins
      containers:
        - name: jenkins
          image: jenkins/jenkins:2.372
          ports:
            - containerPort: 8080
            - containerPort: 50000
          resources:
            limits:
              cpu: 1.5
              memory: 2Gi
            requests:
              cpu: 1
              memory: 1Gi
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home

            - name: maven
              mountPath: /usr/local/maven/apache-maven-3.8.5

            - name: jdk
              mountPath: /usr/local/jdk/jdk1.8.0_201

            - name: system
              mountPath: /etc/profile

            - name: warehouse
              mountPath: /root/.m2

      securityContext:
              runAsUser: 0
      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: jenkins
          
      - name: maven
        hostPath:
          path: /usr/local/maven/apache-maven-3.8.5
          # 宿主机的maven包地址

      - name: jdk
        hostPath:
          path: /usr/local/jdk/jdk1.8.0_201
          # 宿主机的java包地址

      - name: system
        hostPath:
          path: /etc/profile
          # 宿主机的环境变量

      - name: warehouse
        hostPath:
          path: /root/.m2
          # 宿主机的mvn本地仓库

---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
spec:
  selector:
    name: jenkins
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 8080
      protocol: TCP
      nodePort: 30008
    - name: agent
      port: 50000
      protocol: TCP

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins
spec:
  storageClassName: "managed-nfs-storage"
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins
rules:
- apiGroups: [""]
  resources: ["pods","events"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets","events"]
  verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins

kubectl apply -f jenkins.yaml

4、验证mvn、jdk、仓库、环境变量是否都挂载到容器内

[root@k8s-master1 jenkins]# kubectl exec -it jenkins-8654c79996-6bxff -- bash

[root@k8s-master1 jenkins]# source /etc/profile

# 1、检测容器内是否可以上网
root@jenkins-8654c79996-6bxff:~# curl www.baidu.com
#如果能访问到百度代表可以连上网，如果不行，将8.8.8.8加到resolv.conf，在重新试下
root@jenkins-8654c79996-6bxff:~# echo "nameserver 8.8.8.8" >> /etc/resolv.conf

5、更改jenkins插件地址

# 找到nfs服务器，进入挂载目录
[root@k8s-node2 kubernetes]# ll
drwxrwxrwx 16 root   root   4096 Oct 25 17:56 default-jenkins-pvc-eba3b68b-1891-4521-becd-8f0fa6936e9a
drwxrwxrwx  3 root   root   4096 Sep 21 14:11 monitor-grafana-pvc-0a3154e7-e28d-4c5f-b6d7-19829997f209
drwxrwxrwx 27 nobody nobody 4096 Oct 26 09:00 monitor-prometheus-pvc-ee8a8670-d657-47ce-b2f4-1905e1921d76

[root@k8s-node2 default-jenkins-pvc-eba3b68b-1891-4521-becd-8f0fa6936e9a]# cat hudson.model.UpdateCenter.xml 
# 找到url那一行，替换成阿里云插件地址
<?xml version='1.1' encoding='UTF-8'?>
<sites>
  <site>
    <id>default</id>
    <url> https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json</url>
  </site>
</sites>

6、安装jenkins

[root@k8s-master1 jenkins]# kubectl get pod,svc -o wide
# 找到pod容器分配在哪台机器和service访问外网的端口。然后在浏览器上访问
NAME                                          READY   STATUS    RESTARTS        AGE   IP           NODE        NOMINATED NODE   READINESS GATES
pod/jenkins-8654c79996-6bxff                  1/1     Running   0               17h   172.17.0.3   k8s-node1   <none>           <none>
pod/nfs-client-provisioner-6b97c8fd96-rz7x9   1/1     Running   5 (5d19h ago)   34d   172.17.0.2   k8s-node1   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                        AGE     SELECTOR
service/jenkins      NodePort    10.0.0.121   <none>        80:30008/TCP,50000:30705/TCP   5d19h   name=jenkins
service/kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP                        35d     <none>

---

在jenkins首页找到系统配置-----> 插件管理

• Git
• Git Parameter：Git参数化构建
• gogs
• pipeline：流水线
• Extended Choice Parameter：扩展选择框参数，支持多选

注意：要拉取gogs仓库的代码，需提前在gogs仓库中创建钩子。

7、制作pipeline

1、在全局配置中配置jdk和mvn地址

2、配置mvn和jdk的环境变量

3、创建pipeline，实现自动打包和编译

注意：pipeline拉取代码是在jenkins中自己生成的

pipeline{
    agent any
    stages {
        stage('1、拉取代码') {
            steps {
                checkout([$class: 'GitSCM', branches: [[name: '*/master']], browser: [$class: 'GogsGit', repoUrl: ''], extensions: [], userRemoteConfigs: [[credentialsId: '73f9c4f6-9224-450c-a07c-f3a7191ddbba', url: 'http://代码仓库地址:端口/xxx/xxx-group-buy.git']]])
            }
        }
        stage('2、打包构建') {
            steps {
                sh """
                ls
                mvn clean package
                """
            }
        }
    }
}

---

jenkins安装遇到的错误

错误1：jenkins安装时显示以离线

• 解决方案1：复制jenkins地址在后面加上/pluginManager/advanced，找到Update Site或者升级站点。将地址替换掉以下地址，在尝试重启jenkins
• https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json
• http://updates.jenkins-ci.org/update-center.json
• 解决方案2：在容器内新增dns 8.8.8.8、114.114.114.114，在尝试重启jenkins