在k8s中安装jenkins并实现动态生成 jenkins slave

原创

dongyali521521 2019-09-04 11:34:22 ©著作权

©著作权归作者所有：来自51CTO博客作者dongyali521521的原创作品，请联系作者获取转载授权，否则将追究法律责任

安装jenkins 1、创建一个命名空间 $ kubectl create namespace kube-ops 2、为jenkins创建pvc（也可以使用存储类创建）

apiVersion: v1
kind: PersistentVolume
metadata:
  name: opspv
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Delete
  nfs:
    server: 192.168.1.244
    path: /data/k8s
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: opspvc
  namespace: kube-ops
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi

3、创建jenkins需要的rbac权限

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins2
  namespace: kube-ops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins2
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jenkins2
  namespace: kube-ops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins2
subjects:
  - kind: ServiceAccount
    name: jenkins2
    namespace: kube-ops

也可以为ServiceAccoun绑定一个系统现有的 cluster-admin 集群角色权限 4、创建jenkins pod $ docker pull docker.io/jenkins/jenkins:lts $ docker pull cnych/jenkins:jnlp6

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins2
  namespace: kube-ops
spec:
  template:
    metadata:
      labels:
        app: jenkins2
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccount: jenkins2
      containers:
      - name: jenkins
        image: docker.io/jenkins/jenkins:lts
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        - containerPort: 50000
          name: agent
          protocol: TCP
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        volumeMounts:
        - name: jenkinshome
          subPath: jenkins2
          mountPath: /var/jenkins_home
        env:
        - name: LIMITS_MEMORY
          valueFrom:
            resourceFieldRef:
              resource: limits.memory
              divisor: 1Mi
        - name: JAVA_OPTS
          value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvision
er.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai      
      securityContext:
        fsGroup: 1000
      volumes:
      - name: jenkinshome
        persistentVolumeClaim:
          claimName: opspvc
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins2
  namespace: kube-ops
  labels:
    app: jenkins2
spec:
  selector:
    app: jenkins2
  type: NodePort
  ports:
  - name: web
    port: 8080
    targetPort: web
    nodePort: 30003
  - name: agent
    port: 50000
    targetPort: agent

$ kubectl apply -f jenkins2.yaml $ kubectl get pod -n kube-ops jenkins2-76644dbc9b-llcsp 0/1 Running 0 #不能正常启动 $ kubectl describe pod jenkins2-76644dbc9b-llcsp -n kube-ops $ kubectl logs -f jenkins2-76644dbc9b-llcsp -n kube-ops 5、在nfs服务器上修改jenkins持久目录的权限并重新创建jenkins pod $ chown -R 1000 /data/k8s/jenkins2 #在192.168.1.244上 $ kubectl delete -f jenkins2.yaml $ kubectl apply -f jenkins2.yaml $ kubectl get pod -n kube-ops jenkins2-76644dbc9b-llcsp 1/1 Running 0
为什么是1000？上述镜像的Dockerfile文件中定义的是：user=jenkins group=jenkins uid=1000 gid=1000 Dockerfile文件的地址： https://github.com/jenkinsci/docker/blob/master/Dockerfile ARG user=jenkins ARG group=jenkins ARG uid=1000 ARG gid=1000 ARG http_port=8080 ARG agent_port=50000 ARG JENKINS_HOME=/var/jenkins_home $ kubectl get svc -n kube-ops jenkins2 NodePort 10.105.121.176 <none> 8080:30003/TCP,50000:30936/TCP http://192.168.1.243:30003 初始密码在nfs服务器上 $ cat /data/k8s/jenkins2/secrets/initialAdminPassword

在jenkins上创建kubernetes云 enkins Master 和 Jenkins Slave 以 Pod 形式运行在 Kubernetes 集群的 Node 上，Master 运行在其中一个节点，并且将其配置数据存储到一个 Volume 上去，Slave 运行在各个节点上，并且它不是一直处于运行状态，它会按照需求动态的创建并自动删除这种方式的工作流程大致为：当 Jenkins Master 接受到 Build 请求时，会根据配置的 Label 动态创建一个运行在 Pod 中的 Jenkins Slave 并注册到 Master 上，当运行完 Job 后，这个 Slave 会被注销并且这个 Pod 也会自动删除，恢复到最初状态。 1、安装插件安装kubernetes plugin，点击 Manage Jenkins -> Manage Plugins -> Available -> Kubernetes plugin 2、增加kubernetes云点击 Manage Jenkins —> Configure System —> (拖到最下方)Add a new cloud —> 选择 Kubernetes，然后填写 Kubernetes 和 Jenkins 配置信息----连接测试 name：kubernetes Kubernetes 地址：https://kubernetes.default.svc.cluster.local Kubernetes 命名空间：kube-ops Jenkins 地址：http://jenkins2.kube-ops.svc.cluster.local:8080（jenkins2是svc） 3、添加pod模板添加pod模板----Kubernetes Pod Template 名称：jnlp 命名空间：kube-ops 标签列表：dongyali-jnlp 4、添加容器模板添加容器----Container Template 名称：jnlp Docker 镜像：cnych/jenkins:jnlp6（Jenkins 版本在 2.176.x以下的镜像名字去掉6）工作目录：/home/jenkins/agent 运行的命令：清空命令参数：清空 5、添加两个卷添加卷----Host Path Volume 主机路径：/var/run/docker.sock 挂载路径：/var/run/docker.sock 主机路径：/root/.kube 挂载路径：/root/.kube 6、可能需要配置ServiceAccount $ kubectl get sa -n kube-ops jenkins2 1 14h 点击添加卷下面的高级----Service Account----jenkins2 7、用shell测试 Kubernetes 动态生成 jenkins slave 新建任务----名字----自由风格通用----勾选限制项目的运行节点----标签表达式：dongyali-jnlp 构建----执行shell----输入如下内容----保存立即构建

echo "测试 Kubernetes 动态生成 jenkins slave"
echo "==============docker in docker==========="
docker info
echo "=============kubectl============="
kubectl get pods

观察 Kubernetes 集群中 Pod 的变化： $ kubectl get pod -n kube-ops jenkins2-76644dbc9b-llcsp 1/1 Running 0 3h49m jnlp-tl1km 1/1 Running 0 44s 当任务运行完毕，jnlp这个slave pod就会自动消失。 8、用pipeline测试 Kubernetes 动态生成 jenkins slave 新建任务----名字----流水线在流水线脚本中输入如下内容----保存立即构建

node('dongyali-jnlp') {
    stage('Clone') {
      echo "1.Clone Stage"
    }
    stage('Test') {
      echo "2.Test Stage"
    }
    stage('Build') {
      echo "3.Build Stage"
    }
    stage('Deploy') {
      echo "4. Deploy Stage"
    }
}