k8s部署持续集成环境

原创

草根追逐 2020-03-05 11:39:27 博主文章分类：容器编排|虚拟化 ©著作权

文章标签 k8s kubernetes 持续集成 git jenkins 文章分类 虚拟化云计算

©著作权归作者所有：来自51CTO博客作者草根追逐的原创作品，请联系作者获取转载授权，否则将追究法律责任

k8s部署持续集成环境

安装前的准备：关闭防火墙关闭selinux 安装docker-ce 1、安装harbor参考前面笔记 2、安装并配置git，参考前面笔记 3、在git这台部署nfs服务器，并且在各个节点安装nfs-utils客户端 4、创建nfs-client-provisioner客户端 cat class.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: managed-nfs-storage provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME' parameters: archiveOnDelete: "true"

cat deployment.yaml apiVersion: v1 kind: ServiceAccount metadata: name: nfs-client-provisioner

kind: Deployment apiVersion: extensions/v1beta1 metadata: name: nfs-client-provisioner spec: replicas: 1 strategy: type: Recreate template: metadata: labels: app: nfs-client-provisioner spec: serviceAccountName: nfs-client-provisioner containers: - name: nfs-client-provisioner image: lizhenliang/nfs-client-provisioner:latest volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: fuseim.pri/ifs - name: NFS_SERVER value: 192.168.31.64 - name: NFS_PATH value: /ifs/kubernetes volumes: - name: nfs-client-root nfs: server: 192.168.31.64 path: /ifs/kubernetes cat rbac.yaml kind: ServiceAccount apiVersion: v1 metadata: name: nfs-client-provisioner

kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: nfs-client-provisioner-runner rules:

apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"]
apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"]
apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"]
apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"]

kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: run-nfs-client-provisioner subjects:

kind: ServiceAccount name: nfs-client-provisioner namespace: default roleRef: kind: ClusterRole name: nfs-client-provisioner-runner apiGroup: rbac.authorization.k8s.io

kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner rules:

apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"]

kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner subjects:

kind: ServiceAccount name: nfs-client-provisioner
replace with namespace where provisioner is deployed
namespace: default roleRef: kind: Role name: leader-locking-nfs-client-provisioner apiGroup: rbac.authorization.k8s.io 5、部署jenkins服务器，前提是需要部署k8s的core-dns（安装coredns查看前面笔记），否则没法做解析就没法安装插件 cat ingress.yml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: jenkins annotations: nginx.ingress.kubernetes.io/ssl-redirect: "true" kubernetes.io/tls-acme: "true"
如果上传插件超出默认会报"413 Request Entity Too Large", 增加 client_max_body_size
nginx.ingress.kubernetes.io/proxy-body-size: 50m nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
nginx-ingress controller版本小于 0.9.0.beta-18 的配置
ingress.kubernetes.io/ssl-redirect: "true" ingress.kubernetes.io/proxy-body-size: 50m ingress.kubernetes.io/proxy-request-buffering: "off" spec: rules:
host: jenkins.example.com http: paths:
- path: / backend: serviceName: jenkins servicePort: 80 cat rbac.yml

创建名为jenkins的ServiceAccount

apiVersion: v1 kind: ServiceAccount metadata: name: jenkins

创建名为jenkins的Role，授予允许管理API组的资源Pod

kind: Role apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: jenkins rules:

apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update","watch"]
apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"]
apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"]
apiGroups: [""] resources: ["secrets"] verbs: ["get"]

将名为jenkins的Role绑定到名为jenkins的ServiceAccount

apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: jenkins roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: jenkins subjects:

kind: ServiceAccount name: jenkins

cat service.yml apiVersion: v1 kind: Service metadata: name: jenkins spec: selector: name: jenkins type: NodePort ports: - name: http port: 80 targetPort: 8080 protocol: TCP nodePort: 30006 - name: agent port: 50000 protocol: TCP

cat statefulset.yml apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: jenkins labels: name: jenkins spec: serviceName: jenkins replicas: 1 updateStrategy: type: RollingUpdate template: metadata: name: jenkins labels: name: jenkins spec: terminationGracePeriodSeconds: 10 serviceAccountName: jenkins containers: (如果出现dns没法解析的情况，需要在container这一行上面加一行dnsPolicy: Default) - name: jenkins image: jenkins/jenkins imagePullPolicy: Always ports: - containerPort: 8080 - containerPort: 50000 resources: limits: cpu: 1 memory: 1Gi requests: cpu: 0.5 memory: 500Mi env: - name: LIMITS_MEMORY valueFrom: resourceFieldRef: resource: limits.memory divisor: 1Mi - name: JAVA_OPTS value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 volumeMounts: - name: jenkins-home mountPath: /var/jenkins_home livenessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 readinessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 securityContext: fsGroup: 1000 volumeClaimTemplates:

metadata: name: jenkins-home spec: storageClassName: "managed-nfs-storage" accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi 6、使用这几个yaml文件启动安装jenkins PS：当提示以下出错时 'FailedCreate' create Pod jenkins-0 in StatefulSet jenkins failed error: pods "jenkins-0" is forbidden: pod.Spec.SecurityContext.FSGroup is forbidden 修改/opt/kubernetes/cfg/kube-apiserver，删除里面的安全字段SecurityContext 7、使用浏览器访问http://10.1.2.190:30006进行安装(使用kubectl get svc -o wide查看jenkins当前在190这个pod运行) 8、安装时不选择任何插件，使用手动安装插件，安装git和kubernetes插件