k8s搭建gitlab gitlab-runner k8s

转载

mob64ca13fc5fb6 2024-05-09 17:25:18

文章标签 k8s搭建gitlab python docker linux git 文章分类 运维

在K8S中部署gilab-runner

文章目录

开始写yml之前的思考：
最终成品
注释版-StatefulSet(此版本只是参考作用)
daemonset
遇到的问题

问题一:脚本中注册runner的是什么方式进行注册的？
问题二：configmap中的变量从哪里可以看到？
问题四：RUNNER_NAME这个变量是自带的吗？
问题五：kill %1 是什么意思？
问题六：设置容忍度
问题七：gitlab-runner创建的容器需要设置容忍度

补充gitlab-ci-multi-runner命令解释

开始写yml之前的思考：

写一个gitlab-runner的yaml文件进行在K8S上运行runner
方案：
1、daemonset 每个node上运行一个runner
缺点： 如果集群中的node并不都是来做runner时 此路就不通了（注册了不用 也是可以的）
优点：yaml文件简单，操作步骤少
2、deployment
缺点：操作步骤较多，需要对node进行打标签以及选择标签的方式来选择机器进行部署runner
优点：灵活安排那些node进行运行runner，解决“集群中并不是所有node都是来进行运行runner的”情况

这两种方式都是需要进行配合configmap以及serct进行关键词以及配置文件的传入


借鉴博主思路：
1、configmap 来存储runner注册时的url地址以及限制pod的资源
2、configmap资源类型来定义一个脚本 “用于注册、运行和取消注册”
3、sercet来存储runner注册的token
4、rbac的一个授权
5、StatefulSet来运行runner并使用以上定义的configmap以及sercet等资源

最终成品

apiVersion: v1
kind: Namespace
metadata:
  name: gitlab-ci
---
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-ci-token
  namespace: gitlab-ci
  labels:
    app: gitlab-ci-runner
data:
  GITLAB_CI_TOKEN: WmF0Y3JNbUxUbmNncnpKem5CX1UK
---
apiVersion: v1
data:
  run.sh: |
    #!/bin/bash
    unregister() {
        kill %1
        echo "Unregistering runner ${RUNNER_NAME} ..."
        /usr/bin/gitlab-ci-multi-runner unregister -t "$(/usr/bin/gitlab-ci-multi-runner list 2>&1 | tail -n1 | awk '{print $4}' | cut -d'=' -f2)" -n ${RUNNER_NAME}
        exit $?
    }
    trap 'unregister' EXIT HUP INT QUIT PIPE TERM
    echo "Registering runner ${RUNNER_NAME} ..."
    /usr/bin/gitlab-ci-multi-runner register -r ${GITLAB_CI_TOKEN}
    sed -i 's/^concurrent.*/concurrent = '"${RUNNER_REQUEST_CONCURRENCY}"'/' /etc/gitlab-runner/config.toml
    echo "Starting runner ${RUNNER_NAME} ..."
    /usr/bin/gitlab-ci-multi-runner run -n ${RUNNER_NAME} &
    wait
kind: ConfigMap
metadata:
  labels:
    app: gitlab-ci-runner
  name: gitlab-ci-runner-scripts
  namespace: gitlab-ci
---
apiVersion: v1
data:
  KUBERNETES_NODE_TOLERATIONS: "node-role.kubernetes.io/master:NoSchedule"
  KUBERNETES_IMAGE: "python"
  RUNNER_TAG_LIST: "training"
  REGISTER_NON_INTERACTIVE: "true"
  REGISTER_LOCKED: "false"
  METRICS_SERVER: "0.0.0.0:9100"
  CI_SERVER_URL: "http://git.enflame.cn/"
  RUNNER_REQUEST_CONCURRENCY: "4"
  RUNNER_EXECUTOR: "kubernetes"
  KUBERNETES_NAMESPACE: "gitlab-ci"
  KUBERNETES_PRIVILEGED: "true"
  KUBERNETES_CPU_LIMIT: "1"
  KUBERNETES_CPU_REQUEST: "500m"
  KUBERNETES_MEMORY_LIMIT: "1Gi"
  KUBERNETES_SERVICE_CPU_LIMIT: "1"
  KUBERNETES_SERVICE_MEMORY_LIMIT: "1Gi"
  KUBERNETES_HELPER_CPU_LIMIT: "500m"
  KUBERNETES_HELPER_MEMORY_LIMIT: "100Mi"
  KUBERNETES_PULL_POLICY: "if-not-present"
  KUBERNETES_TERMINATIONGRACEPERIODSECONDS: "10"
  KUBERNETES_POLL_INTERVAL: "5"
  KUBERNETES_POLL_TIMEOUT: "360"
kind: ConfigMap
metadata:
  labels:
    app: gitlab-ci-runner
  name: gitlab-ci-runner-cm
  namespace: gitlab-ci
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-ci
  namespace: gitlab-ci
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-ci
  namespace: gitlab-ci
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-ci
  namespace: gitlab-ci
subjects:
  - kind: ServiceAccount
    name: gitlab-ci
    namespace: gitlab-ci
roleRef:
  kind: Role
  name: gitlab-ci
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: gitlab-ci-runner
  namespace: gitlab-ci
  labels:
    app: gitlab-ci-runner
spec:
  updateStrategy:
    type: RollingUpdate
  replicas: 1
  selector:
    matchLabels:
      app: gitlab-ci-runner
  serviceName: gitlab-ci-runner
  template:
    metadata:
      labels:
        app: gitlab-ci-runner
    spec:
      volumes:
      - name: gitlab-ci-runner-scripts
        projected:
          sources:
          - configMap:
              name: gitlab-ci-runner-scripts
              items:
              - key: run.sh
                path: run.sh
                mode: 0755
      serviceAccountName: gitlab-ci
      containers:
      - image: gitlab/gitlab-runner:alpine-v14.2.0
        name: gitlab-ci-runner
        command:
        - /scripts/run.sh
        envFrom:
        - configMapRef:
            name: gitlab-ci-runner-cm
        - secretRef:
            name: gitlab-ci-token
        env:
        - name: RUNNER_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        ports:
        - containerPort: 9100
          name: http-metrics
          protocol: TCP
        volumeMounts:
        - name: gitlab-ci-runner-scripts
          mountPath: "/scripts"
          readOnly: true
      restartPolicy: Always
      tolerations:
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"

注释版-StatefulSet(此版本只是参考作用)

#定义namespace
apiVersion: v1 #K8S的api接口版本
kind: Namespace #定义命名空间
metadata: #属性
  name: kube-ops #命名空间的名字
---
#定义注册的token
apiVersion: v1 #K8S的api接口版本
kind: Secret #资源类型
metadata: #属性
  name: gitlab-ci-token #secret的名字
  namespace: kube-ops #选择命名空间
  labels: #给资源打上标签
    app: gitlab-ci-runner #标签内容
data: #定义键值对数据 存储runner注册时的token
  GITLAB_CI_TOKEN: WURnUG14aUFBZFE0dFRrZ21RSkgK
---
#脚本用来注册和取消注册的命令，会在runner中进行执行
apiVersion: v1 #K8S的api接口版本
kind: ConfigMap #资源类型
data: #定义configmap中的内容 键值对方式定义
  run.sh: |
    #!/bin/bash
    unregister() {
        kill %1 #杀死后台运行程序为1的job
        echo "Unregistering runner ${RUNNER_NAME} ..."
        /usr/bin/gitlab-ci-multi-runner unregister -t "$(/usr/bin/gitlab-ci-multi-runner list 2>&1 | tail -n1 | awk '{print $4}' | cut -d'=' -f2)" -n ${RUNNER_NAME} #根据runner的token注销注册runner
        exit $?
    }
    trap 'unregister' EXIT HUP INT QUIT PIPE TERM #当接收到 关闭的信号后 进行取消注册runner 好处就是当pod删除时 gitlab上自动删除此runner，不会导致任务分配过来而不执行的情况。
    echo "Registering runner ${RUNNER_NAME} ..."
    /usr/bin/gitlab-ci-multi-runner register -r ${GITLAB_CI_TOKEN}
    sed -i 's/^concurrent.*/concurrent = '"${RUNNER_REQUEST_CONCURRENCY}"'/' /etc/gitlab-runner/config.toml
    echo "Starting runner ${RUNNER_NAME} ..."
    /usr/bin/gitlab-ci-multi-runner run -n ${RUNNER_NAME} & #进行注册runner
    wait
metadata: #属性
  labels: #标签
    app: gitlab-ci-runner #标签具体内容
  name: gitlab-ci-runner-scripts #configmap的名字
  namespace: kube-ops #选择命名空间
---
#定义runner运行时的环境变量(这些变量runner部分是借助/usr/bin/gitlab-ci-multi-runner进行使用的，查看有哪些变量 /usr/bin/gitlab-ci-multi-runner   register  --help)
apiVersion: v1
kind: ConfigMap
data:
  REGISTER_NON_INTERACTIVE: "true" #非交互式注册
  REGISTER_LOCKED: "false" #寄存器锁定
  METRICS_SERVER: "0.0.0.0:9100" #度量服务器
  CI_SERVER_URL: "http://10.0.0.6/" #gitlab的url 注册地址
  RUNNER_REQUEST_CONCURRENCY: "4" #请求并发数量
  RUNNER_EXECUTOR: "kubernetes" #runner的执行器类型
  KUBERNETES_NAMESPACE: "kube-ops" #指定命名空间
  KUBERNETES_PRIVILEGED: "true" #有特权的
  KUBERNETES_CPU_LIMIT: "1" #限制CPU资源为 1
  KUBERNETES_CPU_REQUEST: "500m"  #要求500m
  KUBERNETES_MEMORY_LIMIT: "1Gi" #内存限制为1G
  KUBERNETES_SERVICE_CPU_LIMIT: "1" #限制CPU为1
  KUBERNETES_SERVICE_MEMORY_LIMIT: "1Gi" #内存为1G
  KUBERNETES_HELPER_CPU_LIMIT: "500m" #辅助程序CPU限制
  KUBERNETES_HELPER_MEMORY_LIMIT: "100Mi" #
  KUBERNETES_PULL_POLICY: "if-not-present"
  KUBERNETES_TERMINATIONGRACEPERIODSECONDS: "10"
  KUBERNETES_POLL_INTERVAL: "5"
  KUBERNETES_POLL_TIMEOUT: "360"
metadata:
  labels:
    app: gitlab-ci-runner
  name: gitlab-ci-runner-cm
  namespace: kube-ops
---
#进行rbac授权
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-ci
  namespace: kube-ops
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-ci
  namespace: kube-ops
rules:
  - apiGroups: [""]
    resources: ["*"]
    verbs: ["*"]
---
apiVersion: v1
kind: RoleBinding
metadata:
  name: gitlab-ci
  namespace: kube-ops
subjects:
  - kind: ServiceAccount
    name: gitlab-ci
    namespace: kube-ops
roleRef:
  kind: Role
  name: gitlab-ci
  apiGroup: rbac.authorization.k8s.io

---
#运行runner
apiVersion: apps/v1 #api版本
kind: StatefulSet #资源类型
metadata: 元数据
  name: gitlab-ci-runner 指定资源的名称
  namespace: kube-ops 指定名称空间
  labels: 标签
    app: gitlab-ci-runner 标签具体内容
spec: 属性
  updateStrategy: #指定更新策略 
    type: RollingUpdate
  replicas: 1 指定副本数
  selector: 指定选择
    matchLabels:
      app: gitlab-ci-runner
  serviceName: gitlab-ci-runner
  template:
    metadata:
      labels:
        app: gitlab-ci-runner
    spec: 进行数据卷挂在
      volumes:
      - name: gitlab-ci-runner-scripts #使用脚本
        projected:  
          sources: 
          - configMap:
              name: gitlab-ci-runner-scripts #引用configmap中定义的那个脚本
              items: #引用key
              - key: run.sh
                path: run.sh
                mode: 0755
      serviceAccountName: gitlab-ci #指定serviceaccount 进行绑定
      containers: #容器信息
      - image: gitlab/gitlab-runner:alpine-v12.10.1 #指定容器运行时的镜像
        name: gitlab-ci-runner #容器名字
        command: #运行的命令
        - /scripts/run.sh
        envFrom:
        - configMapRef:
            name: gitlab-ci-runner-cm #引入环境变量
        - secretRef:
            name: gitlab-ci-token #引入token
        env: #环境变量
        - name: RUNNER_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name #runner-name的名称 pod的
        ports: #容器端口
        - containerPort: 9100
          name: http-metrics
          protocol: TCP
        volumeMounts:
        - name: gitlab-ci-runner-scripts
          mountPath: "/scripts"
          readOnly: true #只读的方式挂载
      restartPolicy: Always

daemonset

#定义runner工作的namespace
apiVersion: v1
kind: Namespace
metadata:
  name: kube-ops
---
#定义runner注册时用到的token
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-ci-token
  namespace: kube-ops
  labels:
    app: gitlab-ci-runner
data:
  GITLAB_CI_TOKEN: WURnUG14aUFBZFE0dFRrZ21RSkgK
---
#定义一个脚本（暂时不明白博主为何这么操作）
apiVersion: v1
data:
  run.sh: |
    #!/bin/bash
    unregister() {
        kill %1
        echo "Unregistering runner ${RUNNER_NAME} ..."
        /usr/bin/gitlab-ci-multi-runner unregister -t "$(/usr/bin/gitlab-ci-multi-runner list 2>&1 | tail -n1 | awk '{print $4}' | cut -d'=' -f2)" -n ${RUNNER_NAME}
        exit $?
    }
    trap 'unregister' EXIT HUP INT QUIT PIPE TERM
    echo "Registering runner ${RUNNER_NAME} ..."
    /usr/bin/gitlab-ci-multi-runner register -r ${GITLAB_CI_TOKEN}
    sed -i 's/^concurrent.*/concurrent = '"${RUNNER_REQUEST_CONCURRENCY}"'/' /home/gitlab-runner/.gitlab-runner/config.toml
    echo "Starting runner ${RUNNER_NAME} ..."
    /usr/bin/gitlab-ci-multi-runner run -n ${RUNNER_NAME} &
    wait
kind: ConfigMap
metadata:
  labels:
    app: gitlab-ci-runner
  name: gitlab-ci-runner-scripts
  namespace: kube-ops
---
#定义runner运行时的环境变量
apiVersion: v1
data:
  REGISTER_NON_INTERACTIVE: "true"
  REGISTER_LOCKED: "false"
  METRICS_SERVER: "0.0.0.0:9100"
  CI_SERVER_URL: "http://10.0.0.6/"
  RUNNER_REQUEST_CONCURRENCY: "4"
  RUNNER_EXECUTOR: "kubernetes"
  KUBERNETES_NAMESPACE: "kube-ops"
  KUBERNETES_PRIVILEGED: "true"
  KUBERNETES_CPU_LIMIT: "1"
  KUBERNETES_CPU_REQUEST: "500m"
  KUBERNETES_MEMORY_LIMIT: "1Gi"
  KUBERNETES_SERVICE_CPU_LIMIT: "1"
  KUBERNETES_SERVICE_MEMORY_LIMIT: "1Gi"
  KUBERNETES_HELPER_CPU_LIMIT: "500m"
  KUBERNETES_HELPER_MEMORY_LIMIT: "100Mi"
  KUBERNETES_PULL_POLICY: "if-not-present"
  KUBERNETES_TERMINATIONGRACEPERIODSECONDS: "10"
  KUBERNETES_POLL_INTERVAL: "5"
  KUBERNETES_POLL_TIMEOUT: "360"
kind: ConfigMap
metadata:
  labels:
    app: gitlab-ci-runner
  name: gitlab-ci-runner-cm
  namespace: kube-ops
---
#定义RBAC授权
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-ci
  namespace: kube-ops
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-ci
  namespace: kube-ops
rules:
  - apiGroups: [""]
    resources: ["*"]
    verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-ci
  namespace: kube-ops
subjects:
  - kind: ServiceAccount
    name: gitlab-ci
    namespace: kube-ops
roleRef:
  kind: Role
  name: gitlab-ci
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: gitlab-ci-runner
  namespace: kube-ops
  labels:
    app: gitlab-ci-runner
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: gitlab-ci-runner
  template:
    metadata:
      labels:
        app: gitlab-ci-runner
    spec:
      volumes:
      - name: gitlab-ci-runner-scripts
        projected:
          sources:
          - configMap:
              name: gitlab-ci-runner-scripts
              items:
              - key: run.sh
                path: run.sh
                mode: 0755
      serviceAccountName: gitlab-ci
      containers:
      - image: gitlab/gitlab-runner:alpine-v12.10.1
        name: gitlab-ci-runner
        command:
        - /scripts/run.sh
        envFrom:
        - configMapRef:
            name: gitlab-ci-runner-cm
        - secretRef:
            name: gitlab-ci-token
        env:
        - name: RUNNER_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        ports:
        - containerPort: 9100
          name: http-metrics
          protocol: TCP
        volumeMounts:
        - name: gitlab-ci-runner-scripts
          mountPath: "/scripts"
          readOnly: true
      restartPolicy: Always

遇到的问题

问题一:脚本中注册runner的是什么方式进行注册的？

问题描述：当我尝试以手动的方式执行脚本的内容时，发现脚本中用到了runner的token 并不是gitlab-server的注册runner时的token从而产生了疑惑：难道注册gitlab-runner还能用自己的token（gitlab-runner的token）

思考：应该不是使用token的方式因为命令行方式获取得到的token和gitlab-ci获取的token值是不一样的

k8s搭建gitlab gitlab-runner k8s_git

解决：

此yaml中 使用runner的token进行的注销操作
注册runner是根据Secret中的token进行注册的（也就是gitlab-server中的token）
提问问题时没搞明白脚本的具体含义，脚本中是 借助gitlab-runner的token来进行注销runner的一个操作 而注册的操作还是使用gitlab-server提供的token

问题二：configmap中的变量从哪里可以看到？

在configmap中gitlab-ci-runner-cm的其他变量怎么没看到使用？

这些变量是借助/usr/bin/gitlab-ci-multi-runner进行使用的，自带的内置变量参数
查看有哪些变量 /usr/bin/gitlab-ci-multi-runner   register  --help

问题四：RUNNER_NAME这个变量是自带的吗？

根据K8S中的pod自带获取名称方式获取的
metadata.name

问题五：kill %1 是什么意思？

kill %1这条命令表示杀死一个后台程序，这个后台程序的“工作号码（jobnumber）”是1号。job命令的作用就是将放在后台运行的程序（& nohup等方式运行的进程）给显示出来，kill %1 则是将后台程序的job号码为1的给杀掉。

问题六：设置容忍度

问题描述：将K8S自动设置的系统污点误认为是人为设置的，导致找不到对应的key:value值，陷入不知道该如何设置容忍度的问题

正常情况下 kubectl describe node work1 |grep Taint 命令是可以看到人为设置的key:value的污点的。

我误认为磁盘故障而导致K8S系统自动给node节点打的污点是人为设置的污点，导致使用kubectl describe node work1|grep Taint 而没有找到key:value，从而陷入不知道该如何设置容忍的困境

解决：如果是人为设置的污点，kubectl describe node work1 其实是可以看到设置的污点的key:value的

work1因为磁盘的故障而自动打污点

Warning  Evicted    1s    kubelet            The node had condition: [DiskPressure].

问题七：gitlab-runner创建的容器需要设置容忍度

问题描述：gitlab-runner在接收到任务时自动创建执行具体内容的容器无法容忍污点一直处于pending的状态

因为在yml文件中可以设置gitlab-runner的容忍，但是不清楚gitlab-runner运行后自动创建的容器是怎么设置污点容忍的。

解决：

须知：①gitlab-runner创建的容器是从configmap中定义的环境变量中来获取信息的；②configmap中设置的gitlab-runner的环境变量获取方式为:/usr/bin/gitlab-ci-multi-runner register --help|grep kubernetes-node-tolerations

root@c75cacba4598:/# /usr/bin/gitlab-ci-multi-runner  register --help|grep kubernetes-node-tolerations
Runtime platform                                    arch=amd64 os=linux pid=50 revision=8925d9a0 version=14.1.0
   --kubernetes-node-tolerations value                                         A toml table/json object of key=value:effect. Value and effect are expected to be strings. When set, pods will tolerate the given taints. Only one toleration is supported through environment variable configuration. (default: "{}") [$KUBERNETES_NODE_TOLERATIONS]

具体解决方法：在yml中的configmap中设置 gitlab-runner创建容器时的特性

KUBERNETES_NODE_TOLERATIONS: "node-role.kubernetes.io/master:NoSchedule"

补充gitlab-ci-multi-runner命令解释

NAME:
   gitlab-ci-multi-runner - a GitLab Runner

USAGE:
   gitlab-ci-multi-runner [global options] command [command options] [arguments...]

VERSION:
   14.1.0 (8925d9a0)

AUTHOR:
   GitLab Inc. <support@gitlab.com>

COMMANDS:
     exec                  在本地执行构建
     list                  列出所有配置的运行程序
     run                   运行多运行程序服务
     register              登记新的runner
     install               安装服务
     uninstall             卸载服务
     start                 开始服务
     stop                  停止服务
     restart               重启服务
     status                获取服务状态
     run-single            启动单线程runner
     unregister            取消注册特定的运行程序
     verify                验证所有已注册的runner
     artifacts-downloader  下载并提取构建工件（内部）
     artifacts-uploader    创建和上载构建工件（内部）
     cache-archiver        创建和上载缓存工件（内部）
     cache-extractor       下载并提取缓存工件（内部）
     cache-init            已更改缓存路径的权限（内部）
     health-check          检查特定地址的运行状况
     read-logs             从kubernetes executor（内部）使用的文件读取作业日志
     help, h               Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --cpuprofile value           write cpu profile to file [$CPU_PROFILE]
   --debug                      debug mode [$DEBUG]
   --log-format value           Choose log format (options: runner, text, json) [$LOG_FORMAT]
   --log-level value, -l value  Log level (options: debug, info, warn, error, fatal, panic) [$LOG_LEVEL]
   --help, -h                   show help
   --version, -v                print the version

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。