K8S 常见的资源对象

一、configmap

configmap 用于保存非机密性的配置，数据可以用 key/value 键值对的形式保存，也可通过文件的形式保存。

1.1 样例 nginx configmap

以 nginx 为例，创建 nginx 的配置文件。

cat >> nginx-default.conf << EOF
server {
    listen 443 ssl http2;
    server_name test.belkuy.top;
    ssl_certificate ssl/test.belkuy.top.crt;
    ssl_certificate_key ssl/test.belkuy.top.key;
    ssl_session_cache   shared:SSL:50m;
    ssl_session_timeout 15m;
    ssl_ciphers EECDH+CHACHA20:EECDH+CHACHA20-draft:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256::!MD5;
    ssl_protocols TLSv1.1 TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;

    access_log  /var/log/nginx/host.access.log  main;

    #禁止指定UA及UA为空的访问
    if ($http_user_agent ~ "Scrapy|FeedDemon|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Feedly|UniversalFeedParser|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|lightDeckReports Bot|YYSpider|DigExt|YisouSpider|MJ12bot|heritrix|EasouSpider|LinkpadBot|Ezooms|^$" )
    { return 404; }
 
    location ^~ /shell { return 404; }
    
    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    location ~ /\.ht {
        deny  all;
    }
}
EOF

kubectl create configmap nginx-default --from-file=./nginx-default.conf -n test2023

二、secret

secret 用于保存密码、token、秘钥等敏感数据的配置，通过 volume 挂载或环境变量的方式，避免把这些敏感数据直接暴露在镜像或者 pod 的 spec 中。

2.1 secret 三种参数

• generic：通用类型，通常用于存储密码数据
• tls：用于存储私钥和证书
• docker-registry：保存docker 镜像仓库的认证信息

2.3 secret 类型

• service account：用于被 serviceaccount 引用。serviceaccount 创建时，Kubernetes 会默认创建对应的 secret。pod 如果使用了 serviceaccount，对应的 secret 会自动挂载到 pod 的 /run/secrets/kubernetes.io/serviceaccount 目录中。
• Opaque：base64 编码格式的 secret ，用来存储密码、秘钥等。可以通过 base64 --decode 解码获得原始数据，因此安全性弱。
• kubernets.io/dockerconfigjson：用来存储私有 docker registry 的认证信息。

2.4 使用方式

• 通过环境变量导入

kubectl create secret generic pw-test --from-literal=password=test**0755

环境变量引用：

......
        env:
        - name: MY_PASS
          valueFrom:
            secretKeyRef:
              name: pw-test
              key: password
......

• 通过文件导入

kubectl create secret tls tls-test --cert=test.belkuy.top.crt --key=test.belkuy.top.key

挂载卷引用：

......
    volumeMounts:
      - name: secret-volume
        mountPath: /etc/secret
        readOnly: true

......

  volumes:
    - name: secret-volume
      secret:
        secretName: secret-test
......

三、deployment

Deployment 为 ReplicaSet 和 pod 的创建提供了一种声明式的定义方法，在 Deployment 对象中描述一个期望的状态，Deployment 控制器就会按照一定的控制速率把实际状态改成期望状态，通过定义一个 Deployment 控制器会创建一个新的 ReplicaSet 控制器，通过 ReplicaSet 创建 pod，删除 Deployment 控制器，也会删除 Deployment 控制器下对应的 ReplicaSet 控制器和pod 资源。

3.1 样例 nginx deployment

以 nginx 为例，同时 Deployment 中通过挂载的方式使用上述的 configmap、secret，另外镜像拉取使用了myharbor 这个 secret 下载镜像。

kubectl apply -f nginx.yaml -n test2023

---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    description: nginx-desc
  labels:
    k8s-app: nginx
  name: nginx
  namespace: test2023
spec:
  replicas: 1
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      k8s-app: nginx
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        k8s-app: nginx
    spec:
      containers:
        - image: 'myharbor.belkuy.top/base/nginx:1.22.0-alpine'
          imagePullPolicy: Always
          name: nginx
          resources: {}
          securityContext:
            privileged: false
            procMount: Default
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          # 挂载配置文件和TLS证书
          volumeMounts:
            - mountPath: /etc/nginx/conf.d/default.conf
              name: conf-nginx-default
              subPath: default.conf
            - mountPath: /etc/nginx/conf.d/ssl/test.belkuy.top.crt
              name: secret-crt
              subPath: test.belkuy.top.crt
            - mountPath: /etc/nginx/conf.d/ssl/test.belkuy.top.key
              name: secret-key
              subPath: test.belkuy.top.key
      dnsPolicy: ClusterFirst
      # 使用 secret —— myharbor 拉去私有镜像
      imagePullSecrets:
        - name: myharbor
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
        - configMap:
            defaultMode: 420
            items:
              - key: nginx-default.conf
                path: default.conf
            name: nginx-default
          name: conf-nginx-default
        - configMap:
            defaultMode: 420
            items:
              - key: test.belkuy.top.crt
                path: test.belkuy.top.crt
            name: tls-test
          name: secret-crt
        - configMap:
            defaultMode: 420
            items:
              - key: test.belkuy.top.key
                path: test.belkuy.top.key
            name: tls-test
          name: secret-key

四、service

service 是一个固定接入层，客户端可以通过访问 service 的 ip 和端口访问到 service 关联的后端 pod。

常见的两种就是 ClusterIP、NodePort。

• ClusterIP：通过 k8s 集群内部 IP 暴露服务，服务只能够在集群内部访问，这也是默认的 ServiceType。
• NodePort：通过每个 Node 节点上的 IP 和静态端口暴露 k8s 集群内部的服务。

4.1 样例 nginx service

以 nginx 为例，为上述 Deployment 创建一个 service。

---
apiVersion: v1
kind: Service
metadata:
  annotations: {}
  labels:
    k8s-app: nginx
  name: nginx
  namespace: test2023
spec:
  ports:
    - name: http
      port: 443
      protocol: TCP
      targetPort: 443
  selector:
    k8s-app: nginx
  sessionAffinity: None
  type: ClusterIP

五、StatefulSet

StatefulSet 是有状态的集合，管理有状态的服务，它所管理的 pod 的名称不能随意变化，数据持久化的目录也是不一样，每一个 pod 都有自己独有的数据持久化存储目录。常见的服务如 MySQL 主从、redis 集群等。

5.1 样例 eureka

---
apiVersion: v1
data:
  eureka_client_serviceUrl_defaultZone: >-
    http://eureka-0.eureka:8761/eureka,http://eureka-1.eureka:8761/eureka,http://eureka-2.eureka:8761/eureka
kind: ConfigMap
metadata:
  annotations: {}
  name: eureka-cm
  namespace: tool

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations: {}
  labels: {}
  name: eureka
  namespace: tool
spec:
  podManagementPolicy: OrderedReady
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: eureka
  serviceName: eureka
  template:
    metadata:
      labels:
        app: eureka
    spec:
      containers:
        - env:
            - name: EUREKA_SERVER_ADDRESS
              valueFrom:
                configMapKeyRef:
                  key: eureka_client_serviceUrl_defaultZone
                  name: eureka-cm
            - name: ENVIRONMENT
              value: prod
            - name: JVM_OPTS
              value: '-Xms1g -Xmx1g'
          image: 'myharbor.belkuy.top/base/eureka:v1'
          imagePullPolicy: Always
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /
              port: 8761
              scheme: HTTP
            initialDelaySeconds: 60
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          name: eureka
          ports:
            - containerPort: 8761
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /
              port: 8761
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          resources:
            limits:
              cpu: 500m
              memory: 1200Mi
            requests:
              cpu: 500m
              memory: 1Gi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      imagePullSecrets:
        - name: drone
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate

---
apiVersion: v1
kind: Service
metadata:
  annotations: {}
  labels:
    app: eureka
  name: eureka
  namespace: tool
spec:
  ports:
    - name: eureka
      port: 8761
      protocol: TCP
      targetPort: 8761
  selector:
    app: eureka
  sessionAffinity: None
  type: ClusterIP

5.2 样例 zookeeper

---
# 自定义配置文件
# for i in ${seq 0 2}; do kubectl exec zookeeper-$i -n zk -- zkServer.sh status; done
apiVersion: v1
data:
  java.env: 'JVMFLAGS="-Xms128M -Xmx3G -XX:+UseG1GC -XX:+CMSParallelRemarkEnabled"'
  log4j.properties: >-
    zookeeper.root.logger=CONSOLE

    zookeeper.console.threshold=INFO

    log4j.rootLogger=${zookeeper.root.logger}

    log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender

    log4j.appender.CONSOLE.Threshold=${zookeeper.console.threshold}

    log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout

    log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} [myid:%X{myid}] - %-5p [%t:%C{1}@%L] - %m%n

  zoo.cfg: >-
    clientPort=2181

    tickTime=2000

    initLimit=300

    syncLimit=10

    maxClientCnxns=2000

    maxSessionTimeout=60000000

    dataDir=/data

    dataLogDir=/datalog

    autopurge.snapRetainCount=10

    autopurge.purgeInterval=1

    preAllocSize=131072

    snapCount=3000000

    leaderServes=yes

    standaloneEnabled=false

    4lw.commands.whitelist=*

    metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider

    metricsProvider.httpPort=7000

    server.1=zookeeper-0.zookeeper-headless.zk.svc.cluster.local:2888:3888

    server.2=zookeeper-1.zookeeper-headless.zk.svc.cluster.local:2888:3888

    server.3=zookeeper-2.zookeeper-headless.zk.svc.cluster.local:2888:3888
kind: ConfigMap
metadata:
  name: conf-zoo
  namespace: zk
---
# 创建service服务
apiVersion: v1
kind: Service
metadata:
  name: zookeeper
  namespace: zk
  labels:
    app: zookeeper
spec:
  ports:
    - port: 2181
      name: client
    - port: 7000
      name: prometheus
  selector:
    app: zookeeper
---
# 创建headless服务
apiVersion: v1
kind: Service
metadata:
  name: zookeeper-headless
  namespace: zk
  labels:
    app: zookeeper
spec:
  ports:
    - port: 2888
      name: server
    - port: 3888
      name: leader-election
  clusterIP: None
  selector:
    app: zookeeper
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: zookeeper-pdb
  namespace: zk
spec:
  selector:
    matchLabels:
      app: zookeeper
  maxUnavailable: 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zookeeper
  namespace: zk
spec:
  selector:
    matchLabels:
      app: zookeeper
  serviceName: zookeeper-headless
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        app: zookeeper
      annotations:
        prometheus.io/port: '7000'
        prometheus.io/scrape: 'true'
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: "app"
                operator: In
                values:
                - zookeeper
            topologyKey: "kubernetes.io/hostname"
      containers:
        - name: zookeeper
          imagePullPolicy: IfNotPresent
          image: harbor-sit.jrtzcloud.cn/base/zookeeper:3.6.4
          volumeMounts:
            - mountPath: /conf/java.env
              name: conf-zoo-java
              subPath: java.env
            - mountPath: /conf/log4j.properties
              name: conf-zoo-log
              subPath: log4j.properties
            - mountPath: /conf/zoo.cfg
              name: conf-zoo-cfg
              subPath: zoo.cfg
          resources:
            requests:
              memory: "500Mi"
              cpu: "0.5"
            limits:
              memory: "4Gi"
              cpu: "1"
          ports:
            - containerPort: 2181
              name: client
            - containerPort: 2888
              name: server
            - containerPort: 3888
              name: leader-election
            - containerPort: 7000
              name: prometheus
          command:
            - bash
            - -x
            - -c
            - |
              HOST=`hostname -s` &&
              if [[ $HOST =~ (.*)-([0-9]+)$ ]]; then
                  ORD=${BASH_REMATCH[2]}
              else
                  exit 1
              fi &&
              export MY_ID=$((ORD+1)) &&
              echo $MY_ID > /data/myid &&
              zkServer.sh start-foreground
          readinessProbe:
            exec:
              command:
                - bash
                - -c
                - "OK=$(echo ruok | nc 127.0.0.1 2181); if [[ \"$OK\" == \"imok\" ]]; then exit 0; else exit 1; fi"
            initialDelaySeconds: 10
            timeoutSeconds: 5
          livenessProbe:
            exec:
              command:
                - bash
                - -c
                - "OK=$(echo ruok | nc 127.0.0.1 2181); if [[ \"$OK\" == \"imok\" ]]; then exit 0; else exit 1; fi"
            initialDelaySeconds: 10
            timeoutSeconds: 5
      volumes:
        - configMap:
            defaultMode: 420
            items:
              - key: java.env
                path: java.env
            name: conf-zoo
          name: conf-zoo-java
        - configMap:
            defaultMode: 420
            items:
              - key: log4j.properties
                path: log4j.properties
            name: conf-zoo
          name: conf-zoo-log
        - configMap:
            defaultMode: 420
            items:
              - key: zoo.cfg
                path: zoo.cfg
            name: conf-zoo
          name: conf-zoo-cfg

六、DaemonSet

DaemonSet 控制器使 k8s 集群所有的节点都运行一个相同的 pod 副本。

当向 k8s 集群中增加 node 节点时，会自动在这个 node 节点创建一个 pod 副本；当 node 节点从集群移除，该节点上的 pod 也会自动删除；删除 Daemonset 也会删除相应的 pod。

6.1 样例 node-exporter

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: '1'
  labels:
    app: node-exporter
  name: node-exporter
  namespace: monitoring
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
      creationTimestamp: null
      labels:
        app: node-exporter
    spec:
      containers:
        - args:
            - '--web.listen-address=127.0.0.1:9100'
            - '--path.procfs=/host/proc'
            - '--path.sysfs=/host/sys'
            - '--path.rootfs=/host/root'
            - '--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)'
            - '--collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$'
          image: 'prom/node-exporter:v1.4.0'
          imagePullPolicy: IfNotPresent
          name: node-exporter
          resources:
            limits:
              cpu: '1'
              memory: 256Mi
            requests:
              cpu: 100m
              memory: 200Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /host/proc
              name: proc
            - mountPath: /host/sys
              name: sys
            - mountPath: /host/root
              mountPropagation: HostToContainer
              name: root
              readOnly: true
      dnsPolicy: ClusterFirst
      hostNetwork: true
      hostPID: true
      nodeSelector:
        beta.kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
      serviceAccount: node-exporter
      serviceAccountName: node-exporter
      terminationGracePeriodSeconds: 30
      tolerations:
        - effect: NoExecute
          operator: Exists
        - effect: NoSchedule
          operator: Exists
      volumes:
        - hostPath:
            path: /proc
            type: ''
          name: proc
        - hostPath:
            path: /sys
            type: ''
          name: sys
        - hostPath:
            path: /
            type: ''
          name: root
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate

---
apiVersion: v1
kind: Service
metadata:
  annotations: {}
  labels:
    k8s-app: node-exporter
  name: node-exporter
  namespace: monitoring
spec:
  ports:
    - name: https
      port: 9100
      protocol: TCP
      targetPort: https
  selector:
    app: node-exporter
  sessionAffinity: None
  type: ClusterIP