k8s1.21搭建ingress-nginx高可用集群

在k8s1.21 安装ingress-nginx这篇文章中，我们借助ingress,通过Deployment + nodePort的方式将集群内部的服务暴露出去。此时，整个集群中只有一个ingress-nginx实例，可以通过

kubectl get svc -n ingress-nginx -o wide

查看ingress-nginx绑定的端口。如果这个ingress-nginx出现了故障，将导致整个集群不可用。本文介绍一种DaemonSet+HostNetwork+nodeSelector方式搭建的多ingress-nginx实例高可用集群。

  1. HostNetwork
    是为了打通Cluster和node的网络，让Cluster直接监听node的端口，一般是80和443，不用再通过随机绑定的nodePort来访问集群服务。拿k8s1.21 安装ingress-nginx这里的例子来说就是
以前访问：http://www.zcx.com:30080/web1/contextpath
现在访问：http://www.zcx.com/web1/contextpath
可以看到不再需要随机绑定的nodePort了，因为hostNetwork直接监听集群的80端口。
  1. nodeSelector
    可以通过nodeSelector确定将ingress-nginx与哪些主机的
    在DNS解析的时候，一定要解析这些被select的节点，不再是解析集群中任意节点了
    比如我们的nodeSelector把ingress-nginx的pod调度到k8snode02节点了，那么DNS解析时 只能 这么写
192.168.56.121 k8snode01 
192.168.56.122 k8snode02  www.zcx.com
192.168.56.123 k8snode03

因为ingress-nginx的pod被调度到k8snode02上，就会把k8snode02的80端口和443端口和集群打通，而不会把k8snode01或k8snode03的80端口和443端口和集群打通。

  1. DaemonSet
    因为Deployment可能会把多个pod调度到同一个node，那就失去高可用的意义了。而DaemonSet在一个节点上只会有一个Pod，符合我们的要求。
    要根据标签调度Pod，那先打标签。我们有k8snode01(master)、k8snode02、k8snode03三个节点，计划在k8snode02和k8snode03上安装ingress-nginx。
#打标签
kubectl label node k8snode02 hasIngress=true
kubectl label node k8snode03 hasIngress=true

下载ingress-nginx镜像和yaml文件参考基于K8S 1.21.2集群安装Ingress-Nginx 0.48.1

修改yaml文件

只需要改Deployment部分，一共修改三处

# Source: ingress-nginx/templates/controller-deployment.yaml
#有文章把这里改成apiVersion: extensions/v1beta1，我这里会报错，不改不报错，应该跟k8s版本有关，这里是v1.21
apiVersion: apps/v1
#kind: Deployment
kind: DaemonSet  #这里把Deployment改成DaemonSet
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/component: controller
  revisionHistoryLimit: 10
  minReadySeconds: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/component: controller
    spec:
      dnsPolicy: ClusterFirst
      hostNetwork: true  #这里加一句
      containers:
        - name: controller
          image: k8s.gcr.io/ingress-nginx/controller:v0.48.1
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown
          args:
            - /nginx-ingress-controller
            - --election-id=ingress-controller-leader
            - --ingress-class=nginx
            - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
            - --validating-webhook=:8443
            - --validating-webhook-certificate=/usr/local/certificates/cert
            - --validating-webhook-key=/usr/local/certificates/key
          .....省略
      nodeSelector:
        kubernetes.io/os: linux
        hasIngress: "true"  # 这里个面具标签选择node
      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      volumes:
        - name: webhook-cert
          secret:
            secretName: ingress-nginx-admission

lvs ingresscontroller高可用_kubernetes


可以看到启动了两个ingress-nginx实例，已经调度到指定的node上了。

此时在k8snode02和k8snode03上查看端口经停情况

netstat -lntup | grep nginx

lvs ingresscontroller高可用_kubernetes_02


可以看到ingress-nginx已经在监听本机80和443端口了。

Deploy、Svc和Ingress

记得要自己整一个springboot项目，参见k8s1.21 安装ingress-nginx

kind: Ingress
metadata:
  name: nginx-http
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/use-regex: "true"
  namespace: default
spec:
  rules:
    - host: "www.zcx.com"
      http:
        paths:
        - path: /web1
          pathType: Prefix
          backend:
            service:
              name: ws1-svc
              port:
                number: 9090
        - path: /web2
          pathType: Prefix
          backend:
            service:
              name: ws2-svc
              port:
                number: 9090
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ws1-deploy
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ws1
  template:
    metadata:
      labels:
        app: ws1
    spec:
      containers:
      - name: ws1
        image: ws1:latest
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 9090
---
apiVersion: v1
kind: Service
metadata:
  name: ws1-svc
  namespace: default
spec:
  type: ClusterIP  # 默认类型
  selector:
    app: ws1
  ports:
  - name: http
    port: 9090
    targetPort: 9090
        
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ws2-deploy
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ws2
  template:
    metadata:
      labels:
        app: ws2
    spec:
      containers:
      - name: ws2
        image: ws2:latest
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 9090
---
apiVersion: v1
kind: Service
metadata:
  name: ws2-svc
  namespace: default
spec:
  type: ClusterIP  # 默认类型
  selector:
    app: ws2
  ports:
  - name: http
    port: 9090
    targetPort: 9090

DNS解析

前面也提到了，此时哪个node安装了ingress-nginx，就得解析哪个node。

#在测试机上
vim /etc/hosts

lvs ingresscontroller高可用_TCP_03

测试

curl www.zcx.com/web1/contextpath
curl www.zcx.com/web2/contextpath

lvs ingresscontroller高可用_kubernetes_04


此时，驱逐k8snode02

#在master上
kubectl drain k8snode02 --delete-local-data --force --ignore-daemonsets node/k8snode02 
kubectl delete node k8snode02
#在k8snode02上
kubeadm reset
#在master上检查ingress-nginx的Pod是不是少了，如下图，可知k8snode02已经从集群删除
kubectl get pod -n ingress-nginx -o wide
#再发起请求
curl www.zcx.com/web1/contextpath
curl www.zcx.com/web2/contextpath

lvs ingresscontroller高可用_TCP_05


可以看到，服务仍然可用。

后记

  1. 把k8snode02再加回来后，可能会出现Pod无法调度的问题：failed to set bridge addr: “cni0“ already has an IP address different from xxx问题，可以参照这篇文章解决。
  2. 虽然ingress-nginx直接监听node的80和443端口，其实此时仍然有随机绑定的端口
kubectl get svc -n ingress-nginx

NAME                                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.1.70.66    <none>        80:32034/TCP,443:31032/TCP   50m
ingress-nginx-controller-admission   ClusterIP   10.1.72.104   <none>        443/TCP                      50m

可以看到绑定了32034端口，此时仍然可以通过http://www.zcx.com:32034/web1/contextpath来请求服务。当然这是因为我们的安装文件是一个大而全的文件，完全可以把这个Service去掉，这样就不会随机绑定端口了。
随机绑定端口的nodePort是这个Service。

# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
	  nodePort: 30080 #可以指定绑定的端口
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
	  nodePort: 30443 #可以指定绑定的端口
      protocol: TCP
      targetPort: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller