K8S部署多个ingress用于多环境访问

目前K8S集群部署了dev、test、prod3套环境，为了统一管理都使用ingress统一对外暴露，这就需要部署多个ingress，部署后，只需要在创建ingress的时候指定ingress.class即可,我们这里统一使用ingress-nginx(v1.3.1版本)来进行部署。

1.下载yaml文件，打开​​https://github.com/kubernetes/ingress-nginx/blob/tag-1.3.1/deploy/static/provider/baremetal/deploy.yaml​​页面，将里面的代码复制出来保存为ingress.yaml文件

2.修改镜像,由于官方的image是放在谷歌服务器，无法直接下载，这里需要修改镜像的地址.

可以使用shell在进行修改：

sed -i 's#registry.k8s.io/ingress-nginx/controller:v1.3.1.*#dyrnq/ingress-nginx-controller:v1.3.1#g' ingress.yaml
sed -i 's#registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0.*#dyrnq/kube-webhook-certgen:v1.3.0#g' ingress.yaml

436c436
<         image: registry.k8s.io/ingress-nginx/controller:v1.3.1@sha256:54f7fe2c6c5a9db9a0ebf1131797109bb7a4d91f56b9b362bde2abd237dd1974
---
>         image: dyrnq/ingress-nginx-controller:v1.3.1
533c533
<         image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47
---
>         image: dyrnq/kube-webhook-certgen:v1.3.0
582c582
<         image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.3.0@sha256:549e71a6ca248c5abd51cdb73dbc3083df62cf92ed5e6147c780e30f7e007a47
---
>         image: dyrnq/kube-webhook-certgen:v1.3.0

3.这个时候如果不需要多ingress部署的话，直接应用即可。但是我们这里不是，我们需要对文件进行改造

#我们对yaml复制一份出来进行备份，后面我们就拿ingress-namespace.yaml来进行编辑
cp ingress.yaml ingress-namespace.yaml

#下面我就对ingress-namespace.yaml进行编辑
sed -i 's#namespace: .*#namespace: {namespace}#g' ingress-namespace.yaml
sed -i 's#instance: ingress-nginx#instance: {namespace}-ingress-nginx#g' ingress-namespace.yaml
sed -i 's#app.kubernetes.io/name: ingress-nginx#app.kubernetes.io/name: {namespace}-ingress-nginx#g' ingress-namespace.yaml
sed -i 's#ingress-class=nginx#ingress-class={namespace}#g' ingress-namespace.yaml
sed -i 's#controller-class=k8s.io/ingress-nginx#controller-class=k8s.io/{namespace}-ingress-nginx#g' ingress-namespace.yaml
sed -i 's#controller: k8s.io/ingress-nginx#controller: k8s.io/{namespace}-ingress-nginx#g' ingress-namespace.yaml
sed -i 's#  name: nginx#  name: {namespace}#g' ingress-namespace.yaml
sed -i 's#name: ingress-nginx-admission#name: {namespace}-ingress-nginx-admission#g' ingress-namespace.yaml
sed -i 's#  name: ingress-nginx$#  name: {namespace}-ingress-nginx#g' ingress-namespace.yaml
sed -i 's#serviceAccountName: ingress-nginx-admission#serviceAccountName: {namespace}-ingress-nginx-admission#g' ingress-namespace.yaml
sed -i 's#serviceAccountName: ingress-nginx$#serviceAccountName: {namespace}-ingress-nginx#g' ingress-namespace.yaml
sed -i 's#webhook-name=ingress-nginx-admission#webhook-name={namespace}-ingress-nginx-admission#g' ingress-namespace.yaml
sed -i 's#secret-name=ingress-nginx-admission#secret-name={namespace}-ingress-nginx-admission#g' ingress-namespace.yaml
sed -i 's#secretName: ingress-nginx-admission#secretName: {namespace}-ingress-nginx-admission#g' ingress-namespace.yaml
sed -i 's#host=ingress-nginx-controller-admission,ingress-nginx-controller-admission#host={namespace}-ingress-nginx-controller-admission,{namespace}-ingress-nginx-controller-admission#g' ingress-namespace.yaml
sed -i 's#name: ingress-nginx-controller-admission#name: {namespace}-ingress-nginx-controller-admission#g' ingress-namespace.yaml
#最后修改文件的第7行由  name: {namespace}-ingress-nginx 修改  name: {namespace}即可

4.使用如下语句即可创建dev、test、prod 3个Ingress

sed 's#{namespace}#dev#g'  ingress-namespace.yaml |kubectl apply -f -
sed 's#{namespace}#test#g'  ingress-namespace.yaml |kubectl apply -f -
sed 's#{namespace}#prod#g'  ingress-namespace.yaml |kubectl apply -f -

5.创建一个ingress用于测试

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: www
  namespace: default
  annotations:
    kubernetes.io/ingress.class: dev   #修改这里即可
spec:
  rules:
  - host: www.xxxx.cn
    http:
      paths:
      - backend:
          service:
            name: web
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific

6.最终的代码如下

apiVersion: v1
kind: Namespace
metadata:
  labels:
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
  name: {namespace}
---
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: {namespace}-ingress-nginx
  namespace: {namespace}
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: {namespace}-ingress-nginx-admission
  namespace: {namespace}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: {namespace}-ingress-nginx
  namespace: {namespace}
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - configmaps
  - pods
  - secrets
  - endpoints
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses/status
  verbs:
  - update
- apiGroups:
  - networking.k8s.io
  resources:
  - ingressclasses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resourceNames:
  - ingress-controller-leader
  resources:
  - configmaps
  verbs:
  - get
  - update
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - create
- apiGroups:
  - coordination.k8s.io
  resourceNames:
  - ingress-controller-leader
  resources:
  - leases
  verbs:
  - get
  - update
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - create
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: {namespace}-ingress-nginx-admission
  namespace: {namespace}
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: {namespace}-ingress-nginx
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - endpoints
  - nodes
  - pods
  - secrets
  - namespaces
  verbs:
  - list
  - watch
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses/status
  verbs:
  - update
- apiGroups:
  - networking.k8s.io
  resources:
  - ingressclasses
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: {namespace}-ingress-nginx-admission
rules:
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - validatingwebhookconfigurations
  verbs:
  - get
  - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: {namespace}-ingress-nginx
  namespace: {namespace}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: {namespace}-ingress-nginx
subjects:
- kind: ServiceAccount
  name: {namespace}-ingress-nginx
  namespace: {namespace}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: {namespace}-ingress-nginx-admission
  namespace: {namespace}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: {namespace}-ingress-nginx-admission
subjects:
- kind: ServiceAccount
  name: {namespace}-ingress-nginx-admission
  namespace: {namespace}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: {namespace}-ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: {namespace}-ingress-nginx
subjects:
- kind: ServiceAccount
  name: {namespace}-ingress-nginx
  namespace: {namespace}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: {namespace}-ingress-nginx-admission
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: {namespace}-ingress-nginx-admission
subjects:
- kind: ServiceAccount
  name: {namespace}-ingress-nginx-admission
  namespace: {namespace}
---
apiVersion: v1
data:
  allow-snippet-annotations: "true"
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: ingress-nginx-controller
  namespace: {namespace}
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: ingress-nginx-controller
  namespace: {namespace}
spec:
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - appProtocol: http
    name: http
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: {namespace}-ingress-nginx-controller-admission
  namespace: {namespace}
spec:
  ports:
  - appProtocol: https
    name: https-webhook
    port: 443
    targetPort: webhook
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: ingress-nginx-controller
  namespace: {namespace}
spec:
  minReadySeconds: 0
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: {namespace}-ingress-nginx
      app.kubernetes.io/name: {namespace}-ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: {namespace}-ingress-nginx
        app.kubernetes.io/name: {namespace}-ingress-nginx
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --election-id=ingress-controller-leader
        - --controller-class=k8s.io/{namespace}-ingress-nginx
        - --ingress-class={namespace}
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: LD_PRELOAD
          value: /usr/local/lib/libmimalloc.so
        image: dyrnq/ingress-nginx-controller:v1.3.1
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - /wait-shutdown
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: controller
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        - containerPort: 8443
          name: webhook
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 100m
            memory: 90Mi
        securityContext:
          allowPrivilegeEscalation: true
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - ALL
          runAsUser: 101
        volumeMounts:
        - mountPath: /usr/local/certificates/
          name: webhook-cert
          readOnly: true
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: {namespace}-ingress-nginx
      terminationGracePeriodSeconds: 300
      volumes:
      - name: webhook-cert
        secret:
          secretName: {namespace}-ingress-nginx-admission
---
apiVersion: batch/v1
kind: Job
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: {namespace}-ingress-nginx-admission-create
  namespace: {namespace}
spec:
  template:
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: {namespace}-ingress-nginx
        app.kubernetes.io/name: {namespace}-ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.3.1
      name: {namespace}-ingress-nginx-admission-create
    spec:
      containers:
      - args:
        - create
        - --host={namespace}-ingress-nginx-controller-admission,{namespace}-ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
        - --namespace=$(POD_NAMESPACE)
        - --secret-name={namespace}-ingress-nginx-admission
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        image: dyrnq/kube-webhook-certgen:v1.3.0
        imagePullPolicy: IfNotPresent
        name: create
        securityContext:
          allowPrivilegeEscalation: false
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: OnFailure
      securityContext:
        fsGroup: 2000
        runAsNonRoot: true
        runAsUser: 2000
      serviceAccountName: {namespace}-ingress-nginx-admission
---
apiVersion: batch/v1
kind: Job
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: {namespace}-ingress-nginx-admission-patch
  namespace: {namespace}
spec:
  template:
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: {namespace}-ingress-nginx
        app.kubernetes.io/name: {namespace}-ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.3.1
      name: {namespace}-ingress-nginx-admission-patch
    spec:
      containers:
      - args:
        - patch
        - --webhook-name={namespace}-ingress-nginx-admission
        - --namespace=$(POD_NAMESPACE)
        - --patch-mutating=false
        - --secret-name={namespace}-ingress-nginx-admission
        - --patch-failure-policy=Fail
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        image: dyrnq/kube-webhook-certgen:v1.3.0
        imagePullPolicy: IfNotPresent
        name: patch
        securityContext:
          allowPrivilegeEscalation: false
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: OnFailure
      securityContext:
        fsGroup: 2000
        runAsNonRoot: true
        runAsUser: 2000
      serviceAccountName: {namespace}-ingress-nginx-admission
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: {namespace}
spec:
  controller: k8s.io/{namespace}-ingress-nginx
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: {namespace}-ingress-nginx
    app.kubernetes.io/name: {namespace}-ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: {namespace}-ingress-nginx-admission
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: {namespace}-ingress-nginx-controller-admission
      namespace: {namespace}
      path: /networking/v1/ingresses
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: validate.nginx.ingress.kubernetes.io
  rules:
  - apiGroups:
    - networking.k8s.io
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - ingresses
  sideEffects: None