k8s中负载均衡器【ingress-nginx】部署

原创

进击云原生 2021-07-07 09:40:20 ©著作权

文章标签 kubernetes 文章分类 网络安全

©著作权归作者所有：来自51CTO博客作者进击云原生的原创作品，请联系作者获取转载授权，否则将追究法律责任

在Kubernetes中，服务和Pod的IP地址仅可以在集群网络内部使用，对于集群外的应用是不可见的。为了使外部的应用能够访问集群内的服务，在Kubernetes 目前提供了以下几种方案：

NodePort
LoadBalancer
Ingress

本节主要就ingress和ingress控制器ingress-nginx-controller的部署作简单介绍和记录。

以下系统组件版本：

云服务器：centos版本7.6.1810、k8s版本1.15.0、docker版本18.06.1-ce、ingress-nginx-controller版本0.25.0

Ingress

Ingress 组成？

将Nginx的配置抽象成一个Ingress对象，每添加一个新的服务只需写一个新的Ingress的yaml文件即可

将新加入的Ingress转化成Nginx的配置文件并使之生效

ingress controller
ingress服务

Ingress 工作原理?
ingress controller通过和kubernetes api交互，动态的去感知集群中ingress规则变化，
然后读取它，按照自定义的规则，规则就是写明了哪个域名对应哪个service，生成一段nginx配置，
再写到nginx-ingress-controller的pod里，这个Ingress controller的pod里运行着一个Nginx服务，控制器会把生成的nginx配置写入/etc/nginx.conf文件中，
然后reload一下使配置生效。
以此达到域名分配置和动态更新的问题。

Ingress 可以解决什么问题？
动态配置服务
如果按照传统方式, 当新增加一个服务时, 我们可能需要在流量入口加一个反向代理指向我们新的服务. 而如果用了Ingress, 只需要配置好这个服务, 当服务启动时, 会自动注册到Ingress的中, 不需要而外的操作.
减少不必要的端口暴露
配置过k8s的都清楚, 第一步是要关闭防火墙的, 主要原因是k8s的很多服务会以NodePort方式映射出去, 这样就相当于给宿主机打了很多孔, 既不安全也不优雅. 而Ingress可以避免这个问题, 除了Ingress自身服务可能需要映射出去, 其他服务都不要用NodePort方式

Ingress当前的实现方式？

k8s中负载均衡器【ingress-nginx】部署_kubernetes

ingress-nginx-controller

目前最新版本的ingress-nginx-controller，用lua实现了当upstream变化时不用reload，大大减少了生产环境中由于服务的重启、升级引起的IP变化导致的nginx reload。

以下就ingress-nginx-controller的部署做简单记录：

yaml如下：

kubectl apply -f {如下文件}

apiVersion: v1kind: Namespacemetadata:  name: ingress-nginx  labels:    /name: ingress-nginx    /part-of: ingress-nginx
---
kind: ConfigMapapiVersion: v1metadata:  name: nginx-configuration  namespace: ingress-nginx  labels:    /name: ingress-nginx    /part-of: ingress-nginx
---kind: ConfigMapapiVersion: v1metadata:  name: tcp-services  namespace: ingress-nginx  labels:    /name: ingress-nginx    /part-of: ingress-nginx
---kind: ConfigMapapiVersion: v1metadata:  name: udp-services  namespace: ingress-nginx  labels:    /name: ingress-nginx    /part-of: ingress-nginx
---apiVersion: v1kind: ServiceAccountmetadata:  name: nginx-ingress-serviceaccount  namespace: ingress-nginx  labels:    /name: ingress-nginx    /part-of: ingress-nginx
---apiVersion: /v1beta1kind: ClusterRolemetadata:  name: nginx-ingress-clusterrole  labels:    /name: ingress-nginx    /part-of: ingress-nginxrules:  - apiGroups:      - ""    resources:      - configmaps      - endpoints      - nodes      - pods      - secrets    verbs:      - list      - watch  - apiGroups:      - ""    resources:      - nodes    verbs:      - get  - apiGroups:      - ""    resources:      - services    verbs:      - get      - list      - watch  - apiGroups:      - ""    resources:      - events    verbs:      - create      - patch  - apiGroups:      - "extensions"      - ""    resources:      - ingresses    verbs:      - get      - list      - watch  - apiGroups:      - "extensions"      - ""    resources:      - ingresses/status    verbs:      - update
---apiVersion: /v1beta1kind: Rolemetadata:  name: nginx-ingress-role  namespace: ingress-nginx  labels:    /name: ingress-nginx    /part-of: ingress-nginxrules:  - apiGroups:      - ""    resources:      - configmaps      - pods      - secrets      - namespaces    verbs:      - get  - apiGroups:      - ""    resources:      - configmaps    resourceNames:      # Defaults to "<election-id>-<ingress-class>"      # Here: "<ingress-controller-leader>-<nginx>"      # This has to be adapted if you change either parameter      # when launching the nginx-ingress-controller.      - "ingress-controller-leader-nginx"    verbs:      - get      - update  - apiGroups:      - ""    resources:      - configmaps    verbs:      - create  - apiGroups:      - ""    resources:      - endpoints    verbs:      - get
---apiVersion: /v1beta1kind: RoleBindingmetadata:  name: nginx-ingress-role-nisa-binding  namespace: ingress-nginx  labels:    /name: ingress-nginx    /part-of: ingress-nginxroleRef:  apiGroup:   kind: Role  name: nginx-ingress-rolesubjects:  - kind: ServiceAccount    name: nginx-ingress-serviceaccount    namespace: ingress-nginx
---apiVersion: /v1beta1kind: ClusterRoleBindingmetadata:  name: nginx-ingress-clusterrole-nisa-binding  labels:    /name: ingress-nginx    /part-of: ingress-nginxroleRef:  apiGroup:   kind: ClusterRole  name: nginx-ingress-clusterrolesubjects:  - kind: ServiceAccount    name: nginx-ingress-serviceaccount    namespace: ingress-nginx
---
apiVersion: apps/v1kind: Deploymentmetadata:  name: nginx-ingress-controller  namespace: ingress-nginx  labels:    /name: ingress-nginx    /part-of: ingress-nginxspec:  replicas: 1  selector:    matchLabels:      /name: ingress-nginx      /part-of: ingress-nginx  template:    metadata:      labels:        /name: ingress-nginx        /part-of: ingress-nginx      annotations:        prometheus.io/port: "10254"        prometheus.io/scrape: "true"    spec:      serviceAccountName: nginx-ingress-serviceaccount      containers:        - name: nginx-ingress-controller          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.25.0          args:            - /nginx-ingress-controller            - --configmap=$(POD_NAMESPACE)/nginx-configuration            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services            - --publish-service=$(POD_NAMESPACE)/ingress-nginx            - --annotations-prefix=          securityContext:            allowPrivilegeEscalation: true            capabilities:              drop:                - ALL              add:                - NET_BIND_SERVICE            # www-data -> 33            runAsUser: 33          env:            - name: POD_NAME              valueFrom:                fieldRef:                  fieldPath: metadata.name            - name: POD_NAMESPACE              valueFrom:                fieldRef:                  fieldPath: metadata.namespace          ports:            - name: http              containerPort: 80            - name: https              containerPort: 443          livenessProbe:            failureThreshold: 3            httpGet:              path: /healthz              port: 10254              scheme: HTTP            initialDelaySeconds: 10            periodSeconds: 10            successThreshold: 1            timeoutSeconds: 10          readinessProbe:            failureThreshold: 3            httpGet:              path: /healthz              port: 10254              scheme: HTTP            periodSeconds: 10            successThreshold: 1            timeoutSeconds: 10
---

在墙内会拉取不到以下镜像：

quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.25.0

可以使用国内阿里云镜像：

docker pull /google_containers/nginx-ingress-controller:0.25.0

安装后，在ingress-nginx命名空间下可以看到pod一直pending，describe pod报如下警告：

k8s中负载均衡器【ingress-nginx】部署_kubernetes_02

查看master节点默认加了污点，一般不允许pod调度到master节点：

k8s中负载均衡器【ingress-nginx】部署_kubernetes_03

如果k8s集群只有一个节点，可以在pod的spec下设置容忍该污点：

k8s中负载均衡器【ingress-nginx】部署_kubernetes_04

即：

   spec:            tolerations:      - effect: NoSchedule        key: /master

可以看到ingress-nginx pod被调度到master节点，且变为Running

k8s中负载均衡器【ingress-nginx】部署_kubernetes_05

看日志报以下警告：

W0714 13:31:04.883127       6 queue.go:130] requeuing &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],}, err services "ingress-nginx" not found

需要创一个名为ingress-nginx的service：

kubectl apply -f {如下文件}

kind: ServiceapiVersion: v1metadata:  name: ingress-nginx  namespace: ingress-nginx  labels:    /name: ingress-nginx    /part-of: ingress-nginxspec:  externalTrafficPolicy: Local  type: LoadBalancer  selector:    /name: ingress-nginx    /part-of: ingress-nginx  ports:    - name: http      port: 80      targetPort: http    - name: https      port: 443      targetPort: https

参考：

ingress deploy

https:///kubernetes/ingress-nginx/blob/master/docs/deploy/

Taint和Toleration（污点和容忍）

https://jimmysong.io/kubernetes-handbook/concepts/taint-and-toleration.html

k8s 1.12部署ingress-nginx

https://www.jianshu.com/p/e30b06906b77

k8s中负载均衡器【ingress-nginx】部署_kubernetes_06