前言:
ingress直译:进口;入口;初切;进入;进入资格;进入权。在kubernetes中,它指的是网络入口。
ingress概述:
通俗来讲,Ingress和之前提到的Service、Deployment等类似,也是一 个Kubernetes的资源对象,Deployment是用来部署应用的,Ingress就是实现 用域名的方式访问应用。Ingress实现的方式有很多,比如Nginx、HAProxy、 Treafik等(也就是控制器),就Nginx而言,和上述提到的传统服务架构用Nginx类似。 Ingress控制器在每个符合条件的宿主机上部署一个Pod,这个Pod里面运行的 就是Nginx进程,里面的实现逻辑和宿主机部署Nginx的方式并无太大区别, 关键区别是宿主机部署的Nginx需要更改Nginx的配置文件配置域名,而 Ingress则和其他Kubernetes资源文件一样,使用YAML文件进行配置,之后 Ingress控制器根据YAML文件定义的内容自动生成对应的配置文件。 在Kubernetes v1.1版中正式引用Ingress class的概念,用于从集群外部到集 群内部Service的HTTP和HTTPS路由,可以配置提供服务外部访问的URL、负载 均衡和终止SSL,并提供基于域名的虚拟主机。流量从Internet到Ingress再 到Services最后到Pod上,通常情况下,Ingress部署在所有的Node节点上, 暴露443和80端口(一般通过hostNetwork的方式部署Ingress),之后再通过 F5或公有云LB代理到对应的Ingress节点上,之后将域名解析到F5或公有云LB 即可实现基于域名的服务发布。
OK,为什么要用ingress呢?这个问题我先给打个10分。
确实,我们希望任何东西都是简单可靠可用的,但一台服务器上的端口是有限的,总共是65535好像,并且如果一个服务器如果像一面墙一样打满了孔洞,光管理这些端口就是一个非常大的工作量了,何况这些孔洞还有安全方面的风险(所以喽,有的安全性要求高的服务器会安装防火墙,就是堵这些孔洞的)。
回到k8s集群内,我们知道service的网络模式可以是NodePort或者EXTERNAL-IP或者是CLUSTER-IP或者是什么都没有的无头SERVICE(CLUSTER-IP 没有值)。
而想要把服务发布到集群外无外乎就是NodePort和EXTERNAL-IP两种方式了,CLUSTER-IP只能在集群内部使用,毕竟虚拟出来的IP地址嘛。
NodePort会在集群内的所有节点开辟一个随机的唯一的不重复的端口,并且这个端口还是由kube-apiserver限定的2700多个端口,关键的地方是随机和唯一不重复。
kube-apiserver的端口限定通常是它的配置文件(30000到32767):
KUBE_APISERVER_OPTS="--v=2 \
--logtostderr=false \
--service-node-port-range=30000-32767 \
--log-dir=/opt/kubernetes/logs \
--etcd-servers=https://192.168.217.16:2379,https://192.168.217.17:2379,https://192.168.217.18:2379 \
。。。。省略
--service-cluster-ip-range=10.0.0.0/16 \
。。。。省略
那么,如果有几十个service需要发布到集群外,使用NodePort就会有几十个上百个随机的端口,是不是十分的不安全和不方便呢(要查询出随机的端口,并记住的哦)。
EXTERNAL-IP 是使用互联网的IP,而互联网的IP资源是比较少的,这一条就把服务发布的路堵的差不多了。
而ingress 是在集群内创建一个service,此service可以绑定N个自定义的域名,并且它使用的端口是一个固定的端口,除非你重新生成这个service,而这就给了我们的无数可能,也因此,借由这个service,我们可以实现服务发布的特定流程,比如红蓝发布。
例如我这个ingress controller的service:
[root@k8s-master ~]# k get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.0.191.241 <none> 80:32296/TCP,443:30766/TCP 9h
只要利用ingress资源清单文件绑定一个域名+32296这个端口就可以在集群外访问到了,如果是tls类型的域名,那么就是域名+30766端口在集群外就可以访问到。可以看到,端口是固定了的两个,那么孔洞也就十分少了,安全性自然是提高了嘛,剩下的管理工作就是管理好这些ingress清单文件就可以了,这就自然得达到了服务治理的目的。
总结一哈,ingress可以实现服务治理(管理集群内的每一个service),服务发布(比如红蓝,金丝雀等等都是基于ingress),网络黑白名单,https支持等等功能(主要的还是服务service)。
简单的流程说一哈:
(1)某一个web应用(例如,门户网站,WordPress这样的cms等等,此时的表现形式是一个pod实例)---->基于web应用创建一个需要发布到集群外的service(这个service一般不需要使用NodePort或者EXTERNAL-IP,因为如果是这两个类型已经可以发布service到集群外了,在使用ingress就是脱裤子放P了)
(2)ingress资源清单文件(此文件绑定上面的web应用的service和一个自定义域名)---->ingress 控制器绑定一个ingress class(指定使用哪个控制器,如果有多个controller的话)---->ingress service(ingress 控制器发出一个service(通常这个service是NodePort或者EXTERNAL-IP的网络类型,这个服务非常重要,是总入口,是在部署ingress控制器的时候生成)---->ingress service通过控制器处理web应用衍生的那个service,也就是(1)的service
一,
ingress-nginx的部署:
ingress-nginx是一种网络插件,部署比较简单,说实话,部署简单,用好ingress真的非常不容易
版本问题:
由于kubernetes的版本众多,因此,ingress作为其扩展功能插件是也随着kubernetes的版本有更新,并且ingress资源定义也随着版本变迁有一定的改变。
在本例中,使用的kubernetes版本是1.18,经测试ingress是和其匹配的,但如果用在高版本的kubernetes会有问题,在此提醒一哈(具体表现是在高版本kubernetes内部署不会成功,高版本指的是1.22及以后的版本)。
Support Versions table
Ingress-nginx version | k8s supported version | Alpine Version | Nginx Version |
v1.1.1 | 1.23, 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.1.0 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.5 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.4 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.3 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.2 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.1 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.0 | 1.22, 1.21, 1.20, 1.19 | 3.13.5 | 1.20.1 |
v0.50.0 | 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v0.49.3 | 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v0.49.2 | 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v0.49.1 | 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v0.49.0 | 1.21, 1.20, 1.19 | 3.13.5 | 1.20.1 |
v0.48.1 | 1.21, 1.20, 1.19 | 3.13.5 | 1.20.1 |
v0.47.0 | 1.21, 1.20, 1.19 | 3.13.5 | 1.20.1 |
v0.46.0 | 1.21, 1.20, 1.19 | 3.13.2 | 1.19.6 |
#######附:
在Kubernetes 1.19之前,仅支持v1beta1 Ingress资源-从Kubernetes 1.19到1.21,同时支持v1beta1和v1 Ingress资源-在Kubernetes 1.22及更高版本中,仅支持v1 Ingress资源
yaml文件的下载:
版本号:GitHub - kubernetes/ingress-nginx: Ingress-NGINX Controller for Kubernetes
例如,下载1.1.1版本的yaml文件,如果是其它版本替换版本号就可以了
https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml
vim deploy-ingress.yaml
这么长的文件不用说了,很多同学看完必定会说:我的头要裂开了。没错,我也是这样的。
此文件主要有几个地方需要注意,第一,修改部署方式为daemonsets,第二,镜像部分我做了处理都是国内可以直接pull的,也就是本地化,第三,如果要部署的话,建议master节点打effect为NoScheduler的污点,毕竟ingress是一个重要服务,放到master和apiserver抢资源不太合适。
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
automountServiceAccountToken: true
---
apiVersion: v1
kind: ConfigMap
metadata:
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
data:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
name: ingress-nginx
rules:
- apiGroups:
- ''
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ''
resources:
- nodes
verbs:
- get
- apiGroups:
- ''
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- extensions
- networking.k8s.io # k8s 1.14+
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
- apiGroups:
- extensions
- networking.k8s.io # k8s 1.14+
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io # k8s 1.14+
resources:
- ingressclasses
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
name: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
rules:
- apiGroups:
- ''
resources:
- namespaces
verbs:
- get
- apiGroups:
- ''
resources:
- configmaps
- pods
- secrets
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- extensions
- networking.k8s.io # k8s 1.14+
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- extensions
- networking.k8s.io # k8s 1.14+
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io # k8s 1.14+
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- configmaps
resourceNames:
- ingress-controller-leader-nginx
verbs:
- get
- update
- apiGroups:
- ''
resources:
- configmaps
verbs:
- create
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller-admission
namespace: ingress-nginx
spec:
type: ClusterIP
ports:
- name: https-webhook
port: 443
targetPort: webhook
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
apiVersion: v1
kind: Service
metadata:
annotations:
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
type: NodePort
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
- name: https
port: 443
protocol: TCP
targetPort: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
revisionHistoryLimit: 10
minReadySeconds: 0
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
spec:
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: controller
image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v0.50.0
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
args:
- /nginx-ingress-controller
- --election-id=ingress-controller-leader
- --ingress-class=nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
runAsUser: 101
allowPrivilegeEscalation: true
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: LD_PRELOAD
value: /usr/local/lib/libmimalloc.so
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
ports:
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
protocol: TCP
- name: webhook
containerPort: 8443
protocol: TCP
volumeMounts:
- name: webhook-cert
mountPath: /usr/local/certificates/
readOnly: true
resources:
requests:
cpu: 100m
memory: 90Mi
nodeSelector:
kubernetes.io/os: linux
serviceAccountName: ingress-nginx
terminationGracePeriodSeconds: 300
volumes:
- name: webhook-cert
secret:
secretName: ingress-nginx-admission
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
name: ingress-nginx-admission
webhooks:
- name: validate.nginx.ingress.kubernetes.io
matchPolicy: Equivalent
rules:
- apiGroups:
- networking.k8s.io
apiVersions:
- v1beta1
operations:
- CREATE
- UPDATE
resources:
- ingresses
failurePolicy: Fail
sideEffects: None
admissionReviewVersions:
- v1
- v1beta1
clientConfig:
service:
namespace: ingress-nginx
name: ingress-nginx-controller-admission
path: /networking/v1beta1/ingresses
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: ingress-nginx-admission
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ingress-nginx-admission
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
rules:
- apiGroups:
- admissionregistration.k8s.io
resources:
- validatingwebhookconfigurations
verbs:
- get
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ingress-nginx-admission
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: ingress-nginx-admission
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
namespace: ingress-nginx
rules:
- apiGroups:
- ''
resources:
- secrets
verbs:
- get
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: ingress-nginx-admission
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
apiVersion: batch/v1
kind: Job
metadata:
name: ingress-nginx-admission-create
annotations:
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
namespace: ingress-nginx
spec:
template:
metadata:
name: ingress-nginx-admission-create
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
containers:
- name: create
image: jettech/kube-webhook-certgen:v1.5.1
imagePullPolicy: IfNotPresent
args:
- create
- --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
- --namespace=$(POD_NAMESPACE)
- --secret-name=ingress-nginx-admission
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
securityContext:
runAsNonRoot: true
runAsUser: 2000
---
apiVersion: batch/v1
kind: Job
metadata:
name: ingress-nginx-admission-patch
annotations:
helm.sh/hook: post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
namespace: ingress-nginx
spec:
template:
metadata:
name: ingress-nginx-admission-patch
labels:
helm.sh/chart: ingress-nginx-3.33.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.47.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
containers:
- name: patch
image: docker.io/jettech/kube-webhook-certgen:v1.5.1
imagePullPolicy: IfNotPresent
args:
- patch
- --webhook-name=ingress-nginx-admission
- --namespace=$(POD_NAMESPACE)
- --patch-mutating=false
- --secret-name=ingress-nginx-admission
- --patch-failure-policy=Fail
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
securityContext:
runAsNonRoot: true
runAsUser: 2000
apply上述文件后,将会出现这么些pod:
[root@master ~]# k get po -n ingress-nginx -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx-admission-create-xc2z4 0/1 Completed 0 4d2h 192.168.169.133 k8s-node2 <none> <none>
ingress-nginx-admission-patch-7xgst 0/1 Completed 3 4d2h 192.168.235.197 k8s-master <none> <none>
ingress-nginx-controller-987b747f-xrzn8 1/1 Running 17 4d2h 10.244.169.139 k8s-node2 <none> <none>
ingress-nginx-controller-ph46w 1/1 Running 0 163m 10.244.36.93 k8s-node1 <none> <none>
ingress-nginx-controller-t2nxd 1/1 Running 0 163m 10.244.169.163 k8s-node2 <none> <none>
将会有这么两个service:
注意31702这个端口,这个端口以后会常用的,这两个service是部署文件里产生的。
[root@master ~]# k get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.0.0.102 <none> 80:31702/TCP,443:31675/TCP 4d2h
ingress-nginx-controller-admission ClusterIP 10.0.0.12 <none> 443/TCP 4d2h
节点情况如下:
[root@master ~]# k get no -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-master Ready <none> 36d v1.18.3 192.168.217.16 <none> CentOS Linux 7 (Core) 5.16.9-1.el7.elrepo.x86_64 docker://20.10.7
k8s-node1 Ready <none> 36d v1.18.3 192.168.217.17 <none> CentOS Linux 7 (Core) 5.16.9-1.el7.elrepo.x86_64 docker://20.10.7
k8s-node2 Ready <none> 36d v1.18.3 192.168.217.18 <none> CentOS Linux 7 (Core) 5.16.9-1.el7.elrepo.x86_64 docker://20.10.7
二,
编写ingress资源清单文件
vim ingress-http.yaml
(两个自定义域名nginx.test.com和tomcat.test.com 绑定到了nginx-service和tomcat-service 这两服务上了)
这里要注意了,为什么两个servicePort都是80了,因为上面的ingress-nginx-controller是80端口嘛,由于我的kubernetes版本是1.18,因此还是使用注解方式。
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-http
namespace: dev
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: nginx.test.com
http:
paths:
- path: /
backend:
serviceName: nginx-service
servicePort: 80
- host: tomcat.test.com
http:
paths:
- path: /
backend:
serviceName: tomcat-service
servicePort: 80
此文件执行过后,ingress的情况如下:
可以看到此ingress是绑定到了192.168.217.18也就是node2节点了,绑定了两个域名,ADDRESS是node2节点的IP。
[root@master ~]# k get ing -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
dev ingress-http <none> nginx.test.com,tomcat.test.com 192.168.217.18 80 3h23m
建立namespace:
[root@master ~]# cat tomcat-nginx-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
name: dev
---
vim tomcat-nginx.yaml
建立两个deployment的pod,可提供web功能的,一个nginx 一个tomcat,两个都做了node选择,和ingress-nginx-controller处于同一个节点。
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
namespace: dev
spec:
replicas: 1
selector:
matchLabels:
app: nginx-pod
template:
metadata:
labels:
app: nginx-pod
spec:
containers:
- name: nginx
image: nginx:1.17.1
ports:
- containerPort: 80
nodeName: k8s-node2
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-deployment
namespace: dev
spec:
replicas: 1
selector:
matchLabels:
app: tomcat-pod
template:
metadata:
labels:
app: tomcat-pod
spec:
containers:
- name: tomcat
image: tomcat:8.5-jre10-slim
ports:
- containerPort: 8080
nodeName: k8s-node2
vim tomcat-nginx-svc.yaml
这里又需要注意了,两个service一个无头service,一个普通的clusterip,一会使用了ingress清单文件就可以将这两个服务发布到集群外了。
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
namespace: dev
spec:
ports:
- port: 80
name: nginx
clusterIP: None
selector:
app: nginx-pod
---
apiVersion: v1
kind: Service
metadata:
name: tomcat-service
namespace: dev
spec:
selector:
app: tomcat-pod
type: ClusterIP
ports:
- protocol: TCP
port: 80
targetPort: 8080
三,
OK,以上文件都apply后,就可以看结果了:
宿主机做hosts域名解析:
OK,如果nginx部署到node1节点会怎么样呢?
报错504
这就说明一个问题,kubectl get ingress -A 查询出来的那个IP地址也就是ingress的节点和要发布的service对应的pod要在同一个节点下,和service的类型没有关系,即使service是无头的也是OK的,ingress-nginx-controller会帮我们自己处理好的。并且多个service都是通过同一个端口发不出来的,只是域名不同而已。
四,
改造成https也就是使用ssl的域名(实验性质,当然还是使用自签的证书,实际生产环境肯定是使用备案过的证书哦)
a,
生成自签证书
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=tomcat.test.com/ST=BeiJing/L=BeiJing/O=devops/OU=unicorn"
可以看到,生成了这么两个玩意
[root@master ~]# ls tls*
tls.crt tls.key
b,
生成secret,证书存放到secret里
注意,这里要指定namespace,要不ingress controller找不到证书
kubectl create secret tls tls-secret --key=tls.key --cert tls.crt -n dev
c,
编写ingress文件
vim ingress-https.yaml
主要就是添加了tls相关,域名还是不变的以及一个注解,并且引用了前面打入的证书
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-ingress3
namespace: dev
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
kubernetes.io/ingress.class: nginx
# nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/ssl-redirect: 'true'
# # nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
tls:
- hosts:
- tomcat.test.com
secretName: tls-secret
rules:
- host: tomcat.test.com
http:
paths:
- path: /
backend:
serviceName: tomcat-service
servicePort: 80
查看ingress,可以看到多了一个443,还是绑定的node2节点(ingress 控制器前面部署的时候搞错了,就部署在了一个节点,daemonset没有使用的。)
[root@master ~]# k get ing -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
dev ingress-http <none> nginx.test.com,tomcat.test.com 192.168.217.18 80 11h
dev test-ingress3 <none> tomcat.test.com 192.168.217.18 80, 443 4m16s
d,
验证;
需要先查询一哈ingress的service提供的端口,查询出端口是31675
[root@master ~]# vim ingress-https.yaml
[root@master ~]# k get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 37d
dev nginx-service ClusterIP None <none> 80/TCP 16h
dev tomcat-service ClusterIP 10.0.92.37 <none> 80/TCP 16h
ingress-nginx ingress-nginx-controller NodePort 10.0.0.102 <none> 80:31702/TCP,443:31675/TCP 4d14h
ingress-nginx ingress-nginx-controller-admission ClusterIP 10.0.0.12 <none> 443/TCP 4d14h
kube-system coredns ClusterIP 10.0.0.2 <none> 53/UDP,53/TCP 36d
OK,https证书启用成功,此网站的证书只是没有注册的自产证书,但功能是完好的。
总结:
那么现在这个ingress controller插件是可以使用的,ingress统一了要发布服务的端口,可以看到即使多个门户,也可以简单的以域名来区分,端口是统一的31702(http)或者31675(https),从而达到了服务治理的目的(其它功能,比如黑白名单,重定向,二级域名跳转等等留待以后研究哈):
[root@master ~]# k get ing -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
dev ingress-http <none> nginx.test.com,tomcat.test.com 192.168.217.18 80 11h
dev test-ingress3 <none> tomcat.test.com 192.168.217.18 80, 443 4m16s
ingress-nginx-controller截取的部分日志:
192.168.217.18 - - [03/Oct/2022:09:03:03 +0000] "GET /favicon.ico HTTP/1.1" 404 555 "http://nginx.test.com:31702/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36 Core/1.94.175.400 QQBrowser/11.1.5155.400" 423 0.001 [dev-nginx-service-80] [] 10.244.36.97:80 555 0.001 404 3b851fa39d2268e5fbd230fc5f8d1d59
192.168.217.18 - - [03/Oct/2022:09:03:05 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36 Core/1.94.175.400 QQBrowser/11.1.5155.400" 581 0.001 [dev-nginx-service-80] [] 10.244.36.97:80 0 0.001 304 21366d34e3103a50236738d6b1dd00e7
192.168.217.18 - - [03/Oct/2022:09:03:07 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36 Core/1.94.175.400 QQBrowser/11.1.5155.400" 581 0.002 [dev-nginx-service-80] [] 10.244.36.97:80 0 0.002 304 69bcb2182681ea3be696fe3b449e286c
192.168.217.18 - - [03/Oct/2022:09:03:09 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36 Core/1.94.175.400 QQBrowser/11.1.5155.400" 581 0.001 [dev-nginx-service-80] [] 10.244.36.97:80 0 0.000 304 3622a7bf35abbdf08c43084c89fd0110
192.168.217.18 - - [03/Oct/2022:09:06:15 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36 Core/1.94.175.400 QQBrowser/11.1.5155.400" 500 0.003 [dev-nginx-service-80] [] 10.244.36.97:80 612 0.003 200 5f6ba8d7070c9b7984acf2012fa57a5b
192.168.217.18 - - [03/Oct/2022:09:06:17 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36 Core/1.94.175.400 QQBrowser/11.1.5155.400" 581 0.002 [dev-nginx-service-80] [] 10.244.36.97:80 0 0.001 304 12d114732628f18df5988661bf79fc83
192.168.217.18 - - [03/Oct/2022:09:16:42 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" 460 0.001 [dev-nginx-service-80] [] 10.244.36.97:80 612 0.001 200 298738d5c747741af42d8f13fb4c4566
可以看到我是使用的QQ浏览器(也用了谷歌105版本),192.168.217.18:31702 代理了10.244.36.97:80
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
dev nginx-deployment-b785b4498-5r8jn 1/1 Running 0 16m 10.244.36.97 k8s-node1 <none> <none>