写在前面
- 学习
K8s中Service遇到，单独整理分享给小伙伴 - 博文内容涉及：
-
Metallb的创建 -
LoadBalancer类型的Service的创建
有些人心如花木，皆向阳而生 ——烽火戏诸侯《剑来》
Service 负载均衡问题
NodePort还没有完全解决外部访问Service的所有问题,比如负载均衡问题,假如我们的集群中有10个Node,则此时最好有一个负载均衡器,外部的请求只需访问此负载均衡器的IP地址,由负载均衡器负责转发流量到后面某个Node的NodePort上。如图
NodePort的负载均衡 |
|
LoadBalancer方式
Load balancer组件独立于Kubernetes集群之外,通常是一个硬件的负载均衡器,或者是以软件方式实现的,例如HAProxy或者Nginx。对于每个Service,我们通常需要配置一个对应的Load balancer实例来转发流量到后端的Node上
Kubernetes提供了自动化的解决方案,如果我们的集群运行在谷歌的GCE公有云上,那么只要我们把Service的type-NodePort改为type-LoadBalancer,此时Kubernetes会自动创建一个对应的Load balancer实例并返回它的IP地址供外部客户端使用。在没有公有云的情况下，我们也可以用一些插件来实现，如metallb等
LoadBalancer 需要建立服务之外的负载池。然后给Service分配一个IP。
我们直接创建一个LoadBalancer的Service的时候，会一直处于pending状态，是因为我们没有对应的云负载均衡器
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$kubectl expose --name=blogsvc pod blog --port=80 --type=LoadBalancer
service/blogsvc exposed
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$kubectl get svc -o wide | grep blogsvc
blogsvc LoadBalancer 10.106.28.175 <pending> 80:32745/TCP 26s run=blog
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$
Metallb创建
Metallb可以通过k8s原生的方式提供LB类型的Service支持
使用： metallb https://metallb.universe.tf/ |
|
资源文件https://github.com/metallb/metallb/blob/main/manifests/metallb.yaml |
创建命名空间
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$kubectl create ns metallb-system
namespace/metallb-system created
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$kubectl config set-context $(kubectl config current-context) --namespace=metallb-system
Context "kubernetes-admin@kubernetes" modified.
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$
:set paste 解决粘贴混乱的问题
创建metallb
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl apply -f metallb.yaml
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
controller-66d9554cc-8rxq8 1/1 Running 0 3m36s 10.244.171.170 vms82.liruilongs.github.io <none> <none>
speaker-bbl94 1/1 Running 0 3m36s 192.168.26.83 vms83.liruilongs.github.io <none> <none>
speaker-ckbzj 1/1 Running 0 3m36s 192.168.26.81 vms81.liruilongs.github.io <none> <none>
speaker-djmpr 1/1 Running 0 3m36s 192.168.26.82 vms82.liruilongs.github.io <none> <none>
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$
创建地址池- 192.168.26.240-192.168.26.250
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$vim pool.yaml
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl apply -f pool.yaml
configmap/config created
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.26.240-192.168.26.250
使用type=LoadBalancer的配置通过metallb分配192.168.26.240这个地址给blogsvc
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl get svc
No resources found in metallb-system namespace.
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl config set-context $(kubectl config current-context) --namespace=liruilong-svc-create
Context "kubernetes-admin@kubernetes" modified.
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dbsvc ClusterIP 10.102.137.59 <none> 3306/TCP 101m
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl expose --name=blogsvc pod blog --port=80 --type=LoadBalancer
service/blogsvc exposed
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
blogsvc LoadBalancer 10.108.117.197 192.168.26.240 80:30230/TCP 9s run=blog
dbsvc ClusterIP 10.102.137.59 <none> 3306/TCP 101m run=dbpod
直接访问 |
|
在创建一个也可以访问
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl expose --name=blogsvc-1 pod blog --port=80 --type=LoadBalancer
service/blogsvc-1 exposed
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
blogsvc LoadBalancer 10.108.117.197 192.168.26.240 80:30230/TCP 11m run=blog
blogsvc-1 LoadBalancer 10.110.58.143 192.168.26.241 80:31827/TCP 3s run=blog
dbsvc ClusterIP 10.102.137.59 <none> 3306/TCP 113m run=dbpod
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$
也可以访问 |
|
相关的资源文件
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$cat namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: metallb-system
labels:
app: metallb
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$cat pool.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.26.240-192.168.26.250
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$cat metallb.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
labels:
app: metallb
name: controller
namespace: metallb-system
spec:
allowPrivilegeEscalation: false
allowedCapabilities: []
allowedHostPaths: []
defaultAddCapabilities: []
defaultAllowPrivilegeEscalation: false
fsGroup:
ranges:
- max: 65535
min: 1
rule: MustRunAs
hostIPC: false
hostNetwork: false
hostPID: false
privileged: false
readOnlyRootFilesystem: true
requiredDropCapabilities:
- ALL
runAsUser:
ranges:
- max: 65535
min: 1
rule: MustRunAs
seLinux:
rule: RunAsAny
supplementalGroups:
ranges:
- max: 65535
min: 1
rule: MustRunAs
volumes:
- configMap
- secret
- emptyDir
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
labels:
app: metallb
name: speaker
namespace: metallb-system
spec:
allowPrivilegeEscalation: false
allowedCapabilities:
- NET_RAW
allowedHostPaths: []
defaultAddCapabilities: []
defaultAllowPrivilegeEscalation: false
fsGroup:
rule: RunAsAny
hostIPC: false
hostNetwork: true
hostPID: false
hostPorts:
- max: 7472
min: 7472
- max: 7946
min: 7946
privileged: true
readOnlyRootFilesystem: true
requiredDropCapabilities:
- ALL
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- configMap
- secret
- emptyDir
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app: metallb
name: controller
namespace: metallb-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app: metallb
name: speaker
namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app: metallb
name: metallb-system:controller
rules:
- apiGroups:
- ''
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- services/status
verbs:
- update
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
- apiGroups:
- policy
resourceNames:
- controller
resources:
- podsecuritypolicies
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app: metallb
name: metallb-system:speaker
rules:
- apiGroups:
- ''
resources:
- services
- endpoints
- nodes
verbs:
- get
- list
- watch
- apiGroups: ["discovery.k8s.io"]
resources:
- endpointslices
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
- apiGroups:
- policy
resourceNames:
- speaker
resources:
- podsecuritypolicies
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app: metallb
name: config-watcher
namespace: metallb-system
rules:
- apiGroups:
- ''
resources:
- configmaps
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app: metallb
name: pod-lister
namespace: metallb-system
rules:
- apiGroups:
- ''
resources:
- pods
verbs:
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app: metallb
name: controller
namespace: metallb-system
rules:
- apiGroups:
- ''
resources:
- secrets
verbs:
- create
- apiGroups:
- ''
resources:
- secrets
resourceNames:
- memberlist
verbs:
- list
- apiGroups:
- apps
resources:
- deployments
resourceNames:
- controller
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app: metallb
name: metallb-system:controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: metallb-system:controller
subjects:
- kind: ServiceAccount
name: controller
namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app: metallb
name: metallb-system:speaker
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: metallb-system:speaker
subjects:
- kind: ServiceAccount
name: speaker
namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app: metallb
name: config-watcher
namespace: metallb-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: config-watcher
subjects:
- kind: ServiceAccount
name: controller
- kind: ServiceAccount
name: speaker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app: metallb
name: pod-lister
namespace: metallb-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: pod-lister
subjects:
- kind: ServiceAccount
name: speaker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app: metallb
name: controller
namespace: metallb-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: controller
subjects:
- kind: ServiceAccount
name: controller
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app: metallb
app.kubernetes.io/component: speaker
name: speaker
namespace: metallb-system
spec:
selector:
matchLabels:
app: metallb
app.kubernetes.io/component: speaker
template:
metadata:
annotations:
prometheus.io/port: '7472'
prometheus.io/scrape: 'true'
labels:
app: metallb
app.kubernetes.io/component: speaker
spec:
containers:
- args:
- --port=7472
- --config=config
- --log-level=info
env:
- name: METALLB_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: METALLB_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: METALLB_ML_BIND_ADDR
valueFrom:
fieldRef:
fieldPath: status.podIP
# needed when another software is also using memberlist / port 7946
# when changing this default you also need to update the container ports definition
# and the PodSecurityPolicy hostPorts definition
#- name: METALLB_ML_BIND_PORT
# value: "7946"
- name: METALLB_ML_LABELS
value: "app=metallb,app.kubernetes.io/component=speaker"
- name: METALLB_ML_SECRET_KEY
valueFrom:
secretKeyRef:
name: memberlist
key: secretkey
image: quay.io/metallb/speaker:main
name: speaker
ports:
- containerPort: 7472
name: monitoring
- containerPort: 7946
name: memberlist-tcp
- containerPort: 7946
name: memberlist-udp
protocol: UDP
livenessProbe:
httpGet:
path: /metrics
port: monitoring
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /metrics
port: monitoring
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 3
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_RAW
drop:
- ALL
readOnlyRootFilesystem: true
hostNetwork: true
nodeSelector:
kubernetes.io/os: linux
serviceAccountName: speaker
terminationGracePeriodSeconds: 2
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: metallb
app.kubernetes.io/component: controller
name: controller
namespace: metallb-system
spec:
revisionHistoryLimit: 3
selector:
matchLabels:
app: metallb
app.kubernetes.io/component: controller
template:
metadata:
annotations:
prometheus.io/port: '7472'
prometheus.io/scrape: 'true'
labels:
app: metallb
app.kubernetes.io/component: controller
spec:
containers:
- args:
- --port=7472
- --config=config
- --log-level=info
env:
- name: METALLB_ML_SECRET_NAME
value: memberlist
- name: METALLB_DEPLOYMENT
value: controller
image: quay.io/metallb/controller:main
name: controller
ports:
- containerPort: 7472
name: monitoring
livenessProbe:
httpGet:
path: /metrics
port: monitoring
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /metrics
port: monitoring
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 3
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- all
readOnlyRootFilesystem: true
nodeSelector:
kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 65534
fsGroup: 65534
serviceAccountName: controller
terminationGracePeriodSeconds: 0
