异常值探测配置示例
- 常用的错误标识
- consecutiveLocalOriginFailures
- consecutiveGatewayErrors
- 只包括502、503和504错误
- 0值表示禁用
- consecutive5xxErrors
- 其它常用参数
- interval
- baseEjectionTime
- maxEjectionPercent：最大弹出比例，默认为10，即10%
- minHealthPercent：低于该比例时，Outlier Detection将被禁用
destinationrule-demoapp.yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: demoapp
spec:
host: demoapp
trafficPolicy:
loadBalancer:
simple: RANDOM
connectionPool:
tcp:
maxConnections: 100
connectTimeout: 30ms
tcpKeepalive:
time: 7200s
interval: 75s
http:
http2MaxRequests: 1000
maxRequestsPerConnection: 10
outlierDetection:
maxEjectionPercent: 50
consecutive5xxErrors: 5
interval: 10s
baseEjectionTime: 1m
minHealthPercent: 40
subsets:
- name: v10
labels:
version: v1.0
- name: v11
labels:
version: v1.1
virutalservice-demoapp.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: demoapp
spec:
hosts:
- demoapp
http:
- name: canary
match:
- uri:
prefix: /canary
rewrite:
uri: /
route:
- destination:
host: demoapp
subset: v11
- name: default
route:
- destination:
host: demoapp
subset: v10
更新demoapp dr
kubectl apply -f destinationrule-demoapp.yaml -f virutalservice-demoapp.yaml
destinationrule.networking.istio.io/demoapp configured
virutalservice.networking.istio.io/demoapp configured
访问demoapp
访问v10
~# while true;do curl app.wgs.com; sleep 0.5;done
Proxying value: iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.6, ServerName: demoappv10-78b6586d58-kdv4m, ServerIP: 172.20.154.206!
- Took 333 milliseconds.
Proxying value: iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.6, ServerName: demoappv10-78b6586d58-hq957, ServerIP: 172.20.44.212!
- Took 25 milliseconds.
Proxying value: iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.6, ServerName: demoappv10-78b6586d58-hq957, ServerIP: 172.20.44.212!
- Took 20 milliseconds.
Proxying value: iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.6, ServerName: demoappv10-78b6586d58-kdv4m, ServerIP: 172.20.154.206!
- Took 7 milliseconds.
Proxying value: iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.6, ServerName: demoappv10-78b6586d58-kdv4m, ServerIP: 172.20.154.206!
- Took 4 milliseconds.
访问v11
~# while true;do curl app.wgs.com/canary; sleep 0.5;done
Proxying value: iKubernetes demoapp v1.1 !! ClientIP: 127.0.0.6, ServerName: demoappv11-78bf898c74-sn9qp, ServerIP: 172.20.154.214!
- Took 37 milliseconds.
Proxying value: iKubernetes demoapp v1.1 !! ClientIP: 127.0.0.6, ServerName: demoappv11-78bf898c74-hckjf, ServerIP: 172.20.89.133!
- Took 24 milliseconds.
Proxying value: iKubernetes demoapp v1.1 !! ClientIP: 127.0.0.6, ServerName: demoappv11-78bf898c74-hckjf, ServerIP: 172.20.89.133!
- Took 7 milliseconds.
Proxying value: iKubernetes demoapp v1.1 !! ClientIP: 127.0.0.6, ServerName: demoappv11-78bf898c74-sn9qp, ServerIP: 172.20.154.214!
- Took 19 milliseconds.
Proxying value: iKubernetes demoapp v1.1 !! ClientIP: 127.0.0.6, ServerName: demoappv11-78bf898c74-hckjf, ServerIP: 172.20.89.133!
- Took 14 milliseconds.
Proxying value: iKubernetes demoapp v1.1 !! ClientIP: 127.0.0.6, ServerName: demoappv11-78bf898c74-sn9qp, ServerIP: 172.20.154.214!
- Took 4 milliseconds.
访问kiali
测试异常值探测
访问v10 livez
~# while true;do curl app.wgs.com/livez; sleep 0.5;done
Proxying value: OK - Took 5010 milliseconds.
Proxying value: OK - Took 4 milliseconds.
Proxying value: OK - Took 5011 milliseconds.
Proxying value: OK - Took 6 milliseconds.
Proxying value: OK - Took 6 milliseconds.
Proxying value: OK - Took 5 milliseconds.
Proxying value: OK - Took 4 milliseconds.
Proxying value: OK - Took 5 milliseconds.
Proxying value: OK - Took 6 milliseconds.
Proxying value: OK - Took 4 milliseconds.
Proxying value: OK - Took 6 milliseconds.
Proxying value: OK - Took 7 milliseconds.
Proxying value: OK - Took 5 milliseconds.
Proxying value: OK - Took 5 milliseconds.
查看kiali
查看v10 pod
~# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
demoappv10-78b6586d58-hq957 2/2 Running 6 (29m ago) 2d4h 172.20.44.212 192.168.174.107 <none> <none>
demoappv10-78b6586d58-kdv4m 2/2 Running 6 (27m ago) 2d4h 172.20.154.206 192.168.174.106 <none> <none>
demoappv11-78bf898c74-hckjf 2/2 Running 6 (29m ago) 2d4h 172.20.89.133 192.168.174.108 <none> <none>
demoappv11-78bf898c74-sn9qp 2/2 Running 6 (27m ago) 2d4h 172.20.154.214 192.168.174.106 <none> <none>
proxy-649b4d887d-zbx26 2/2 Running 6 (27m ago) 2d4h 172.20.154.205 192.168.174.106 <none> <none>
v10故障注入
~# curl -I 172.20.44.212:8080/livez
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
content-length: 2
server: istio-envoy
date: Thu, 27 Oct 2022 06:30:55 GMT
x-envoy-upstream-service-time: 2
x-envoy-decorator-operation: demoapp.default.svc.cluster.local:8080/*
~# curl -X POST -d 'livez=FAIL' 172.20.44.212:8080/livez
~# curl -I 172.20.44.212:8080/livez
HTTP/1.1 506 Variant Also Negotiates
content-type: text/html; charset=utf-8
content-length: 4
server: istio-envoy
date: Thu, 27 Oct 2022 06:32:02 GMT
x-envoy-upstream-service-time: 0
x-envoy-decorator-operation: demoapp.default.svc.cluster.local:8080/*
查看访问结果
Proxying value: OK - Took 5 milliseconds.
Proxying value: FAIL - Took 16 milliseconds.
Proxying value: OK - Took 4 milliseconds.
Proxying value: OK - Took 5 milliseconds.
Proxying value: FAIL - Took 21 milliseconds.
Proxying value: FAIL - Took 8 milliseconds.
Proxying value: OK - Took 8 milliseconds.
Proxying value: FAIL - Took 6 milliseconds.
Proxying value: OK - Took 6 milliseconds.
Proxying value: OK - Took 5 milliseconds.
Proxying value: FAIL - Took 7 milliseconds.
查看kiali
参考文档
https://istio.io/latest/zh/docs/reference/config/networking/destination-rule/#OutlierDetection
