异常值探测配置示例

• 常用的错误标识

  • consecutiveLocalOriginFailures

  • consecutiveGatewayErrors

    • 只包括502、503和504错误

    • 0值表示禁用

  • consecutive5xxErrors

• 其它常用参数

  • interval

  • baseEjectionTime

  • maxEjectionPercent：最大弹出比例，默认为10，即10%

  • minHealthPercent：低于该比例时，Outlier Detection将被禁用

destinationrule-demoapp.yaml

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: demoapp
spec:
  host: demoapp
  trafficPolicy:
    loadBalancer:
      simple: RANDOM
    connectionPool:
      tcp:
        maxConnections: 100
        connectTimeout: 30ms
        tcpKeepalive:
          time: 7200s
          interval: 75s
      http:
        http2MaxRequests: 1000
        maxRequestsPerConnection: 10
    outlierDetection:
      maxEjectionPercent: 50
      consecutive5xxErrors: 5
      interval: 10s 
      baseEjectionTime: 1m
      minHealthPercent: 40
  subsets:
  - name: v10
    labels:
      version: v1.0
  - name: v11
    labels:
      version: v1.1

virutalservice-demoapp.yaml

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: demoapp
spec:
  hosts:
  - demoapp
  http:
  - name: canary
    match:
    - uri:
        prefix: /canary
    rewrite:
      uri: /
    route:
    - destination:
        host: demoapp
        subset: v11
  - name: default
    route:
    - destination:
        host: demoapp
        subset: v10

更新demoapp dr

kubectl apply -f destinationrule-demoapp.yaml -f virutalservice-demoapp.yaml
destinationrule.networking.istio.io/demoapp configured
virutalservice.networking.istio.io/demoapp configured

访问demoapp

访问v10

~# while true;do curl app.wgs.com; sleep 0.5;done
Proxying value: iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.6, ServerName: demoappv10-78b6586d58-kdv4m, ServerIP: 172.20.154.206!
 - Took 333 milliseconds.
Proxying value: iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.6, ServerName: demoappv10-78b6586d58-hq957, ServerIP: 172.20.44.212!
 - Took 25 milliseconds.
Proxying value: iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.6, ServerName: demoappv10-78b6586d58-hq957, ServerIP: 172.20.44.212!
 - Took 20 milliseconds.
Proxying value: iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.6, ServerName: demoappv10-78b6586d58-kdv4m, ServerIP: 172.20.154.206!
 - Took 7 milliseconds.
Proxying value: iKubernetes demoapp v1.0 !! ClientIP: 127.0.0.6, ServerName: demoappv10-78b6586d58-kdv4m, ServerIP: 172.20.154.206!
 - Took 4 milliseconds.

访问v11

~# while true;do curl app.wgs.com/canary; sleep 0.5;done
Proxying value: iKubernetes demoapp v1.1 !! ClientIP: 127.0.0.6, ServerName: demoappv11-78bf898c74-sn9qp, ServerIP: 172.20.154.214!
 - Took 37 milliseconds.
Proxying value: iKubernetes demoapp v1.1 !! ClientIP: 127.0.0.6, ServerName: demoappv11-78bf898c74-hckjf, ServerIP: 172.20.89.133!
 - Took 24 milliseconds.
Proxying value: iKubernetes demoapp v1.1 !! ClientIP: 127.0.0.6, ServerName: demoappv11-78bf898c74-hckjf, ServerIP: 172.20.89.133!
 - Took 7 milliseconds.
Proxying value: iKubernetes demoapp v1.1 !! ClientIP: 127.0.0.6, ServerName: demoappv11-78bf898c74-sn9qp, ServerIP: 172.20.154.214!
 - Took 19 milliseconds.
Proxying value: iKubernetes demoapp v1.1 !! ClientIP: 127.0.0.6, ServerName: demoappv11-78bf898c74-hckjf, ServerIP: 172.20.89.133!
 - Took 14 milliseconds.
Proxying value: iKubernetes demoapp v1.1 !! ClientIP: 127.0.0.6, ServerName: demoappv11-78bf898c74-sn9qp, ServerIP: 172.20.154.214!
 - Took 4 milliseconds.

访问kiali

image

测试异常值探测

访问v10 livez

~# while true;do curl app.wgs.com/livez; sleep 0.5;done
Proxying value: OK - Took 5010 milliseconds.
Proxying value: OK - Took 4 milliseconds.
Proxying value: OK - Took 5011 milliseconds.
Proxying value: OK - Took 6 milliseconds.
Proxying value: OK - Took 6 milliseconds.
Proxying value: OK - Took 5 milliseconds.
Proxying value: OK - Took 4 milliseconds.
Proxying value: OK - Took 5 milliseconds.
Proxying value: OK - Took 6 milliseconds.
Proxying value: OK - Took 4 milliseconds.
Proxying value: OK - Took 6 milliseconds.
Proxying value: OK - Took 7 milliseconds.
Proxying value: OK - Took 5 milliseconds.
Proxying value: OK - Took 5 milliseconds.

查看kiali

[图片上传中...(image-f133c8-1673919029135-1)]

查看v10 pod

~# kubectl get pod -o wide
NAME                          READY   STATUS    RESTARTS      AGE    IP               NODE              NOMINATED NODE   READINESS GATES
demoappv10-78b6586d58-hq957   2/2     Running   6 (29m ago)   2d4h   172.20.44.212    192.168.174.107   <none>           <none>
demoappv10-78b6586d58-kdv4m   2/2     Running   6 (27m ago)   2d4h   172.20.154.206   192.168.174.106   <none>           <none>
demoappv11-78bf898c74-hckjf   2/2     Running   6 (29m ago)   2d4h   172.20.89.133    192.168.174.108   <none>           <none>
demoappv11-78bf898c74-sn9qp   2/2     Running   6 (27m ago)   2d4h   172.20.154.214   192.168.174.106   <none>           <none>
proxy-649b4d887d-zbx26        2/2     Running   6 (27m ago)   2d4h   172.20.154.205   192.168.174.106   <none>           <none>

v10故障注入

~# curl -I 172.20.44.212:8080/livez
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
content-length: 2
server: istio-envoy
date: Thu, 27 Oct 2022 06:30:55 GMT
x-envoy-upstream-service-time: 2
x-envoy-decorator-operation: demoapp.default.svc.cluster.local:8080/*

~# curl -X POST -d 'livez=FAIL' 172.20.44.212:8080/livez
~# curl -I 172.20.44.212:8080/livez
HTTP/1.1 506 Variant Also Negotiates
content-type: text/html; charset=utf-8
content-length: 4
server: istio-envoy
date: Thu, 27 Oct 2022 06:32:02 GMT
x-envoy-upstream-service-time: 0
x-envoy-decorator-operation: demoapp.default.svc.cluster.local:8080/*

查看访问结果

Proxying value: OK - Took 5 milliseconds.
Proxying value: FAIL - Took 16 milliseconds.
Proxying value: OK - Took 4 milliseconds.
Proxying value: OK - Took 5 milliseconds.
Proxying value: FAIL - Took 21 milliseconds.
Proxying value: FAIL - Took 8 milliseconds.
Proxying value: OK - Took 8 milliseconds.
Proxying value: FAIL - Took 6 milliseconds.
Proxying value: OK - Took 6 milliseconds.
Proxying value: OK - Took 5 milliseconds.
Proxying value: FAIL - Took 7 milliseconds.

查看kiali

image

参考文档

https://istio.io/latest/zh/docs/reference/config/networking/destination-rule/#OutlierDetection