环境准备
httpbin 服务
httpbin 是一个使用 Python + Flask 编写的 HTTP 请求响应服务。
该服务主要用于测试 HTTP 库,你可以向他发送请求,然后他会按照指定的规则将你的请求返回。
部署 httpbin
服务:
kubectl apply -f samples/httpbin/httpbin.yaml
fortio 服务
Fortio是一个微服务(http, grpc)负载测试库,命令行工具,高级回显服务器和web UI在go (golang)。Fortio允许指定一组每秒查询负载,并记录延迟直方图和其他有用的统计数据。
部署 fortio
服务:
kubectl apply -f samples/httpbin/sample-client/fortio-deploy.yaml
httpbin
服务作为接收请求的服务端, fortio
服务作为发送请求的客户端。
熔断配置
创建目标规则,访问httpbin
服务时应用熔断配置:
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: httpbin
spec:
host: httpbin
trafficPolicy:
connectionPool:
http:
http1MaxPendingRequests: 1
maxRequestsPerConnection: 1
EOF
在 fortio
服务中向 httpbin
服务的发出并发请求:
export FORTIO_POD=$(kubectl get pod -l app=fortio -n istio-demo -o jsonpath={.items..metadata.name})
kubectl exec -it "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 5 -qps 0 -n 5 -loglevel Warning http://httpbin:8000/get
返回结果如下:
I0725 11:43:55.895950 4039039 request.go:621] Throttling request took 1.130727121s, request: GET:https://172.24.28.71:6443/apis/ceph.rook.io/v1?timeout=32s
03:43:58 I logger.go:127> Log level is now 3 Warning (was 2 Info)
Fortio 1.17.1 running at 0 queries per second, 104->104 procs, for 5 calls: http://httpbin:8000/get
Starting at max qps with 5 thread(s) [gomax 104] for exactly 5 calls (1 per thread + 0)
03:43:58 W http_client.go:806> [3] Non ok http code 503 (HTTP/1.1 503)
03:43:58 W http_client.go:806> [1] Non ok http code 503 (HTTP/1.1 503)
03:43:58 W http_client.go:806> [2] Non ok http code 503 (HTTP/1.1 503)
Ended after 5.398808ms : 5 calls. qps=926.13
Aggregated Function Time : count 5 avg 0.0035955544 +/- 0.001035 min 0.002619591 max 0.005337051 sum 0.017977772
# range, mid point, percentile, count
>= 0.00261959 <= 0.003 , 0.0028098 , 60.00, 3
> 0.004 <= 0.005 , 0.0045 , 80.00, 1
> 0.005 <= 0.00533705 , 0.00516853 , 100.00, 1
# target 50% 0.0029049
# target 75% 0.00475
# target 90% 0.00516853
# target 99% 0.0053202
# target 99.9% 0.00533537
Sockets used: 5 (for perfect keepalive, would be 5)
Jitter: false
Code 200 : 2 (40.0 %)
Code 503 : 3 (60.0 %)
Response Header Sizes : count 5 avg 92 +/- 112.7 min 0 max 230 sum 460
Response Body/Total Sizes : count 5 avg 419.8 +/- 177.3 min 275 max 637 sum 2099
All done 5 calls (plus 0 warmup) 3.596 ms avg, 926.1 qps
可以看到,在发送的5个请求中,有3个请求返回 503 。
查看Envoy日志
执行下面的命令,查看fortio
服务的Envoy日志:
kubectl logs -l app=fortio -c istio-proxy
可以看到fortio
服务对httpbin
服务的调用的日志:
{
"authority": "httpbin:8000",
"bytes_received": 0,
"bytes_sent": 81,
"connection_termination_details": null,
"downstream_local_address": "172.24.146.239:8000",
"downstream_remote_address": "172.24.158.166:55030",
"duration": 1,
"method": "GET",
"path": "/get",
"protocol": "HTTP/1.1",
"request_id": "3aa81a10-cddf-9b3b-b704-c1099b05c6fb",
"requested_server_name": null,
"response_code": 503,
"response_code_details": "via_upstream",
"response_flags": "-",
"route_name": "allow_any",
"start_time": "2022-07-22T09:19:04.465Z",
"upstream_cluster": "PassthroughCluster",
"upstream_host": "172.24.146.239:8000",
"upstream_local_address": "172.24.158.166:55042",
"upstream_service_time": "1",
"upstream_transport_failure_reason": null,
"user_agent": "fortio.org/fortio-1.17.1",
"x_forwarded_for": null
}
其中,response_flags
为-
,没有任何异常。
执行下面的命令,查看httpbin
服务的Envoy日志:
kubectl logs -l app=httpbin -c istio-proxy
可以看到httpbin
服务被fortio
服务调用的Envoy日志:
{
"authority": "httpbin:8000",
"bytes_received": 0,
"bytes_sent": 81,
"connection_termination_details": null,
"downstream_local_address": "172.24.158.96:80",
"downstream_remote_address": "172.24.158.166:55042",
"duration": 0,
"method": "GET",
"path": "/get",
"protocol": "HTTP/1.1",
"request_id": "3aa81a10-cddf-9b3b-b704-c1099b05c6fb",
"requested_server_name": null,
"response_code": 503,
"response_code_details": "upstream_reset_before_response_started{overflow}",
"response_flags": "UO",
"route_name": "default",
"start_time": "2022-07-22T09:19:04.466Z",
"upstream_cluster": "inbound|80||",
"upstream_host": null,
"upstream_local_address": null,
"upstream_service_time": null,
"upstream_transport_failure_reason": null,
"user_agent": "fortio.org/fortio-1.17.1",
"x_forwarded_for": null
}
其中,response_flags
为UO
,表示因为熔断产生的上游(upstream)溢出。
深入分析
通过Envoy日志,我们可以做出一些分析和判断:
当httpbin
服务的请求正常的时候,调用过程如下图:
当httpbin
服务的熔断被触发的时候,调用过程如下图:
当httpbin
服务熔断被触发时,返回503状态码的是httpbin
服务的Envoy。
清理
kubectl delete destinationrule httpbin
kubectl delete -f samples/httpbin/httpbin.yaml
kubectl delete -f samples/httpbin/sample-client/fortio-deploy.yaml
封面最后,感谢你这么帅,还给我点赞。
网友评论