排查过很多次pod网络有问题的场景 ，一直没太弄明白，pod内的网络报文怎么抓，今天遇到一个liveness健康检查失败的问题，liveness是kubelet去访问pod（发生get请求）来确认的，那么对应的访问日志在pod内是可以看到的，因此可以尝试抓包排查，但是pod并不能简简单单的使用tcpdump抓包即可，docker/kubectl cp 一个tcpdump进去也不好使，那么，pod内的包怎么抓呢？
容器的网络隔离使用的是linux的network namespace ，因此我们到对应的ns里面抓包即可，下面演示
1.查看指定 pod 运行在哪个宿主机上
拿到宿主机的信息，然后登陆上去 cn-shenzhen.192.168.0.178
# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
busybox-5fc755d7d9-nc8bz 1/1 Running 2 5d5h 172.20.2.21 cn-shenzhen.192.168.0.178 <none>
2.获得容器的 pid
登录到对应的node上，下面演示的是两种找pid的方法
# docker ps|grep busy
d5ae39bad811 busybox "sleep 360000" 29 minutes ago Up 29 minutes k8s_busybox_busybox-5fc755d7d9-nc8bz_default_b9a845f1-f09b-11e9-a7ea-00163e0e34c8_2
cdab20715cd9 registry-vpc.cn-shenzhen.aliyuncs.com/acs/pause-amd64:3.0 "/pause" 29 minutes ago Up 29 minutes k8s_POD_busybox-5fc755d7d9-nc8bz_default_b9a845f1-f09b-11e9-a7ea-00163e0e34c8_4
[root@izwz9314kt10onbwuw6odez ~]# docker inspect -f {{.State.Pid}} d5ae39bad811
6875
3.进入该容器的 network namespace
nsenter依赖util-linux，因此提前安装
# yum -y install util-linux.x86_64
进入到对应容器的network ns里面，并指向ip a查看ip
# nsenter --target 6875 -n
[root@izwz9314kt10onbwuw6odez ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether d2:cc:e9:b1:f6:9d brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.20.2.21/25 brd 172.20.2.127 scope global eth0
valid_lft forever preferred_lft forever
对比一下第一步拿到的pod ip 是一样的
4.使用 tcpdump 抓包，指定 eth0 网卡
直接抓包尝试
tcpdump -i eth0 tcp and port 80 -vvv
新开一个窗口进入到pod里面配合访问测试
# kubectl exec -it busybox-5fc755d7d9-nc8bz sh
/ # nc -vz www.baidu.com 80
www.baidu.com (14.215.177.38:80) open
5.退出network namespace
直接exit即可
# exit
logout
[root@izwz9314kt10onbwuw6odez ~]# ifconfig
cali4f23c4d62c6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.20.2.1 netmask 255.255.255.255 broadcast 172.20.2.1
ether 3e:65:de:6b:77:69 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
......
实战：模拟kubelet的liveness健康检查失败抓包
1.起个nginx的deployment，加上一个不存在的路径做liveness
livenessProbe:
failureThreshold: 5
httpGet:
path: /muyuan.html
port: 80
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 20
successThreshold: 1
timeoutSeconds: 1
2，确认node，并登陆获取pid
# kubectl get pods -o wide -l app=nginx
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
nginx-deployment-basic-58d68cc69d-cg9xr 1/1 Running 0 10s 172.20.2.233 cn-shenzhen.192.168.0.131 <none>
获取pid
# docker inspect -f {{.State.Pid}} ca32f2599302
7030
进入ns
# nsenter --target 7030 -n
3,抓到kubelet的包了
# tcpdump -i any port 80 -nnvv
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
18:33:15.996389 IP (tos 0x0, ttl 64, id 9563, offset 0, flags [DF], proto TCP (6), length 60)
172.20.2.129.34098 > 172.20.2.233.80: Flags [S], cksum 0x5dc1 (incorrect -> 0x3346), seq 2310580335, win 29200, options [mss 1460,sackOK,TS val 3575092 ecr 0,nop,wscale 9], length 0
18:33:15.996426 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
172.20.2.233.80 > 172.20.2.129.34098: Flags [S.], cksum 0x5dc1 (incorrect -> 0xdef8), seq 2455778657, ack 2310580336, win 28960, options [mss 1460,sackOK,TS val 3575092 ecr 3575092,nop,wscale 9], length 0
18:33:15.996448 IP (tos 0x0, ttl 64, id 9564, offset 0, flags [DF], proto TCP (6), length 52)
172.20.2.129.34098 > 172.20.2.233.80: Flags [.], cksum 0x5db9 (incorrect -> 0x7ead), seq 1, ack 1, win 58, options [nop,nop,TS val 3575092 ecr 3575092], length 0
18:33:15.996591 IP (tos 0x0, ttl 64, id 9565, offset 0, flags [DF], proto TCP (6), length 176)
172.20.2.129.34098 > 172.20.2.233.80: Flags [P.], cksum 0x5e35 (incorrect -> 0x3eab), seq 1:125, ack 1, win 58, options [nop,nop,TS val 3575092 ecr 3575092], length 124: HTTP, length: 124
GET /muyuan.html HTTP/1.1
Host: 172.20.2.233:80
User-Agent: kube-probe/1.12+
Accept-Encoding: gzip
Connection: close
18:33:15.996611 IP (tos 0x0, ttl 64, id 47190, offset 0, flags [DF], proto TCP (6), length 52)
172.20.2.233.80 > 172.20.2.129.34098: Flags [.], cksum 0x5db9 (incorrect -> 0x7e32), seq 1, ack 125, win 57, options [nop,nop,TS val 3575092 ecr 3575092], length 0
18:33:15.996721 IP (tos 0x0, ttl 64, id 47191, offset 0, flags [DF], proto TCP (6), length 369)
172.20.2.233.80 > 172.20.2.129.34098: Flags [P.], cksum 0x5ef6 (incorrect -> 0xe1af), seq 1:318, ack 125, win 57, options [nop,nop,TS val 3575092 ecr 3575092], length 317: HTTP, length: 317
HTTP/1.1 404 Not Found
Server: nginx/1.7.9
Date: Tue, 22 Oct 2019 10:33:15 GMT
Content-Type: text/html
Content-Length: 168
Connection: close
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.7.9</center>
</body>
</html>
18:33:15.996747 IP (tos 0x0, ttl 64, id 9566, offset 0, flags [DF], proto TCP (6), length 52)
本文为阿里云内容，未经允许不得转载。
网友评论