前言
这篇文章主要想探讨一下数据包是如何经过iptables的4表5链最终到达Docker容器内部。
由于启用firewalld之后Docker会通过firewall-cmd来操作iptables,暂时不讨论这种情况，只讨论Docker直接操作iptables的这种，所以以下内容是在先关闭firewall后进行的操作(很重要)
由于数据包进入netfilter内核有2个入口(参考最经典的iptables processing flowchart图，下面有)，所以我们分别从外部机器和本机访问映射出来的nginx容器的80端口来查看数据包走向。
先说结论
docker分别在nat表的prerouting output postrouting这3个链和filter表的forward链新建规则，主要实现DNAT，SNAT，访问控制，网络隔离等功能。
且外部数据包进入后的关键链为nat[prerouting] -> filter[forward] -> nat[postrouing]
本地进程出的包经过关键链为nat[output]
那Docker在每一链中都做了些什么呢？我们从数据包来源分成2个部分来说明
主机外部数据包进入容器后相关链及作用：
| 表 | 链 | 操作 |
|---|---|---|
| nat | prerouting | DNAT,目的地址端口转换为容器ip和端口 |
| filter | forward | 自定义DOCKER-USER，DOCKER-ISOLATION-STAGE，DOCKER这3条链 |
| nat | postrouting | SNAT,利用MASQUERADE转换源地址 |
其中
DOCKER-USER链为Docker定义给用户用来添加自定义规则来限制访问策略，这里官网有说明
DOCKER-ISOLATION-STAGE链用来实现Docker多network中的容器互相隔离，不能进行互通
DOCKER链用来实现用户配置的端口映射策略，如192.168.1.2:80映射到容器80端口
主机内进程数据包进入容器后相关链及作用：
| 表 | 链 | 操作 |
|---|---|---|
| nat | output | DNAT,目的地址端口转换为容器ip和端口 |
| nat | postrouting | SNAT,利用MASQUERADE转换源地址 |
只做了DNAT
可以看到其实很简单，无非就是转换地址之类的，因为Docker在这之前已经为每个容器网络建立了独立的网桥，配置了路由等等，不过这里篇幅有限就不讨论这些了。下面我们就进行实验，看如何得出这些结论的。
实验准备工作
- 我们在一台新的机器(centos)上关闭防护墙，安装好docker，并启动一个nginx容器，映射为80端口。
systemctl stop firewalld
systemctl start docker
docker run -d -p 80:80 nginx
现在我们查看iptables的nat和filter表，有如下输出：
[root@localhost ~]# iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 2 packets, 289 bytes)
pkts bytes target prot opt in out source destination
7 508 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 2 packets, 289 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 34 packets, 2399 bytes)
pkts bytes target prot opt in out source destination
4 240 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 42 packets, 2879 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
0 0 MASQUERADE tcp -- * * 172.17.0.2 172.17.0.2 tcp dpt:80
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
8 480 DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:172.17.0.2:80
[root@localhost ~]# iptables -nvL -t filter
Chain INPUT (policy ACCEPT 4553 packets, 902K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
44 4498 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
44 4498 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
20 2105 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
4 240 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
20 2153 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 6081 packets, 6094K bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
4 240 ACCEPT tcp -- !docker0 docker0 0.0.0.0/0 172.17.0.2 tcp dpt:80
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
20 2153 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
44 4498 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * docker0 0.0.0.0/0 0.0.0.0/0
20 2153 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
44 4498 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
可以看到，docker分别在nat表的prerouting output postrouting这3个链和filter表的forward链添加了内容，后面的实验我们只需要关注这几条链即可。
- 要想看到数据包流经了哪些表和链，还需要启用
ipt_LOG内核模块来追踪数据包并打印日志，使用以下命令来跟踪到达本机80端口的数据包
modprobe nf_log_ipv4
sysctl net.netfilter.nf_log.2=nf_log_ipv4
iptables -t raw -A PREROUTING -p tcp --dport 80 -j TRACE
iptables -t raw -A OUTPUT -p tcp --dport 80 -j TRACE
-
同时打开最经典的iptables数据流图参考
netfilter数据流图
再打开一个终端查看日志tail -n0 -f /var/log/messages
嗯，至此准备工作就完成了。
分析数据包
还是分2种情况，数据包从主机外部进入和数据包从主机内进程发出(当然都是发给容器的)
直接在本机之外的机器打一个请求过来
curl http://192.168.233.147/
立即查看刚刚打开终端的日志/var/log/messages
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: nat:PREROUTING:rule:1 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: nat:DOCKER:rule:2 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:4 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: nat:POSTROUTING:policy:3 IN= OUT=docker0 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:3 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=130 TOS=0x00 PREC=0x00 TTL=64 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:3 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:3 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:3 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:3 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
有很多重复，应该是一个http请求通过tcp连接传输多个包，每个包都需要走一次iptables，我们只看前几个不重复的即可。
可以看到，数据包经过的顺序为
raw:PREROUTING -> nat:PREROUTING -> filter:FORWARD -> security:FORWARD -> nat:POSTROUTING
其余带Docker字样的链为各个链中的自定义链，看到这里在对比数据流图，才发现这张图真的诚不欺我，数据包流向和图中一模一样(终于实践验证理论)。
图上大概长这样，可以看到是走转发这条路，并不是直接到达本地进程，这也是为什么安装Docker时，要启用内核数据包转发！
而且一般在filter:INPUT链添加安全策略的做法对docker也是不生效的，因为访问容器的数据包根本不走filter:INPUT。
image.png
那本机进程发出的数据包走向如何呢？
直接在本机打一个请求过来测试
curl http://192.168.233.147/
查看日志
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: nat:OUTPUT:rule:1 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: nat:DOCKER:rule:2 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: nat:POSTROUTING:policy:3 IN= OUT=docker0 SRC=192.168.233.147 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49914 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49914 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49914 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=49915 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK PSH URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=49915 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK PSH URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=49915 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK PSH URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49916 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741273973 WINDOW=350 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49916 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741273973 WINDOW=350 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49916 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741273973 WINDOW=350 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49917 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49917 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49917 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49918 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK FIN URGP=0 OPT (0101080A016E02D4016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49918 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK FIN URGP=0 OPT (0101080A016E02D4016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49918 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK FIN URGP=0 OPT (0101080A016E02D4016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49919 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932986 ACK=2741274589 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D4016E02D4)
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49919 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932986 ACK=2741274589 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D4016E02D4)
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49919 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932986 ACK=2741274589 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D4016E02D4)
可以看到，数据包经过的顺序为
raw:OUTPUT -> nat:OUTPUT -> filter:OUTPUT -> security:OUTPUT
图上大概长这样
image.png
在
nat:OUTPUT链做了DNAT和nat:POSTROUTING链做了SNAT至此，数据包流向就分析完了。
做完实验收获最大的应该还是如何跟踪数据包走向，毕竟有了这项技能，网络不通也可以debug找原因了。
感谢互联网的前辈们，他们早已把坑踩平，我们只需要找到他们的脚印。
参考
https://backreference.org/2010/06/11/iptables-debugging/
https://docs.docker.com/network/packet-filtering-firewalls/#add-iptables-policies-before-dockers-rules
网友评论