容器安全-容器网络

作者: 潘猛_9f76 | 来源:发表于2020-06-16 18:03 被阅读0次

容器安全-容器网络
docker 收录
docker容器之间通讯-network
配置网络
容器（11）----容器网络
同步容器
容器coredns问题排查整理
swarm集群
Docker 入门教程之容器网络
容器安全

1、docker network

从4种容器中各取一个来解释容器网络：

1、docker run --network=host启动的container
2、docker run启动的container
3、kubernetes启动的属性pod.spec.hostNetwork:True的Pod中的pause container
4、kubernetes启动的Pod中的的pause container
注意：pod中的网络由基础容器Pause构建
查看容器，为了描述清楚，4种容器简称 C1H[b42997252c7d],C1[43fc60ebaec1],P1[c8136af65b74],P1H[2f15f6bbc20d]

# docker ps -a
CONTAINER ID        IMAGE                                                           COMMAND                  CREATED             STATUS              PORTS               NAMES
b42997252c7d        centos                                                          "top"                    17 minutes ago      Up 17 minutes                           hopeful_swartz
43fc60ebaec1        centos                                                          "/bin/top"               22 minutes ago      Up 22 minutes                           distracted_noyce
c8136af65b74        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1   "/pause"                 4 days ago          Up 4 days                                       k8s_POD_coredns-7f9c544f75-j9bg2_kube-system_8015b244-d496-4639-821b-64d7ddeb2475_1
2f15f6bbc20d        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1   "/pause"                 4 days ago           Up 4 days                                       k8s_POD_kube-apiserver-master1_kube-system_a5600473b2823e758eacb4dc23ac9634_1

Docker容器中有3种网络:

bridge
host
none

# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
b611a305447b        bridge              bridge              local
e7b7cc1d481e        host                host                local
0a749951e537        none                null                local

通过docker inspec network，可以看到每个network具体有哪些container在使用。
查看网络bridge，可以看到容器“43fc60ebaec1”，即C1[43fc60ebaec1]

# docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "b611a305447b8c009e2bf37efb0ca9457b4260590cf579169853b4224b01daae",
        "Created": "2020-06-11T21:56:29.6621377-04:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "43fc60ebaec11202a2ee176ac4b18c5863a3b4dde66f7841f4459b61ec18b828": {
                "Name": "distracted_noyce",
                "EndpointID": "036404f59d338df87d22509950ab851c800c7080665532748809889bccb62761",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

查看网络host，可以看到容器“b42997252c7d”和“2f15f6bbc20d”,即C1H[b42997252c7d]和P1H[2f15f6bbc20d]

# docker network inspect  host
[
    {
        "Name": "host",
        "Id": "e7b7cc1d481effbf41a842f3556b058ea5b2010e5215a2b5f71a5dcbf9fccc7b",
        "Created": "2020-05-12T23:31:31.06973004-04:00",
        "Scope": "local",
        "Driver": "host",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "2f15f6bbc20d70123517cd23a1ef26af8baf3e665adf82605534a872d55ff0e5": {
                "Name": "k8s_POD_kube-apiserver-master1_kube-system_a5600473b2823e758eacb4dc23ac9634_1",
                "EndpointID": "b175e67c3b2a4b04aacc244d597c57d13826ad4b170bf68079abea8f7f696688",
                "MacAddress": "",
                "IPv4Address": "",
                "IPv6Address": ""
            },
            "b42997252c7d9ea96c43cf4d7428460b04d8ef301f21e235ec1d927de7fed2bd": {
                "Name": "hopeful_swartz",
                "EndpointID": "306a24024509ebf9e3ffa7ceaeb18813c90288cfb418dc11e7ec362685444061",
                "MacAddress": "",
                "IPv4Address": "",
                "IPv6Address": ""
            },
        },
        "Options": {},
        "Labels": {}
    }
]

查看网络none，可以看到容器“b42997252c7d”，即P1[c8136af65b74]

# docker network inspect none
[
    {
        "Name": "none",
        "Id": "0a749951e5376b6d24aac6e92c3ea16c64e833e1636a73ddc960e2475199a664",
        "Created": "2020-05-12T23:31:31.056449662-04:00",
        "Scope": "local",
        "Driver": "null",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "c8136af65b7499be11c281660701a1406ce55c9e8c4dcbab0f70b850c2546b93": {
                "Name": "k8s_POD_coredns-7f9c544f75-j9bg2_kube-system_8015b244-d496-4639-821b-64d7ddeb2475_1",
                "EndpointID": "c52ed427178c80cb0dca7585324f14b95a448266adff1affc4756187827e8050",
                "MacAddress": "",
                "IPv4Address": "",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

综上所述：
docker run 启动的容器使用 bridge网络
k8s pod 使用的是 none网络
使用了--network=host的容器或者pod.spec.hostNetwork:True的pod 使用的是 host 网络

2、Network Namespace

通过进程查看各个容器的net namespace，可以看到C1H[b42997252c7d]和P1H[2f15f6bbc20d]是一样的

# docker inspect b42997252c7d -f '{{.State.Pid}}' | xargs -i ls -l /proc/{}/ns | grep net
lrwxrwxrwx 1 root root 0 Jun 16 03:25 net -> net:[4026531992]
# docker inspect 43fc60ebaec1 -f '{{.State.Pid}}' | xargs -i ls -l /proc/{}/ns | grep net
lrwxrwxrwx 1 root root 0 Jun 16 02:23 net -> net:[4026533125]
# docker inspect c8136af65b74 -f '{{.State.Pid}}' | xargs -i ls -l /proc/{}/ns | grep net
lrwxrwxrwx 1 root root 0 Jun 11 21:57 net -> net:[4026532809]
# docker inspect 2f15f6bbc20d -f '{{.State.Pid}}' | xargs -i ls -l /proc/{}/ns | grep net
lrwxrwxrwx 1 root root 0 Jun 11 21:56 net -> net:[4026531992]

重点关注C1[43fc60ebaec1],P1[c8136af65b74]

# docker inspect c8136af65b74 -f '{{.NetworkSettings.SandboxKey}}'
/var/run/docker/netns/2e1e5b20f82b
# docker inspect 43fc60ebaec1 -f '{{.NetworkSettings.SandboxKey}}'
/var/run/docker/netns/60d249f742b1

这里可以看到关于C1[43fc60ebaec1],P1[c8136af65b74]的namespace有4个

/proc/234319/ns/net
/proc/4547/ns/net
/var/run/docker/netns/2e1e5b20f82b
/var/run/docker/netns/60d249f742b1
使用系统的network namespace命令，却看不到任何netns的信息

# ip netns list

这是因为docker对容器的network namespace进行了隐藏，可以手动开启

# mkdir /var/run/netns/
# ln -s /proc/4547/ns/net /var/run/netns/c8136af65b74
# ln -s /proc/234319/ns/net /var/run/netns/43fc60ebaec1
# ln -s /var/run/docker/netns/2e1e5b20f82b /var/run/netns/1
# ln -s /var/run/docker/netns/60d249f742b1 /var/run/netns/4

再次查看netns，可以看到与C1[43fc60ebaec1]关联的60d249f742b1和net:[4026533125]就是同一个netns

# ip netns list
4 (id: 4)
1 (id: 1)
c8136af65b74 (id: 1)
43fc60ebaec1 (id: 4)

在宿主机上查看网络连接

# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:d6:da:c4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.232.51/24 brd 192.168.232.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::d857:61d7:c87a:20b0/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:eb:b1:dd:12 brd ff:ff:ff:ff:ff:ff
4: tunl0@NONE: <NOARP> mtu 1440 qdisc noqueue state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
8: caliea81128d49b@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
167: vetha459ebf@if166: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
    link/ether 4a:d6:77:a0:34:53 brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet6 fe80::48d6:77ff:fea0:3453/64 scope link
       valid_lft forever preferred_lft forever

部分连接上有link-netnsid，对应veth连接的namespace的id

caliea81128d49b@if4  link-netnsid 1
vetha459ebf@if166    link-netnsid 4

在命名空间查看ip地址和link

# ip netns exec c8136af65b74  ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default
    link/ether 1a:72:e5:67:88:cc brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.16.137.92/32 scope global eth0
       valid_lft forever preferred_lft forever
# ip netns exec 43fc60ebaec1  ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
166: eth0@if167: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

在kubernetes和docker中查看容器的ip地址,可以看到与namespace中查看的一致

# kubectl get pods -o wide  -n kube-system | grep -i coredns
coredns-7f9c544f75-j9bg2                   1/1     Running   2          34d   172.16.137.92    master1   <none>           <none>
# docker inspect 43fc60ebaec1 -f '{{.NetworkSettings.IPAddress}}'
172.17.0.2

3、容器中的veth连接

image.png

查看容器P1 namespace中的link和宿主中的link，可以看见是一对veth，@后面为对端的id

# ip link show caliea81128d49b
8: caliea81128d49b@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP mode DEFAULT group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns 1
# ip netns exec 1 ip link show eth0
4: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP mode DEFAULT group default
    link/ether 1a:72:e5:67:88:cc brd ff:ff:ff:ff:ff:ff link-netnsid 0

查看容器C1 namespace中的link和宿主中的link，可以看见是一对veth，@后面为对端的id,注意"master docker0",表示veth还连接着bridge docker0

# ip link show vetha459ebf
167: vetha459ebf@if166: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default
    link/ether 4a:d6:77:a0:34:53 brd ff:ff:ff:ff:ff:ff link-netns 4
# ip netns exec 4 ip link show eth0
166: eth0@if167: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0

从ID看veth的匹配关系，有两个id为4的link，怎么确定匹配关系？

image.png

不仅要看id，还需要是veth类型的link

image.png
为什么pod网络的mtu为1440，而不是1500？
calico网络可以使用各种封装，比如缺省的ipip封装，需要多增加20字节的ip头，所以需要适当的减少mtu

image.png

容器安全-容器网络
1、docker network 从4种容器中各取一个来解释容器网络： 1、docker run --networ...
docker 收录
查看容器ip 传送文件容器 -> 宿主宿主 -> 容器进入容器错误日志查看网络访问容器访问宿主网络，输...
docker容器之间通讯-network
docker为容器创建独立的网络环境，实现宿主和容器、容器之间的网络隔离，默认使用bridge模式的网络，实现容器...
配置网络
概述容器网络是容器选择连接到其他容器、主机和外部网络的机制。容器的 runtime 提供了各种网络模式，每种模式...
容器（11）----容器网络
Docker使用Linux桥接，在宿主机虚拟一个Docker容器网桥(docker0)，Docker启动一个容器时...
同步容器
同步容器用于解决并发情况下的容器线程安全问题，可以给多线程环境准备一个线程安全的容器对象。线程安全的容器对象： V...
容器coredns问题排查整理
简介：容器coredns问题排查整理 1.问题描述客户侧在变更容器安全组之后出现网络不通。 2.问题排查 1）接...
swarm集群
容器集群 swarm 容器生态包含三个部分容器核心知识架构、镜像、容器、网络、存储容器平台技术容...
Docker 入门教程之容器网络
什么是容器网络容器网络是 Docker 用来解决相互隔离的容器之间通信问题的技术。 Docker 网络驱动的类型...
容器安全
gVisor简介与实战 gVisor的功能并不是来取代docker，而是像一个docker的低级运行时，它跟普通的...