Centos 二进制部署k8s （二）Flannel

工作原理

image.png

Falnnel要用etcd存储自身一个子网信息，所以要保证能成功连接Etcd，写入预定义子网段：

#要在etcd证书目录中执行。
/opt/etcd/bin/etcdctl \
--ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem \
--endpoints="https://22.22.22.10:2379,https://22.22.22.11:2379,https://22.22.22.12:2379" \
set /coreos.com/network/config  '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'

下载二进制包

# wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
# tar zxvf flannel-v0.9.1-linux-amd64.tar.gz
# mkdir /opt/kubernetes/{bin,cfg,ssl} -p
# mv flanneld mk-docker-opts.sh /opt/kubernetes/bin

配置Flannel：

# cat << Rainy > /opt/kubernetes/cfg/flanneld
FLANNEL_OPTIONS="--etcd-endpoints=https://22.22.22.10:2379,https://22.22.22.11:2379,https://22.22.22.12:2379 -etcd-cafile=/opt/etcd/ssl/ca.pem -etcd-certfile=/opt/etcd/ssl/server.pem -etcd-keyfile=/opt/etcd/ssl/server-key.pem"
Rainy

systemd管理Flannel：

[root@k8s-master k8s]# vim /usr/lib/systemd/system/flanneld.service
# cat /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure

[Install]
WantedBy=multi-user.target

配置Docker启动指定子网段：

[root@k8s-master k8s]# vim /usr/lib/systemd/system/docker.service
# cat /usr/lib/systemd/system/docker.service 
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

配置node服务器期

[root@k8s-master cfg]# scp -r /opt/kubernetes 22.22.22.11:/opt/
[root@k8s-master cfg]# scp -r /opt/kubernetes 22.22.22.12:/opt/
[root@k8s-master cfg]# scp /usr/lib/systemd/system/flanneld.service 22.22.22.11:/usr/lib/systemd/system/flanneld.service
[root@k8s-master cfg]# scp /usr/lib/systemd/system/flanneld.service 22.22.22.12:/usr/lib/systemd/system/flanneld.service
[root@k8s-master cfg]# scp /usr/lib/systemd/system/docker.service 22.22.22.11:/usr/lib/systemd/system/docker.service 
[root@k8s-master cfg]# scp /usr/lib/systemd/system/docker.service 22.22.22.12:/usr/lib/systemd/system/docker.servic

重启flannel和docker：

[root@k8s-master cfg]# systemctl daemon-reload
[root@k8s-master cfg]# systemctl start flanneld
[root@k8s-master cfg]# systemctl enable flanneld
[root@k8s-master cfg]# systemctl restart docker

检查是否生效：

[root@k8s-node2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:1c:42:01:46:9d brd ff:ff:ff:ff:ff:ff
    inet 22.22.22.12/24 brd 22.22.22.255 scope global noprefixroute dynamic eth0
       valid_lft 1642sec preferred_lft 1642sec
    inet6 fe80::81e4:2eae:e8a3:a88c/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 96:fb:ba:ca:c6:e6 brd ff:ff:ff:ff:ff:ff
    inet 172.17.46.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::94fb:baff:feca:c6e6/64 scope link 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:b2:60:ea:f9 brd ff:ff:ff:ff:ff:ff
    inet 172.17.46.1/24 brd 172.17.46.255 scope global docker0
       valid_lft forever preferred_lft forever

[root@k8s-node2 ~]# ps -ef | grep docker
root     22797     1  0 13:59 ?        00:00:00 /usr/bin/dockerd --bip=172.17.46.1/24 --ip-masq=false --mtu=1450
root     22804 22797  0 13:59 ?        00:00:01 containerd --config /var/run/docker/containerd/containerd.toml --log-level info
root     23097 22109  0 14:02 pts/2    00:00:00 grep --color=auto docker

确保docker0与flannel.1在同一网段。
测试不同节点互通，在当前节点访问另一个Node节点docker0 IP：

[root@k8s-master opt]# ping 172.17.74.1
PING 172.17.74.1 (172.17.74.1) 56(84) bytes of data.
64 bytes from 172.17.74.1: icmp_seq=1 ttl=64 time=0.554 ms
64 bytes from 172.17.74.1: icmp_seq=2 ttl=64 time=0.296 ms
64 bytes from 172.17.74.1: icmp_seq=3 ttl=64 time=0.275 ms
^C
--- 172.17.74.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.275/0.375/0.554/0.126 ms
[root@k8s-master opt]# ping 172.17.46.1
PING 172.17.46.1 (172.17.46.1) 56(84) bytes of data.
64 bytes from 172.17.46.1: icmp_seq=1 ttl=64 time=0.533 ms
64 bytes from 172.17.46.1: icmp_seq=2 ttl=64 time=0.323 ms
64 bytes from 172.17.46.1: icmp_seq=3 ttl=64 time=0.329 ms
^C
--- 172.17.46.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.323/0.395/0.533/0.097 ms