Vagrant搭建k8s测试集群(Flannel网络方案)

作者: 0d1b415a365b | 来源:发表于2019-08-12 20:38 被阅读0次

提要

本文介绍如何通过 Vagrant 快速搭建 k8s 测试集群(参照官方博客https://kubernetes.io/blog/2019/03/15/kubernetes-setup-using-ansible-and-vagrant/，有改动，网络组件从 calico 改为 flannel )，以及在搭建过程中碰到的坑。需要对 Vagrant， K8s，Ansible 有一些基础知识，一些配置/脚本文件不会做详细说明

安装 Vagrant 和 VBox

https://www.vagrantup.com/downloads.html
https://www.virtualbox.org/wiki/Downloads
去此处下载对应系统的安装包，直接安装即可

创建 Vagrant File

任意位置创建一个目录，作为 Vagrant 工作目录，在此目录下创建文件 Vagrant：

IMAGE_NAME = "bento/ubuntu-16.04"                                                                                                                                                                                                             
N = 2
 
Vagrant.configure("2") do |config|
    config.ssh.insert_key = false
 
    config.vm.provider "virtualbox" do |v|
        v.memory = 1024
        v.cpus = 2
    end
 
    config.vm.define "k8s-master" do |master|
        master.vm.box = IMAGE_NAME
        master.vm.network "private_network", ip: "192.168.50.10"
        master.vm.hostname = "k8s-master"
        master.vm.provision "ansible" do |ansible|
            ansible.playbook = "setup/master-playbook.yml"
            ansible.extra_vars = {
                node_ip: "192.168.50.10",
            }
        end
    end
 
    (1..N).each do |i|
        config.vm.define "node-#{i}" do |node|
            node.vm.box = IMAGE_NAME
            node.vm.network "private_network", ip: "192.168.50.#{i + 10}"
            node.vm.hostname = "node-#{i}"
            node.vm.provision "ansible" do |ansible|
                ansible.playbook = "setup/node-playbook.yml"
                ansible.extra_vars = {
                    node_ip: "192.168.50.#{i + 10}",
                }
            end
        end
    end
end

创建搭建 K8s 的 Ansible Playbook

此步骤需要神秘力量科学上网
在 vagrant 工作目录下建立 setup 目录，让后进入该目录
创建 master 节点部署脚本：master-playbook.yml
该脚本会自动安装 docker 和 kubelet， kubeadm， kubectl 等组件，并生成 kubeadm join 命令到 vagrant 工作目录，如果某步骤失败，可以自己参照脚本手动执行

---
- hosts: all
  become: true
  tasks:
  - name: Install packages that allow apt to be used over HTTPS
    apt:
      name: "{{ packages }}"
      state: present
      update_cache: yes
    vars:
      packages:
      - apt-transport-https
      - ca-certificates
      - curl
      - gnupg-agent
      - software-properties-common

  - name: Add an apt signing key for Docker
    apt_key:
      url: https://download.docker.com/linux/ubuntu/gpg
      state: present

  - name: Add apt repository for stable version
    apt_repository:
      repo: deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable
      state: present

  - name: Install docker and its dependecies
    apt: 
      name: "{{ packages }}"
      state: present
      update_cache: yes
    vars:
      packages:
      - docker-ce 
      - docker-ce-cli 
      - containerd.io
    notify:
      - docker status

  - name: Add vagrant user to docker group
    user:
      name: vagrant
      group: docker

  - name: Remove swapfile from /etc/fstab
    mount:
      name: "{{ item }}"
      fstype: swap
      state: absent
    with_items:
      - swap
      - none

  - name: Disable swap
    command: swapoff -a
    when: ansible_swaptotal_mb > 0

  - name: Add an apt signing key for Kubernetes
    apt_key:
      url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
      state: present

  - name: Adding apt repository for Kubernetes
    apt_repository:
      repo: deb https://apt.kubernetes.io/ kubernetes-xenial main
      state: present
      filename: kubernetes.list

  - name: Install Kubernetes binaries
    apt: 
      name: "{{ packages }}"
      state: present
      update_cache: yes
    vars:
      packages:
        - kubelet 
        - kubeadm 
        - kubectl

  - name: touch kubelet config file
    file:        
      path: /etc/default/kubelet
      state: touch 

  - name: Configure node ip
    lineinfile:
      path: /etc/default/kubelet
      line: KUBELET_EXTRA_ARGS=--node-ip={{ node_ip }}

  - name: Restart kubelet
    service:
      name: kubelet
      daemon_reload: yes
      state: restarted

  - name: Initialize the Kubernetes cluster using kubeadm
    command: kubeadm init --apiserver-advertise-address="192.168.50.10" --apiserver-cert-extra-sans="192.168.50.10"  --node-name k8s-master --pod-network-cidr=10.244.0.0/16

  - name: Setup kubeconfig for vagrant user
    command: "{{ item }}"
    with_items:
     - mkdir -p /home/vagrant/.kube
     - cp -i /etc/kubernetes/admin.conf /home/vagrant/.kube/config
     - chown vagrant:vagrant /home/vagrant/.kube/config

  - name: Install calico pod network
    become: false
    command: kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

  - name: Generate join command
    command: kubeadm token create --print-join-command
    register: join_command

  - name: Copy join command to local file
    local_action: copy content="{{ join_command.stdout_lines[0] }}" dest="./join-command"

  handlers:
    - name: docker status
      service: name=docker state=started

创建 Minion Node 的部署脚本 node.yml
同样，这个脚本会在 Minion 节点上安装 docker 和 kube 组件，并执行 Master 节点生成的 join 命令加入集群，如果某步骤出现问题可参照脚本手动修正。

---
- hosts: all
  become: true
  tasks:
  - name: Install packages that allow apt to be used over HTTPS
    apt:
      name: "{{ packages }}"
      state: present
      update_cache: yes
    vars:
      packages:
      - apt-transport-https
      - ca-certificates
      - curl
      - gnupg-agent
      - software-properties-common

  - name: Add an apt signing key for Docker
    apt_key:
      url: https://download.docker.com/linux/ubuntu/gpg
      state: present

  - name: Add apt repository for stable version
    apt_repository:
      repo: deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable
      state: present

  - name: Install docker and its dependecies
    apt: 
      name: "{{ packages }}"
      state: present
      update_cache: yes
    vars:
      packages:
      - docker-ce 
      - docker-ce-cli 
      - containerd.io
    notify:
      - docker status

  - name: Add vagrant user to docker group
    user:
      name: vagrant
      group: docker

  - name: Remove swapfile from /etc/fstab
    mount:
      name: "{{ item }}"
      fstype: swap
      state: absent
    with_items:
      - swap
      - none

  - name: Disable swap
    command: swapoff -a
    when: ansible_swaptotal_mb > 0

  - name: Add an apt signing key for Kubernetes
    apt_key:
      url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
      state: present

  - name: Adding apt repository for Kubernetes
    apt_repository:
      repo: deb https://apt.kubernetes.io/ kubernetes-xenial main
      state: present
      filename: kubernetes.list

  - name: Install Kubernetes binaries
    apt: 
      name: "{{ packages }}"
      state: present
      update_cache: yes
    vars:
      packages:
        - kubelet 
        - kubeadm 
        - kubectl

  - name: touch kubelet config file
    file:
      path: /etc/default/kubelet
      state: touch

  - name: Configure node ip
    lineinfile:
      path: /etc/default/kubelet
      line: KUBELET_EXTRA_ARGS=--node-ip={{ node_ip }}

  - name: Restart kubelet
    service:
      name: kubelet
      daemon_reload: yes
      state: restarted

  - name: Copy the join command to server location
    copy: src=join-command dest=/tmp/join-command.sh mode=0777

  - name: Join the node to cluster
    command: sh /tmp/join-command.sh

  handlers:
    - name: docker status
      service: name=docker state=started

启动集群

vagrant up --provider=virtualbox

通过 Vagrant File 的定义，我们会创建一个 1 Master 2 Minion 的集群，系统使用的是 Ubuntu 16.04
如果镜像下载比较慢，可以尝试手动从官方仓库下载，地址如下
https://app.vagrantup.com/boxes/search
或者从国内仓库下载 .box 文件，不过国内仓库版本信息不全，很难找到合适的（需要安装 python），需要耐心尝试，国内仓库：
https://mirrors.tuna.tsinghua.edu.cn/ubuntu-cloud-images/
下载完成后通过 vagrant add 添加到本地, 再up

vagrant box add ~/Download/virtualbox.box --name bento/ubuntu-16.04

如果一切顺利的话，到现在为止已经拥有了一个可以用于测试的 k8s 集群。

集群验证

vagrant ssh k8s-master
vagrant@k8s-master:~$ kubectl get nodes -o wide
NAME         STATUS   ROLES    AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k8s-master   Ready    master   2d23h   v1.15.2   192.168.50.10   <none>        Ubuntu 16.04.6 LTS   4.4.0-150-generic   docker://19.3.1
node-1       Ready    <none>   2d21h   v1.15.2   192.168.50.11   <none>        Ubuntu 16.04.6 LTS   4.4.0-150-generic   docker://19.3.1
node-2       Ready    <none>   2d21h   v1.15.2   10.0.2.15       <none>        Ubuntu 16.04.6 LTS   4.4.0-150-generic   docker://19.3.1

Minion 节点已经成功加入集群

vagrant@k8s-master:~$ alias ksys="kubectl -n kube-system"
vagrant@k8s-master:~$ ksys get pods -o wide
NAME                                 READY   STATUS    RESTARTS   AGE     IP              NODE         NOMINATED NODE   READINESS GATES
coredns-5c98db65d4-565hf             1/1     Running   0          3d1h    10.244.0.3      k8s-master   <none>           <none>
coredns-5c98db65d4-gtflf             1/1     Running   0          3d1h    10.244.0.2      k8s-master   <none>           <none>
etcd-k8s-master                      1/1     Running   0          3d1h    192.168.50.10   k8s-master   <none>           <none>
kube-apiserver-k8s-master            1/1     Running   0          3d1h    192.168.50.10   k8s-master   <none>           <none>
kube-controller-manager-k8s-master   1/1     Running   0          3d1h    192.168.50.10   k8s-master   <none>           <none>
kube-flannel-ds-amd64-fbpqj          1/1     Running   0          2d23h   192.168.50.10   k8s-master   <none>           <none>
kube-flannel-ds-amd64-jbtt6          1/1     Running   0          2d23h   192.168.50.11   node-1       <none>           <none>
kube-flannel-ds-amd64-rjvqg          1/1     Running   0          2d23h   10.0.2.15       node-2       <none>           <none>
kube-proxy-hw977                     1/1     Running   0          3d1h    192.168.50.10   k8s-master   <none>           <none>
kube-proxy-p9z84                     1/1     Running   0          2d23h   10.0.2.15       node-2       <none>           <none>
kube-proxy-vg6ms                     1/1     Running   0          2d23h   192.168.50.11   node-1       <none>           <none>

flannel 等组件已成功启动

Pods 通信验证

在上文的 Master 节点 playbook 中，指定了 Pods 的子网为 10.244.0.0/16
启动三个 Pods：

kubectl run -i -t busybox --image=busybox
kubectl run -i -t busybox2 --image=busybox
kubectl run -i -t busybox3 --image=busybox

查看 Pods：

vagrant@k8s-master:~$ kubectl get po -o wide
NAME                        READY   STATUS    RESTARTS   AGE     IP           NODE     NOMINATED NODE   READINESS GATES
busybox-767bf65c49-mzm9v    1/1     Running   1          2d23h   10.244.1.2   node-1   <none>           <none>
busybox2-868b98bf99-bf6sz   1/1     Running   0          2d23h   10.244.2.2   node-2   <none>           <none>
busybox3-7f44486c85-jk7kh   1/1     Running   1          2d23h   10.244.1.3   node-1   <none>           <none>

两个 Pods 已经正常启动，并且分配了正确的 IP

同节点 Pods 通信验证

vagrant@k8s-master:~$ kubectl exec -it busybox-767bf65c49-mzm9v /bin/sh
/ # ping 10.244.1.3
PING 10.244.1.3 (10.244.1.3): 56 data bytes
64 bytes from 10.244.1.3: seq=0 ttl=64 time=0.275 ms
64 bytes from 10.244.1.3: seq=1 ttl=64 time=0.312 ms
64 bytes from 10.244.1.3: seq=2 ttl=64 time=0.094 ms

跨界点 Pods 通信验证

vagrant@k8s-master:~$ kubectl exec -it busybox-767bf65c49-mzm9v /bin/sh
/ # ping 10.244.2.2
PING 10.244.2.2 (10.244.2.2): 56 data bytes
^C
--- 10.244.2.2 ping statistics ---
6 packets transmitted, 0 packets received, 100% packet loss

问题来了，跨节点无法互通。
查看各节点网卡及路由
master：

interface	ip
cni0	10.244.0.1
docker0	172.17.0.1
eth0	10.0.2.15
eth1	192.168.50.10
flannel.1	10.244.0.0

node-1:

interface	ip
cni0	10.244.1.1
docker0	172.17.0.1
eth0	10.0.2.15
eth1	192.168.50.11
flannel.1	10.244.1.0

node-2:

interface	ip
cni0	10.244.2.1
docker0	172.17.0.1
eth0	10.0.2.15
eth1	192.168.50.12
flannel.1	10.244.2.0

node-1 的路由，其他节点差不多，不一一列举

vagrant@node-1:~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.2.2        0.0.0.0         UG    0      0        0 eth0
10.244.0.0      10.244.0.0      255.255.255.0   UG    0      0        0 flannel.1
10.244.1.0      0.0.0.0      255.255.255.0   UG    0      0        0 cni0
10.244.2.0      10.244.2.0         255.255.255.0   U     0      0        0 flannel.1
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.50.0    0.0.0.0         255.255.255.0   U     0      0        0 eth1

发现都配置了通过 flannel.1 前往其他节点子网的路由信息，参照 https://blog.csdn.net/u011394397/article/details/54584819 此文后尝试添加大网路由

sudo route add -net 10.244.0.0/16 dev flannel.1

结果还是不通
查了半天资料都没有解决，所以接下来仔细分析一下网络模型。

Vagrant 部署 k8s 集群跨节点 Pods 网络不通问题调查

网上对于 flannel 网络模型差不多都是用的下图：

image.png

然而这个图已经太过古老，跟现在的网络对应不上，缺少 cni0 网桥，而且通过 tcpdump 对 ping 包进行跟踪，发现 ping 包无论如何都走不到 docker0，同节点的 ping 包只会通过 cni0，跨节点的 ping 包终止于 flannel.1 但并没有经过 docker0，并且达不到另一个 node。
经过多放查找，目前最新的网络结构图如下：

image.png
此图和最新的 k8s cni 网络终于对上了。

查阅相关资料发现 k8s 早已抛弃了 docker0 网桥，改用自己的 cni0 网桥来管理 Pods 网络，所以 docker0 可以无视。
一个数据包从 Pod 里发出后经过 cni 和 flannel 的处理和路由，会最终通过宿主机的 eth0 接口送到集群网络。

然而通过上文列出的各节点网卡可以发现，三个节点的 eth0 都是 10.0.2.15。。明明创建 VM 时指定的是192.168.50.0/24 啊？那试着把默认路由改到 eth1？

sudo route delete default
sudo route add default dev eth1

然后再次验证。结果。。。节点间依然不通，还导致节点不通外网。这是什么鬼？
回过头想想。eth0 的 10.0.2.0/24 网络是哪里来的？打开 vbox 查看，原来是 vagrant 给自动分配了一个 NAT 网口，NAT 网口的特点是

VM 可以 ping 通宿主机
宿主机不能 ping 通 VM
VM 之间不能通信
宿主机可通外网，则 VM 也可

vagrant 管理 VM 是通过 Vagrant File 中指定 ip 的 host-only 网络，即 VM 的 eth1 口，此网络的特点

VM 之间可互通
VM 与宿主机可以互通（VM 的 eth1 口挂在宿主机 vboxnet0 虚拟网卡）
VM 不通外网

所以到这里也就明了，Pods 跨节点无法通信完全是 vagrant NAT 网络的锅，三个节点的 eth0 本来就不是互通的，所以我们把 eth0 改为桥接模式即可。

Vagrant搭建k8s测试集群(Flannel网络方案)

提要

安装 Vagrant 和 VBox

创建 Vagrant File

创建搭建 K8s 的 Ansible Playbook

启动集群

集群验证

Pods 通信验证

同节点 Pods 通信验证

跨界点 Pods 通信验证

Vagrant 部署 k8s 集群跨节点 Pods 网络不通问题调查

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读