如何使用vxlan打通多集群网络

准备集群

集群1

创建vxlantest1.yaml,内容如下

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  kubeProxyMode: "ipvs"
  podSubnet: "10.244.1.0/24"
nodes:
- role: control-plane

执行如下命令

kind create cluster --image kindest/node:v1.29.0 --name vxlantest1 --config vxlantest1.yaml

集群2

创建vxlantest2.yaml,内容如下

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  kubeProxyMode: "ipvs"
  podSubnet: "10.244.2.0/24"
nodes:
- role: control-plane

执行如下命令

kind create cluster --image kindest/node:v1.29.0 --name vxlantest2 --config vxlantest2.yaml

准备负载

创建demo.yaml,内容如下

apiVersion: v1
kind: Pod
metadata:
  name: demo
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: app
  restartPolicy: Always

在两个集群中都创建负载,执行

kubectl apply -f demo.yaml

环境信息

node ip

集群1
172.18.0.2
集群2
172.18.0.3

nginx pod ip

集群1
10.244.1.5
集群2
10.244.2.5

配置网络设备

集群1

ip link add vxlantest type vxlan id 100 dstport 8473 local 172.18.0.2 dev eth0 nolearning
ip addr add 172.19.0.2/16 dev vxlantest
ip link set dev vxlantest address 02:42:ac:13:00:02
ip link set vxlantest up

集群2

ip link add vxlantest type vxlan id 100 dstport 8473 local 172.18.0.3 dev eth0 nolearning
ip addr add 172.19.0.3/16 dev vxlantest
ip link set dev vxlantest address 02:42:ac:13:00:03
ip link set vxlantest up

配置arp

如果不存在arp命令,则通过如下命令安装

apt-get install net-tools

集群1

arp -s 172.19.0.3 02:42:ac:13:00:03

集群2

arp -s 172.19.0.2 02:42:ac:13:00:02

配置fdb

集群1

bridge fdb append 02:42:ac:13:00:03 dst 172.18.0.3 dev vxlantest

集群2

bridge fdb append 02:42:ac:13:00:02 dst 172.18.0.2 dev vxlantest

配置路由

集群1

route add -net 10.244.2.0/24 gw 172.19.0.3 dev vxlantest

集群2

route add -net 10.244.1.0/24 gw 172.19.0.2 dev vxlantest

测试

集群1

curl 10.244.2.5

集群2

curl 10.244.1.5

补充

mac地址由如下代码生成,其中genMAC取自docker源码

package main

import (
    "crypto/rand"
    "fmt"
    "net"
)

func main() {
    ip := "172.19.0.3"
    fmt.Println(genMAC(net.ParseIP(ip)))
}

func genMAC(ip net.IP) net.HardwareAddr {
    hw := make(net.HardwareAddr, 6)
    // The first byte of the MAC address has to comply with these rules:
    // 1. Unicast: Set the least-significant bit to 0.
    // 2. Address is locally administered: Set the second-least-significant bit (U/L) to 1.
    hw[0] = 0x02
    // The first 24 bits of the MAC represent the Organizationally Unique Identifier (OUI).
    // Since this address is locally administered, we can do whatever we want as long as
    // it doesn't conflict with other addresses.
    hw[1] = 0x42
    // Fill the remaining 4 bytes based on the input
    if ip == nil {
        rand.Read(hw[2:])
    } else {
        copy(hw[2:], ip.To4())
    }
    return hw
}