从ipvs理解k8s svc

准备

创建demo负载

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx
  ports:
  - name: http
    port: 80
    targetPort: 80
  type: NodePort

iptables

PREROUTING

查看PREROUTING chain,执行如下命令

iptables -t nat -S PREROUTING

得到如下输出

-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES

查看service chain,执行如下命令

iptables -t nat -S KUBE-SERVICES

得到如下输出

node到pod
-A KUBE-SERVICES ! -s 10.244.0.0/16 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
nodeport
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
匹配ipset中的clusterip
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT

查看mask chain,执行如下命令

iptables -t nat -S KUBE-MARK-MASQ

得到如下输出

-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000

针对nodeport

查看nodeport chain,执行如下命令

iptables -t nat -S KUBE-NODE-PORT

得到如下输出

设置mask
-A KUBE-NODE-PORT -p tcp -m comment --comment "Kubernetes nodeport TCP port for masquerade purpose" -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ

POSTROUTING

查看POSTROUTING chain,执行如下命令

iptables -t nat -S POSTROUTING

得到如下输出

-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING

查看KUBE-POSTROUTING chain,执行如下命令

iptables -t nat -S KUBE-POSTROUTING

得到如下输出

未设置0x4000/0x4000 mask的直接返回
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
进行snat
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully

ipset

查看clusterip,执行如下命令

ipset list KUBE-CLUSTER-IP

得到如下输出

Name: KUBE-CLUSTER-IP
Type: hash:ip,port
Revision: 6
Header: family inet hashsize 1024 maxelem 65536 bucketsize 12 initval 0x8e438a99
Size in memory: 440
References: 3
Number of entries: 5
Members:
10.96.80.95,tcp:80

ipvs

查看网络设备ip,执行如下命令

ip addr show kube-ipvs0

得到如下输出(这些ip就是clusterip)

4: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default
    link/ether 02:19:cd:38:5b:a9 brd ff:ff:ff:ff:ff:ff
    inet 10.96.0.1/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.96.0.10/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.96.80.95/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever

查看ipvs规则,执行如下命令

ipvsadm -S -n

得到如下输出(clusterip对应ipvs的virtual-service)

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
nodeport
-A -t 172.18.0.4:30098 -s rr
-a -t 172.18.0.4:30098 -r 10.244.1.2:80 -m -w 1
-a -t 172.18.0.4:30098 -r 10.244.1.3:80 -m -w 1
-a -t 172.18.0.4:30098 -r 10.244.2.2:80 -m -w 1

clusterip
-A -t 10.96.80.95:80 -s rr
-a -t 10.96.80.95:80 -r 10.244.1.2:80 -m -w 1
-a -t 10.96.80.95:80 -r 10.244.1.3:80 -m -w 1
-a -t 10.96.80.95:80 -r 10.244.2.2:80 -m -w 1