前言
在学习了lvs集群后，看到lvs-dr集群能够实现vip与dip/rip不在同一网段的模式部署，为了加深对lvs的理解，因此尝试设计实现了一下。在设计配置前，我在网上找了很很多资料，也尝试过按照相应的资料来部署验证，但是最终结果还是无法验证成功。后来我参考网络上的资料，按照自己的思路调整了下拓扑，就成功把结果验证出来了。
附上参考链接：https://blog.csdn.net/brad_chen/article/details/47807505
https://www.cnblogs.com/AloneSword/p/3935462.html
未验证拓扑图a
修改后验证成功的拓扑图b
工作原理
在修改后的拓扑中，lvs-dr的数据流向大致为如下：客户端请求VIP，router会将请求转发给director。为了保证第一个接受请求报文的是director，需要在Real Server上修改Arp响应的机制，确保网络中只有director会响应VIP的arp请求。随后当director检查请求报文，发现其请求的是一组集群服务的时候，其会根据ipvsadm设置的调度算法将请求转发给Real Server。此过程中，director不会更改数据报文中的IP地址信息，仅仅修改报文中的源目mac信息。当请求报文根据director修改的源目mac信息到达Real Server后，Real Server会对请求报文进行响应。此时Real Server 发送源地址为VIP，目标地址为client Ip，源mac为Real Server 的mac地址，目标地址为client或网关的mac地址的响应报文。最后路由器将接受到的响应报文转发给client终端，完成通信。
实际上中，在发起请求报文之前，还需要进行三次握手的协商，如下面抓包的前三条记录，到了第四条记录才是真正的数据传输。
[root@client ~]# tcpdump -i eno16777736 -nn -e port 80
13:22:28.360656 00:0c:29:6d:1a:7d > 00:0c:29:21:59:d7, ethertype IPv4 (0x0800), length 74: 188.88.88.10.49672 > 172.16.0.8.80: Flags [S], seq 448411585, win 29200, options [mss 1460,sackOK,TS val 22310691 ecr 0,nop,wscale 7], length 0
13:22:28.364284 00:0c:29:21:59:d7 > 00:0c:29:6d:1a:7d, ethertype IPv4 (0x0800), length 74: 172.16.0.8.80 > 188.88.88.10.49672: Flags [S.], seq 3569473399, ack 448411586, win 28960, options [mss 1460,sackOK,TS val 13371774 ecr 22310691,nop,wscale 7], length 0
13:22:28.364346 00:0c:29:6d:1a:7d > 00:0c:29:21:59:d7, ethertype IPv4 (0x0800), length 66: 188.88.88.10.49672 > 172.16.0.8.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 22310696 ecr 13371774], length 0
13:22:28.365675 00:0c:29:6d:1a:7d > 00:0c:29:21:59:d7, ethertype IPv4 (0x0800), length 140: 188.88.88.10.49672 > 172.16.0.8.80: Flags [P.], seq 1:75, ack 1, win 229, options [nop,nop,TS val 22310697 ecr 13371774], length 74
13:22:28.367973 00:0c:29:21:59:d7 > 00:0c:29:6d:1a:7d, ethertype IPv4 (0x0800), length 66: 172.16.0.8.80 > 188.88.88.10.49672: Flags [.], ack 75, win 227, options [nop,nop,TS val 13371778 ecr 22310697], length 0
13:22:28.369291 00:0c:29:21:59:d7 > 00:0c:29:6d:1a:7d, ethertype IPv4 (0x0800), length 340: 172.16.0.8.80 > 188.88.88.10.49672: Flags [P.], seq 1:275, ack 75, win 227, options [nop,nop,TS val 13371780 ecr 22310697], length 274
13:22:28.369309 00:0c:29:6d:1a:7d > 00:0c:29:21:59:d7, ethertype IPv4 (0x0800), length 66: 188.88.88.10.49672 > 172.16.0.8.80: Flags [.], ack 275, win 237, options [nop,nop,TS val 22310701 ecr 13371780], length 0
13:22:28.369757 00:0c:29:6d:1a:7d > 00:0c:29:21:59:d7, ethertype IPv4 (0x0800), length 66: 188.88.88.10.49672 > 172.16.0.8.80: Flags [F.], seq 75, ack 275, win 237, options [nop,nop,TS val 22310701 ecr 13371780], length 0
13:22:28.371169 00:0c:29:21:59:d7 > 00:0c:29:6d:1a:7d, ethertype IPv4 (0x0800), length 66: 172.16.0.8.80 > 188.88.88.10.49672: Flags [F.], seq 275, ack 76, win 227, options [nop,nop,TS val 13371782 ecr 22310701], length 0
13:22:28.371185 00:0c:29:6d:1a:7d > 00:0c:29:21:59:d7, ethertype IPv4 (0x0800), length 66: 188.88.88.10.49672 > 172.16.0.8.80: Flags [.], ack 276, win 237, options [nop,nop,TS val 22310703 ecr 13371782], length 0
而在拓扑图a中，我在client抓包中发现，客户端并没有跟请求的server 三次握手协商成功，如下所示：
[root@client ~]# tcpdump -i eno16777736 -nn -e port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno16777736, link-type EN10MB (Ethernet), capture size 65535 bytes
13:32:32.392266 00:0c:29:6d:1a:7d > 00:0c:29:21:59:d7, ethertype IPv4 (0x0800), length 74: 188.88.88.10.49675 > 172.16.0.8.80: Flags [S], seq 1074120056, win 29200, options [mss 1460,sackOK,TS val 22914724 ecr 0,nop,wscale 7], length 0
13:32:34.396238 00:0c:29:6d:1a:7d > 00:0c:29:21:59:d7, ethertype IPv4 (0x0800), length 74: 188.88.88.10.49675 > 172.16.0.8.80: Flags [S], seq 1074120056, win 29200, options [mss 1460,sackOK,TS val 22916728 ecr 0,nop,wscale 7], length 0
13:32:38.404080 00:0c:29:6d:1a:7d > 00:0c:29:21:59:d7, ethertype IPv4 (0x0800), length 74: 188.88.88.10.49675 > 172.16.0.8.80: Flags [S], seq 1074120056, win 29200, options [mss 1460,sackOK,TS val 22920736 ecr 0,nop,wscale 7], length 0
13:32:46.420052 00:0c:29:6d:1a:7d > 00:0c:29:21:59:d7, ethertype IPv4 (0x0800), length 74: 188.88.88.10.49675 > 172.16.0.8.80: Flags [S], seq 1074120056, win 29200, options [mss 1460,sackOK,TS val 22928752 ecr 0,nop,wscale 7], length 0
13:33:02.436673 00:0c:29:6d:1a:7d > 00:0c:29:21:59:d7, ethertype IPv4 (0x0800), length 74: 188.88.88.10.49675 > 172.16.0.8.80: Flags [S], seq 1074120056, win 29200, options [mss 1460,sackOK,TS val 22944768 ecr 0,nop,wscale 7], length 0
13:33:34.500116 00:0c:29:6d:1a:7d > 00:0c:29:21:59:d7, ethertype IPv4 (0x0800), length 74: 188.88.88.10.49675 > 172.16.0.8.80: Flags [S], seq 1074120056, win 29200, options [mss 1460,sackOK,TS val 22976832 ecr 0,nop,wscale 7], length 0
但是我在router的Eth1上却抓包了相应的ACK报文：
[root@router ~]# tcpdump -i eno33554984 -nn -e host 172.16.0.8
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno33554984, link-type EN10MB (Ethernet), capture size 262144 bytes
03:09:59.317554 00:0c:29:21:59:cd > 00:0c:29:26:a3:20, ethertype IPv4 (0x0800), length 74: 188.88.88.10.49675 > 172.16.0.8.80: Flags [S], seq 1074120056, win 29200, options [mss 1460,sackOK,TS val 22913721 ecr 0,nop,wscale 7], length 0
03:09:59.318328 00:0c:29:26:a3:20 > 00:0c:29:a9:56:bd, ethertype IPv4 (0x0800), length 74: 188.88.88.10.49675 > 172.16.0.8.80: Flags [S], seq 1074120056, win 29200, options [mss 1460,sackOK,TS val 22913721 ecr 0,nop,wscale 7], length 0
03:09:59.318746 00:0c:29:a9:56:bd > 00:0c:29:21:59:c3, ethertype IPv4 (0x0800), length 74: 172.16.0.8.80 > 188.88.88.10.49675: Flags [S.], seq 3017446266, ack 1074120057, win 28960, options [mss 1460,sackOK,TS val 14025057 ecr 22913721,nop,wscale 7], length 0
03:10:00.319527 00:0c:29:21:59:cd > 00:0c:29:26:a3:20, ethertype IPv4 (0x0800), length 74: 188.88.88.10.49675 > 172.16.0.8.80: Flags [S], seq 1074120056, win 29200, options [mss 1460,sackOK,TS val 22914724 ecr 0,nop,wscale 7], length 0
03:10:00.319786 00:0c:29:26:a3:20 > 00:0c:29:a9:56:bd, ethertype IPv4 (0x0800), length 74: 188.88.88.10.49675 > 172.16.0.8.80: Flags [S], seq 1074120056, win 29200, options [mss 1460,sackOK,TS val 22914724 ecr 0,nop,wscale 7], length 0
03:10:00.319789 00:0c:29:a9:56:bd > 00:0c:29:21:59:c3, ethertype IPv4 (0x0800), length 74: 172.16.0.8.80 > 188.88.88.10.49675: Flags [S.], seq 3017446266, ack 1074120057, win 28960, options [mss 1460,sackOK,TS val 14026142 ecr 22913721,nop,wscale 7], length 0
03:10:01.583522 00:0c:29:a9:56:bd > 00:0c:29:21:59:c3, ethertype IPv4 (0x0800), length 74: 172.16.0.8.80 > 188.88.88.10.49675: Flags [S.], seq 3017446266, ack 1074120057, win 28960, options [mss 1460,sackOK,TS val 14027511 ecr 22913721,nop,wscale 7], length 0
03:10:02.323965 00:0c:29:21:59:cd > 00:0c:29:26:a3:20, ethertype IPv4 (0x0800), length 74: 188.88.88.10.49675 > 172.16.0.8.80: Flags [S], seq 1074120056, win 29200, options [mss 1460,sackOK,TS val 22916728 ecr 0,nop,wscale 7], length 0
03:10:02.323979 00:0c:29:26:a3:20 > 00:0c:29:a9:56:bd, ethertype IPv4 (0x0800), length 74: 188.88.88.10.49675 > 172.16.0.8.80: Flags [S], seq 1074120056, win 29200, options [mss 1460,sackOK,TS val 22916728 ecr 0,nop,wscale 7], length 0
03:10:02.323982 00:0c:29:a9:56:bd > 00:0c:29:21:59:c3, ethertype IPv4 (0x0800), length 74: 172.16.0.8.80 > 188.88.88.10.49675: Flags [S.], seq 3017446266, ack 1074120057, win 28960, options [mss 1460,sackOK,TS val 14028313 ecr 22913721,nop,wscale 7], length 0
也就是说拓扑a中，client到VIP的三次请求都没完成，因此我在拓扑a中抓包也看不到任何的关于http或者数据传输的信息。至于为什么ack都回到了router却没有转发到client，这也是我一直百思不得其解的。如果这现象有朋友有想法或者我的做法有误的话，还望各位提点。
设计配置
设计拓扑
此拓扑中我使用了五个虚拟机分别为Router、client、RS1、RS2和Director。Router划分三个子网，分别是外网192.167.0.0/24、测试网络188.88.88.0/24、内网10.10.10.0/24，其余Ip信息跟拓扑图中一致。
1、配置Router
首先为Router 的虚拟机添加三个虚拟网卡，其中Eth0为桥接到本地网络作为外网，其余两个虚拟网卡分别桥接到188.88.88.0/24和10.10.10.0/24的主机网络中。
使用nmtui命令工具配置接口IP和网关：
[root@router ~]# ifconfig
eno16777736: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.81 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::20c:29ff:fe21:59b9 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:21:59:b9 txqueuelen 1000 (Ethernet)
RX packets 16269 bytes 18572778 (17.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6041 bytes 747550 (730.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eno33554984: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.10.254 netmask 255.255.255.0 broadcast 10.10.10.255
inet6 fe80::20c:29ff:fe21:59c3 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:21:59:c3 txqueuelen 1000 (Ethernet)
RX packets 3143 bytes 270428 (264.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 7899 bytes 14591038 (13.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eno50332208: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 188.88.88.254 netmask 255.255.255.0 broadcast 188.88.88.255
inet6 fe80::20c:29ff:fe21:59cd prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:21:59:cd txqueuelen 1000 (Ethernet)
RX packets 61 bytes 6425 (6.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 118 bytes 9539 (9.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@router ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.0.1 0.0.0.0 UG 100 0 0 eno16777736
10.10.10.0 0.0.0.0 255.255.255.0 U 100 0 0 eno33554984
188.88.88.0 0.0.0.0 255.255.255.0 U 100 0 0 eno50332208
192.168.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eno16777736
在于director直连的eno33554984接口下配置虚拟子接口，Ip为172.16.0.1作为vip网关：
#不配置此虚拟子接口的话，请求报文无法传递给director
[root@router ~]# ifconfig eno33554984:0 172.16.0.1/24 up
配置iptables启用snat：
[root@router ~]# iptables -F
#放开内网三个网段的上网流量
[root@router ~]# iptables -t nat -I POSTROUTING -s 188.88.88.0/24 -j SNAT --to-source 192.168.0.81
[root@router ~]# iptables -t nat -I POSTROUTING -s 10.10.10.0/24 -j SNAT --to-source 192.168.0.81
[root@router ~]# iptables -t nat -I POSTROUTING -s 172.16.0.0/24 -j SNAT --to-source 192.168.0.81
开启路由转发功能：
[root@router ~]# sysctl -w net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1
2、配置director
安装ipvsadm：
[root@director ~]# yum install -y ipvsadm
配置接口：
[root@director ~]# ifconfig ens33 10.10.10.8 netmask 255.255.255.0 up
创建director配置脚本：
[root@director ~]# vim vs.sh
#!/bin/bash
#
vip='172.16.0.8'
iface='ens33:0'
mask='255.255.255.0'
port='80'
rs1='10.10.10.11'
rs2='10.10.10.12'
scheduler='wrr'
type='-g'
drgw='172.16.0.1'
case $1 in
start)
ifconfig $iface $vip netmask $mask broadcast $vip up
route add default gw $drgw
iptables -F
ipvsadm -A -t ${vip}:${port} -s $scheduler
ipvsadm -a -t ${vip}:${port} -r ${rs1} $type -w 1
ipvsadm -a -t ${vip}:${port} -r ${rs2} $type -w 1
;;
stop)
ipvsadm -C
ifconfig $iface down
;;
*)
echo "Usage $(basename $0) start|stop"
exit 1
;;
esac
执行脚本：
[root@director ~]# bash -x vs.sh start
+ vip=172.16.0.8
+ iface=ens33:0
#为了能与172.16.0.1通信，需配置director的vip掩码为255.255.255.0
+ mask=255.255.255.0
+ port=80
+ rs1=10.10.10.11
+ rs2=10.10.10.12
+ scheduler=wrr
+ type=-g
+ drgw=172.16.0.1
+ case $1 in
+ ifconfig ens33:0 172.16.0.8 netmask 255.255.255.0 broadcast 172.16.0.8 up
+ route add default gw 172.16.0.1
+ iptables -F
+ ipvsadm -A -t 172.16.0.8:80 -s wrr
+ ipvsadm -a -t 172.16.0.8:80 -r 10.10.10.11 -g -w 1
+ ipvsadm -a -t 172.16.0.8:80 -r 10.10.10.12 -g -w 1
3、配置RS1和RS2
将RS1和RS2 接入到跟10.10.10.0/24 网卡上，使用nmtui命令工具配置其IP地址信息，网关指向10.10.10.254：
[root@rs1 ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.10.11 netmask 255.255.255.0 broadcast 10.10.10.255
inet6 fe80::c559:fd98:3450:1449 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:a9:56:bd txqueuelen 1000 (Ethernet)
RX packets 27199 bytes 34870271 (33.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6211 bytes 647161 (631.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@rs1 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.10.10.254 0.0.0.0 UG 100 0 0 ens33
10.10.10.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
编辑生成下述脚本：
[root@rs1 ~]# vim rs1.sh
#!/bin/bash
#
vip=172.16.0.8
mask='255.255.255.255'
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig lo:0 $vip netmask $mask broadcast $vip up
route add -host $vip dev lo:0
;;
stop)
ifconfig lo:0 down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
;;
*)
echo "Usage $(basename $0) start|stop"
exit 1
;;
esac
执行脚本：
[root@rs1 ~]# bash -x rs1.sh
+ vip=172.16.0.8
+ mask=255.255.255.255
+ case $1 in
++ basename rs1.sh
+ echo 'Usage rs1.sh start|stop'
Usage rs1.sh start|stop
+ exit 1
[root@rs1 ~]# bash -x rs1.sh start
+ vip=172.16.0.8
+ mask=255.255.255.255
+ case $1 in
+ echo 1
+ echo 1
+ echo 2
+ echo 2
+ ifconfig lo:0 172.16.0.8 netmask 255.255.255.255 broadcast 172.16.0.8 up
+ route add -host 172.16.0.8 dev lo:0
安装httpd服务：
[root@rs1 ~]# yum install -y httpd
编辑index页面：
[root@rs1 ~]# vim /var/www/html/index.html
<h1>This RS1 10.10.10.11</h1>
启动httpd服务和关闭防火墙：
[root@rs1 ~]# systemctl start httpd
[root@rs1 ~]# systemctl stop firewalld
重复以上步骤配置RS2。
4、配置client
将client接入到188.88.88.0/24的网卡上。使用nmtui配置其IP地址：
[root@client ~]# ifconfig
eno16777736: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 188.88.88.10 netmask 255.255.255.0 broadcast 188.88.88.255
inet6 fe80::20c:29ff:fe6d:1a7d prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:6d:1a:7d txqueuelen 1000 (Ethernet)
RX packets 4567 bytes 467167 (456.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1581 bytes 158829 (155.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@client ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 188.88.88.254 0.0.0.0 UG 100 0 0 eno16777736
188.88.88.0 0.0.0.0 255.255.255.0 U 100 0 0 eno16777736
5、测试验证
从client访问虚拟服务172.16.0.8：
[root@client ~]# for i in {1..10};do curl http://172.16.0.8 ;done
<h1>This RS2 10.10.10.12</h1>
<h1>This RS1 10.10.10.11</h1>
<h1>This RS2 10.10.10.12</h1>
<h1>This RS1 10.10.10.11</h1>
<h1>This RS2 10.10.10.12</h1>
<h1>This RS1 10.10.10.11</h1>
<h1>This RS2 10.10.10.12</h1>
<h1>This RS1 10.10.10.11</h1>
<h1>This RS2 10.10.10.12</h1>
<h1>This RS1 10.10.10.11</h1>
另外如果需要模拟映射到外网的场景，可在Router上做DNAT映射，然后访问192.168.0.81来实现访问lvs虚拟服务。如：
在Router上添加DNAT：
[root@router ~]# iptables -t nat -I PREROUTING -d 192.168.0.81 -p tcp --dport 80 -j DNAT --to-destination 172.16.0.8:80
模拟外网访问
网友评论