图1
image.png
图2
image.png
图3
image.png
图4
image.png
SPoF:Single Point of Failure 单点故障
可用性指标 A=MTBF/(MTBF+MTTR)
可用性=平均无故障时长/(平均无故障时长+平均修复时长)
无故障时长:两次故障之间的时长
备用节点取代活动节点(需2点)
备用节点服务本身运行正常，二者的调度没有区别
活动节点上提供服务的IP地址随时可以切换至备用节点
MTTR时间:备用节点探测到活动节点故障所需时长+IP地址转移所需要时长
failover--失效转移/故障转移
"备用节点+活动节点"的集合: 高可用集群
活动节点繁忙，没有发送"心跳信息"给备用节点，误判下线，IP给了备用节点，导致二者互抢IP
谁是真正的"组织"? 如上图1
正常的集群逻辑是大于或等于3个节点
多个节点--避免资源浪费--既做高可用，又做负载均衡
前端dns配置两个A记录(不同的IP)
node1和node2相同服务，同时工作，二者IP不同--负载均衡
当node1-down机，把node1的IP给node2--高可用
Cluster IP(集群IP):多节点使用相同的IP，将请求轮流分发给不同的节点(节点工作时，相当于使用单独的IP)--了解
多种服务--对服务做高可用--了解--如上图2
OSPF协议:生成路由规则
路由1故障时，用户需要手动更改网关连接路由2
配置虚拟路由器:把两个路由设备的内部网卡绑定起来 如上图3和图4
keepalived
转移IP，进程等其他资源没有转移
增强ipvs功能:后端主机健康状态检测/实现多台主机调度--调动内核API自动生成规则
高可用lvs(无后端主机健康状态检测)，keepalived增添了此功能
图1
image.png
图2
image.png
centos7.4 192.168.1.7
centos7.4-2 192.168.1.8
HA Cluster的配置前提
1 时间同步：
建议centos7使用chrony来同步时间
vim /etc/chrony.conf 如上图1
理解意思即可--此处不用更改(已经使用ntpdate的计划任务了)
systemctl restart chronyd.service
chronyc sources 查看时间源
2 确保iptables及selinux不会成为阻碍：
systemctl status firewalld.service
systemctl status iptables.service
yum info iptables-services 安装此包，就可像centos6一样使用iptables
getenforce
3 各节点之间可通过主机名互相通信：
centos7.4和centos7.4-2
vim /etc/hosts
192.168.1.8 centos7.4-2.fgq.com
192.168.1.7 centos7.4.fgq.com
4 确保各节点的用于集群服务的接口支持MULTICAST通信：
ifconfig 显示多播地址通信--MULTICAST
keepalived安装配置
centos7.4和centos7.4-2
yum -y install keepalived
rpm -ql keepalived
man keepalived.conf
TOP HIERACHY 顶级配置段
VRRP synchronization group(s) vrrp同步组 如上图2
演示：单主模型 双主模型 高可用ipvs 高可用haproxy
cp /etc/keepalived/keepalived.conf{,.bak}
vim /etc/keepalived/keepalived.conf 如下图1
注意centos7.4和centos7.4-2配置时的3处区别
更改global_defs(下面括号内的四项全部删除，否则后面实验受影响)
global_defs {
notification_email {
keepalived@fgq.com
}
notification_email_from ka_admin@fgq.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id centos7.4.fgq.com/centos7.4-2.fgq.com centos7.4和centos7.4-2区别处1
vrrp_mcast_group4 224.110.110.18
(vrrp_skip_check_adv_addr) 下面四项全部删除，否则后面实验受影响
(vrrp_strict)
(vrrp_garp_interval 0)
(vrrp_gna_interval 0)
}
光标移动至"virtual_server"(集群服务的配置，此处先不用)上一行
:.,$s@^@#@ 从当前行到最后一行，把行首替换为#
:set nohlsearch 去除高亮
更改vrrp_instance
vrrp_instance VI_1 {
state MASTER/BACKUP centos7.4和centos7.4-2的区别处2
interface ens34
virtual_router_id 17
priority 100/95 centos7.4和centos7.4-2的区别处3
advert_int 1
authentication {
auth_type PASS
auth_pass JqZxY8Dc (openssl rand -base64 8; 取前8位)
}
virtual_ipaddress {
192.168.1.99 注意图中192.168.0.99是错的，要符合自己所在IP段-192.168.1.x
}
}
配置centos7.4完成后，直接复制文件到centos7.4-2，进行编辑即可
scp /etc/keepalived/keepalived.conf centos7.4-2.fgq.com:/etc/keepalived/
systemctl start keepalived.service
systemctl status keepalived.service 显示主/备状态信息
ip a 主--可以看到IP地址:192.168.0.99
ss -ntlu 没有显示多播地址监听，但可通过多播地址向外发送信息
tcpdump -i ens34 host 224.110.110.18 显示出来广播信息
tcpdump -nn -i ens34 host 224.110.110.18 忽略主机名解析
停止服务or阻断广播，转移VIP
centos7.4
systemctl stop keepalived.service ; ip a 服务停止，VIP转移
centos7.4-2
systemctl status keepalived.service ; ip a 显示为"主"，VIP地址出现
centos7.4
systemctl start keepalived.service
systemctl status keepalived.service; ip a 显示为"主"，VIP地址出现(抢占模式)
centos7.4-2
systemctl status keepalived.service ; ip a 显示为"备用"，VIP地址消失
多主模型
每个物理路由都工作起来，配置多个虚拟路由，每个物理路由都有一个专用的VIP
centos7.4
vim /etc/keepalived/keepalived.conf 如下图2
添加一个vrrp_instance
vrrp_instance VI_2 {
state BACKUP
interface ens34
virtual_router_id 27
priority 95
advert_int 1
authentication {
auth_type PASS
auth_pass JqZoY6ec
}
virtual_ipaddress {
192.168.1.89 注意图中192.168.0.98是错的，要符合自己所在IP段-192.168.1.x
}
}
systemctl stop keepalived.service
centos7.4-2
vim /etc/keepalived/keepalived.conf 如下图3
添加一个vrrp_instance
vrrp_instance VI_2 {
state MASTER
interface ens34
virtual_router_id 27
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass JqZoY6ec
}
virtual_ipaddress {
192.168.1.89(VIP2) 图中192.168.0.99是错的,要符合自己所在IP段192.168.1.x
}
}
systemctl stop keepalived.service
systemctl start keepalived.service
systemctl status keepalived.service; ip a 两个VIP地址都在此处
centos7.4
systemctl start keepalived.service
systemctl status keepalived.service; ip a VIP1地址出现
centos7.4-2
ip a 仅仅出现VIP2
systemctl stop keepalived.service
ip a VIP2消失
centos7.4
ip a 两个VIP地址都在此处
tcpdump -nn -i ens34 host 224.110.110.18 一直开着，看centos7.4-2启动后的变化
centos7.4-2
systemctl start keepalived.service
ip a VIP2地址出现
前端dns配置两个A记录(解析分别对应89和99的IP即可)，两个节点就可以工作起来了
请求到达dns，轮询，一部分请求到达节点1，另一部分到达节点2
实现了高可用+负载均衡
图1
image.png
图2
image.png
图3
image.png
centos7.4和centos7.4-2
状态转移时，所调用的通知脚本
vim /etc/keepalived/notify.sh
:1,$s@^[[:blank:]]@@ 多执行几次来删除开头的空格
#!/bin/bash
#
contact='root@localhost'
notify() {
local mailsubject="$(hostname) to be $1, vip floating"
local mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
echo "$mailbody" | mail -s "$mailsubject" $contact
}
case $1 in
master)
notify master
;;
backup)
notify backup
;;
fault)
notify fault
;;
*)
echo "Usage: $(basename $0) {master|backup|fault}"
exit 1
;;
esac
chmod +x /etc/keepalived/notify.sh
bash -n /etc/keepalived/notify.sh 语法检查
scp /etc/keepalived/notify.sh centos7.4-2.fgq.com:/etc/keepalived/
bash -x /etc/keepalived/notify.sh master 执行
tail /var/log/maillog 看邮件日志
mail 输入1查看邮件1 ok
cp /etc/keepalived/keepalived.conf{,.dual_master} 备份之前配置的双节点模型
systemctl stop keepalived.service
vim /etc/keepalived/keepalived.conf
使用单主模型，要删除vrrp_instance VI_2的内容
在vrrp_instance VI_1内部添加notify的信息 如下图1
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
systemctl start keepalived.service
systemctl status keepalived.service; ip a centos7.4会显示VIP地址
mail 查看邮件，centos7.4会显示master
------------------------------------------------------------------------------------------
服务--由进程提供；进程的单主/多主模型；虚拟服务器的实现
负载均衡集群中调度器的高可用
keepalived高可用ipvs集群
lvs-dr模型
centos7.4 director 192.168.1.7
centos7.4-2 director 192.168.1.8
centos7.4-3 real server 192.168.1.6 web服务(httpd/nginx)
centos7.4-4 real server 192.168.1.12 web服务(httpd/nginx)
systemctl restart chronyd.service 先同步时间or命令ntpdate
vim /etc/hosts 主机名解析
192.168.1.7 centos7.4.fgq.com
192.168.1.8 centos7.4-2.fgq.com
192.168.1.6 centos7.4-3.fgq.com
192.168.1.12 centos7.4-4.fgq.com
centos7.4-3和centos7.4-4 设置Real-Server
yum -y install nginx
vim /usr/share/nginx/html/index.html
:1,$d 删除原来的内容，换为自己的内容
<h1>RS1:CentOS7.4-3</h1> centos7.4-3的配置
<h1>RS2:CentOS7.4-4</h1> centos7.4-4的配置
systemctl start nginx.service
ss -ntl
centos7.4
手动测试
curl 192.168.1.6 ok
curl 192.168.1.12 ok
centos7.4-3和centos7.4-4
vim setrs.sh
#!/bin/bash
#
vip=192.168.1.88
mask=255.255.255.255
iface="lo:0"
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
ifconfig $iface $vip netmask $mask broadcast $vip up
route add -host $vip dev $iface
;;
stop)
ifconfig $iface down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
;;
*)
echo "Usage:$(basename $0) start|stop"
exit 1
;;
esac
chmod +x setrs.sh
bash -n setrs.sh
bash -x setrs.sh start
ifconfig 显示"lo:0 192.168.1.88"
cat /proc/sys/net/ipv4/conf/all/arp_ignore 验证 1 ok
cat /proc/sys/net/ipv4/conf/lo/arp_announce 验证 2 ok
scp setrs.sh centos7.4-4.fgq.com:/root/
centos7.4-4
./setrs.sh start
ifconfig 显示"lo:0 192.168.1.88"
配置lvs集群
centos7.4
yum -y install ipvsadm
ifconfig ens34:0 192.168.1.88 netmask 255.255.255.255 broadcast 192.168.1.88 up
ifconfig 显示ens34:0 192.168.1.88 ok
centos7.4-5
ping 192.168.1.88 ok
arp 192.168.1.88
arp 显示VIP仅仅对应centos7.4的mac地址，其他没有响应 ok
centos7.4
ipvsadm -A -t 192.168.1.88:80 -s rr
ipvsadm -a -t 192.168.1.88:80 -r 192.168.1.6:80 -g
pvsadm -a -t 192.168.1.88:80 -r 192.168.1.12:80 -g
ipvsadm -Ln
centos7.4-5
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done 轮询 ok
centos7.4
ipvsadm -C 清空策略
ifconfig ens34:0 down
ifconfig 没有VIP了
在centos7.4上测试完后，再在centos7.4-2上测试lvs是否可以调度
centos7.4-2
yum -y install ipvsadm
ifconfig ens34:0 192.168.1.88 netmask 255.255.255.255 broadcast 192.168.1.88 up
ifconfig
ipvsadm -A -t 192.168.1.88:80 -s rr
ipvsadm -a -t 192.168.1.88:80 -r 192.168.1.6:80 -g
ipvsadm -a -t 192.168.1.88:80 -r 192.168.1.12:80 -g
ipvsadm -Ln
centos7.4-5
arp -d 192.168.1.88 清除缓存记录
ping 192.168.1.88 ok
arp 192.168.1.88
arp 显示VIP仅仅对应centos7.4-2的mac地址，其他没有响应 ok
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done 轮询 ok
centos7.4-2
ipvsadm -C 清空策略
ifconfig ens34:0 down
ifconfig 没有VIP了
配置keepalived后，客户端(centos7.4-5)可自动更新arp记录，不用自己再手动更新了
centos7.4和centos7.4-2
vim /etc/keepalived/keepalived.conf
更改vrrp_instance中的virtual_ipaddress的IP 如下图2
添加virtual_server信息 如下图2和3
光标移动到"#virtual_server 10.10.10.2"的上一行
:.,$d
光标移动到"#virtual_server 192.168.200.100"的上一行
:.,$s@^#@@g
virtual_server 192.168.1.88 80 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
real_server 192.168.1.6 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200 也可使用校验码:genhash -s 192.168.1.6 -p 80 -u /
}
connect_timeout 2
nb_get_retry 3
delay_before_retry 1
}
}
real_server 192.168.1.12 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 2
nb_get_retry 3
delay_before_retry 1
}
}
}
注意：另外一种写法--把HTTP变为TCP(此处不做演示了)
HTTP_GET(产生大量干扰日志) 变为 TCP_CHECK(四层检测，不会产生干扰日志)
删除"url {...} "的内容
systemctl status keepalived.service
systemctl stop keepalived.service
centos7.4-2
systemctl start keepalived.service
systemctl status keepalived.service; ip a 显示VIP
ipvsadm -Ln 显示lvs策略
centos7.4-5
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done 轮询 ok
centos7.4
systemctl start keepalived.service
systemctl status keepalived.service; ip a 显示VIP
ipvsadm -Ln 显示lvs策略
centos7.4-2
systemctl status keepalived.service; ip a VIP转移，变为备用
centos7.4-5
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done 轮询 ok
不用清除arp缓存，可以直接响应
centos7.4
systemctl stop keepalived.service
centos7.4-5
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done 轮询 ok 高可用ipvs
centos7.4-3(Real Server1)
iptables -A INPUT -p tcp --dport 80 -j REJECT
iptables -vnL 发现对应的策略中拦截了几个报文
centos7.4-5
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done 只显示RS2(centos7.4-4)的内容
centos7.4-3(Real Server1)
iptables -F 清空策略
centos7.4-5
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done 轮询 ok
后端的两个Real Server都down掉了，此时需要sorry server显示内容--前端调度器上配置
centos7.4和centos7.4-2
yum -y install nginx
vim /usr/share/nginx/html/index.html
<h1>Sorry From Director1/2</h1>
systemctl start nginx.service
ss -ntl 80端口
centos7.4-5
curl 192.168.1.7/8 ok，显示sorry server的信息
centos7.4和centos7.4-2
vim /etc/keepalived/keepalived.conf
virtual_server中添加信息：
sorry_server 127.0.0.1 80 如下图4 监听在本机
也可以添加如下信息，发送邮件通知
notify_up <STRING>|<QUOTED-STRING> 检测成功，调用此脚本(string)进行通知 不再演示
notify_down <STRING>|<QUOTED-STRING> 检测失败，调用此脚本(string)进行通知 不再演示
systemctl stop keepalived.service
systemctl start keepalived.service
centos7.4-5
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done 轮询 ok
centos7.4-3(关闭rs1)
systemctl stop nginx
centos7.4-5
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done 只显示rs2
centos7.4-4(关闭rs2)
systemctl stop nginx
centos7.4-5
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done
显示director1(centos7.4)的sorry server信息
centos7.4(关闭director1)
systemctl stop keepalived.service
centos7.4-5
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done
显示director2(centos7.4-2)的sorry server信息
centos7.4-3和centos7.4-4(开启rs1和rs2)
systemctl start nginx.service
centos7.4-5
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done 轮询 ok
centos7.4(开启director1)
systemctl start keepalived.service; ip a VIP转移至director1--centos7.4
centos7.4-5
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done 轮询 ok 访问不受影响
图1
image.png
图2
image.png
图3
image.png
图4
image.png
keepalived对服务的高可用
效果好--调度器(本身没有存储数据)
效果差--存储服务(keepalived转移IP和进程，但不会转移数据)
共享存储
NAS:网络附加存储，存储服务器:nfs/cifs-server--文件接口--锁管理系统
SAN:存储区域网络，块级别:分区格式化/挂载使用--块接口--无锁管理系统
两个机器共用一块硬盘，主机数据会更新至硬盘
二者对数据的修改不一致，时间戳紊乱，会导致文件系统崩溃
隔离机制
节点级别:STONITH(爆头)--常用于共享存储--关闭其中一个节点
资源级别:资源隔离
keepalived基于脚本调整权限/优先级/...，来高可用服务--资源级别
centos7.4 keepalived+nginx 之前已经安装
centos7.4-2 keepalived+nginx 之前已经安装
systemctl status keepalived.service; systemctl stop keepalived.service
systemctl status nginx; systemctl stop nginx; systemctl start nginx
grep -i 'vrrp_script' /usr/share/doc/keepalived-1.3.5/samples/*
cat /usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.vrrp.localcheck
参考文档
脚本功能: /etc/keepalived/down文件: 存在--降权，不存在--不降权
centos7.4和centos7.4-2
vim /etc/keepalived/down.sh
#!/bin/bash
if [[ -f /etc/keepalived/down ]];then
weight -10
fi
chmod +x /etc/keepalived/down.sh
vim /etc/keepalived/keepalived.conf
删除virtual_server内容，添加vrrp_script和track_script 如下图1
vrrp_script chk_down {
script "/etc/keepalived/down.sh"
interval 2
fall 2
rise 2
}
track_script {
chk_down
}
systemctl start keepalived
systemctl status keepalived; ip a
centos7.4
tcpdump -nn -i ens34 host 224.110.110.18 一直开着，看变化
touch /etc/keepalived/down; ip a 权重变化，VIP转移
centos7.4-2: ip a 显示VIP
centos7.4
rm -rf /etc/keepalived/down; ip a 权重变化，显示VIP
centos7.4-2: ip a VIP消失
高可用nginx集群
脚本: nginx-ok-权重默认；nginx-no-权重降低
centos7.4和centos7.4-2
systemctl stop nginx.service
vim /etc/nginx/nginx.conf
如下图2
systemctl start nginx.service
centos7.4-5
for i in {1..10};do curl 192.168.1.7;sleep 0.3;done 轮询ok
for i in {1..10};do curl 192.168.1.8;sleep 0.3;done 轮询ok
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done 轮询ok
centos7.4
ip a
touch /etc/keepalived/down; ip a 权重变化，VIP转移
centos7.4-2
ip a 显示处VIP
centos7.4-5
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done 轮询ok
虽然VIP转移，但是客户访问不受影响
centos7.4
rm -rf /etc/keepalived/down; ip a 显示VIP
centos7.4和centos7.4-2
systemctl stop keepalived.service
vim /etc/keepalived/keepalived.conf
添加vrrp_script和track_script 如下图3
centos7.4-2
systemctl start keepalived.service
systemctl status keepalived.service 显示"VRRP_Script(chk_nginx) succeeded"
ip a 显示VIP
centos7.4-5
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done 轮询ok
centos7.4
systemctl start keepalived.service
systemctl status keepalived.service; ip a 显示VIP
centos7.4-5
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done 轮询ok
centos7.4和centos7.4-2
vim notify.sh
在backup选项中添加启动服务的信息，在备用节点中启动服务
当主节点down机，备节点的服务初始时已经启动，可以直接上线工作
backup)
notify backup
systemctl start nginx
;;
centos7.4
systemctl stop nginx
ip a 重复多次，VIP变化从无到有
虽然停止服务，转为备用节点
但是notify.sh中bacckup重启服务，又把VIP抢回来了
yum -y install httpd
systemctl stop nginx.service && systemctl start httpd
停止服务，转为备用节点
又启动httpd服务，占用了80端口
但是如果想nginx服务上线，还要手动操作
解决方案：配置监控系统--zabix(后面讲)
不过已经实现了服务故障，VIP转移的功能
systemctl stop httpd
虽然释放了80端口，但是检查节点的时间已经过去
nginx服务还是不能自动启动，需要手动启动
双主模型
高可用+负载均衡nginx，避免资源浪费，再添加一个VIP即可
centos7.4和centos7.4-2
vim /etc/keepalived/keepalived.conf
添加"vrrp_instance VI_2"的信息
如下图4--centos7.4
如下图5--centos7.4-2
systemctl stop keepalived.service
centos7.4-2
systemctl start keepalived.service
systemctl status keepalived.service; ip a 显示两个VIP
centos7.4-5
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done 轮询ok
for i in {1..10};do curl 192.168.1.99;sleep 0.3;done 轮询ok
centos7.4
systemctl start keepalived.service
systemctl status keepalived.service; ip a 显示VIP1
centos7.4-2
systemctl status keepalived.service; ip a 显示VIP2
centos7.4-5
for i in {1..10};do curl 192.168.1.88;sleep 0.3;done 轮询ok
for i in {1..10};do curl 192.168.1.99;sleep 0.3;done 轮询ok
要求: nginx不要做会话绑定
生产中，很少使用会话绑定来追踪用户，经常使用sesson replication cluster/session server
现在只需要在dns记录上添加两条A记录192.168.1.88/192.168.1.99即可
web站点架构 如下图6 keepalived高可用调度器
图1
image.png
图2
image.png
图3
image.png
图4
image.png
图5
image.png
图6
image.png
网友评论