Nginx+keepalived高可用配置记录

master服务器ip地址：192.168.100.161
slave服务器ip地址：192.168.100.162
虚拟ip（VIP，一个尚未占用的内网ip即可）地址： 192.168.100.168

确认使用的网卡

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:0c:29:f9:1d:13 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.162/24 brd 192.168.100.255 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fef9:1d13/64 scope link 
       valid_lft forever preferred_lft forever

使用第2个网卡ens192

• 下载keepalived

wget http://www.keepalived.org/software/keepalived-2.2.7.tar.gz
#要以不安全的方式连接至 www.keepalived.org，使用“--no-check-certificate
wget http://www.keepalived.org/software/keepalived-2.2.7.tar.gz --no-check-certificate

• 安装依赖包

yum install gcc openssl-devel popt-devel -y  

解压，编译安装

tar zxf keepalived-2.2.7.tar.gz -C /usr/local/src/
cd /usr/local/src/keepalived-2.2.7
./configure --prefix=/usr/local/keepalived

image.png

make && make install

ll /usr/local/src/keepalived-2.2.7/keepalived/etc/init.d/keepalived
-rwxrwxr-x 1 shida shida 1308 9月  15 2016 /usr/local/src/keepalived-2.2.7/keepalived/etc/init.d/keepalived

cp /usr/local/src/keepalived-2.2.7/keepalived/etc/init.d/keepalived    /etc/init.d/
mkdir -p /etc/keepalived
cp /usr/local/keepalived/etc/keepalived/keepalived.conf.sample    /etc/keepalived/keepalived.conf
cp /usr/local/src/keepalived-2.2.7/keepalived/etc/sysconfig/keepalived     /etc/sysconfig/
cp /usr/local/keepalived/sbin/keepalived    /usr/sbin/

• 修改配置文件：

master服务器配置：

! Configuration File for keepalived

global_defs {
   router_id lvs_slb0
            }

# 检测nginx是否运行
vrrp_script chk_nginx {
        script "/etc/keepalived/nginx_check.sh"
        interval 2
        weight -20
                      }

vrrp_instance VI_1 {
    #state MASTER
    #此处不设置为MASTER，通过priority来竞争master
    state MASTER
    #网卡名字，ip a 可查看
    interface ens192
    #同一个keepalived集群的virtual_router_id相同
    virtual_router_id 51
    #权重，master要大于slave
    priority 100
    #主备通讯时间间隔
    advert_int 1
    #主备保持一致
    authentication {
        auth_type PASS
        auth_pass 1111
    }
     # 与上方nginx运行状况检测呼应
    track_script {
        chk_nginx
    }

    virtual_ipaddress {
        # 虚拟ip地址
        192.168.100.168
                      }
}

slave服务器配置：

! Configuration File for keepalived

global_defs {
   router_id lvs_lsb1
            }

# 检测nginx是否运行
vrrp_script chk_nginx {
        script "/etc/keepalived/nginx_check.sh"
        interval 2
        weight -20
                      }

vrrp_instance VI_1 {
    state BACKUP
    interface ens192
    virtual_router_id 51
    priority 90
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass 1111
                   }
#与上方nginx运行状况检测呼应
    track_script {
        chk_nginx
                 }

virtual_ipaddress {
        # 虚拟ip地址
        192.168.100.168
                   }
                    }

• nginx监听脚本：

#创建nginx检测脚本
#touch nginx_check.sh

#给脚本增加可执行权限
#chmod +x nginx_check.sh

• 脚本添加如下内容：

#! /bin/bash
pidof nginx
if [ $? -ne 0 ];then
/etc/init.d/keepalived stop
fi

• 防止出现脑裂现象（主备同时获取了VIP地址） 我没有应用。

# 指定keepalived配置的网卡：enp0s3，固定的VRRP广播地址：224.0.0.18
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface enp0s3 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --out-interface enp0s3 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
firewall-cmd --reload

# 查看配置的规则
firewall-cmd --direct --get-rules ipv4 filter INPUT
firewall-cmd --direct --get-rules ipv4 filter OUTPUT

image.png

• 启动服务：

# 启动服务
service keepalived start

# 配置开机自启动
systemctl enable keepalived

• 查看服务启动情况：

ps -aux |grep keepalived

image.png

• 查看启动日志：

journalctl -xe

image.png

• 查看keepalived日志

tail -f  /var/log/messages

image.png

配置成功后的效果。ens192是网卡名字；192.168.100.168是虚拟ip，已经成功绑定到网卡上。

image.png

• 关于keepalived配置中的 mcast_src_ip 和 unicast_src_ip

如果两节点的上联交换机允许组播，采用组播模式（默认）

如果两节点的上联交换机禁用了组播，则只能采用vrrp单播通告的方式
https://www.jianshu.com/p/7c709c3be4a9

• Keepalived两节点出现双VIP的情况

https://www.cnblogs.com/netonline/archive/2017/10/09/7642595.html

• keepalived 配置文件参数详解

https://blog.csdn.net/mofiu/article/details/76644012

https://www.cnblogs.com/along1226/p/5027838.html

• MySQL共享存储主备模式利用Keepalived实现双机高可用

http://blog.51cto.com/7424593/1893767

• keepalived Unsafe permissions found for script

http://old.gebiji.com/2016/12/21/ka-warning-default-user/

• How do I enable: script_security?

https://github.com/acassen/keepalived/issues/901

• Keepalived+Nginx架构整理版

https://linuxeye.com/448.html

• LVS的原理介绍

https://www.cnblogs.com/along1226/p/5014363.html

更多的监控nginx健康脚本

#!/bin/sh
nginxPidNum=`ps -C nginx --no-header |wc -l`
keepalivedPidNum=`ps -C keepalived --no-header |wc -l`
if [$nginxPidNum -eq 0 ];then
    killall keepalived
elif [$keepalivedPidNum -eq 0 ];then
    service keepalived start
fi

A=`ps -C nginx --no-header |wc -l`                 ## 查看是否有 nginx进程 把值赋给变量A 
if [ $A -eq 0 ];then                               ## 如果没有进程值得为 零
      /usr/local/nginx/sbin/nginx
      sleep 3
      if [ `ps -C nginx --no-header |wc -l` -eq 0 ];then
            killall keepalived                     ## 则结束 keepalived 进程
      fi
fi 

#!/bin/bash
ps -C nginx --no-header
if [ $? -ne 0 ]; then
        /usr/local/nginx/sbin/nginx
        sleep 5
        ps -C nginx --no-header
        if [ $? -ne 0 ]; then
                /etc/init.d/keepalived stop
        fi
fi
sleep 5

#!/bin/bash
while :
do
nginxpid=`ps -C nginx --no-header | wc -l`
if [ $nginxpid -eq 0 ];then
   /usr/local/nginx/sbin/nginx
   sleep 5
   nginxpid=`ps -C nginx --no-header | wc -l`
   echo $nginxpid
      if [ $nginxpid -eq 0 ];then
       /etc/init.d/keepalived stop
      fi
fi
sleep 5
done

#!/bin/sh
white true
do
PNUM=`ps -ef|grep nginx|wc -l`
#这里也可使用nmap 192.168.1.3 -p 80|grep open|wc -l来判断个数
if [ $PNUM -lt 3 ];then
/etc/init.d/keepalived stop >/dec/null 2>&1
kill -9 keealived >/dec/null 2>&1
kill -9 keealived >/dec/null 2>&1
fi
sleep 5
done

检查nginx配置文件

/usr/local/nginx/sbin/nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful