Nginx+Keepalived高可用之Docker

作者: 老鼠AI大米_Java全栈 | 来源:发表于2018-11-05 11:58 被阅读170次

Keepalived 是一种高性能的服务器高可用或热备解决方案， Keepalived 可以用来防止服务器单点故障的发生，通过配合 Nginx 可以实现 web 前端服务的高可用。

Keepalived 以 VRRP 协议为实现基础。 VRRP(Virtual RouterRedundancy Protocol)协议是用于实现路由器冗余的协议，它将两台或多台路由器设备虚拟成一个设备，对外提供虚拟路由器 IP(一个或多个)，而在路由器组内部，如果实际拥有这个对外 IP 的路由器如果工作正常就是 MASTER，或者是通过算法选举产生， MASTER 实现针对虚拟路由器 IP 的各种网络功能，其他设备不拥有该虚拟 IP，状态是 BACKUP，除了接收 MASTER 的VRRP 状态通告信息外，不执行对外的网络功能。

当主机失效时， BACKUP 将接管原先 MASTER 的网络功能。VRRP 协议使用多播数据来传输 VRRP 数据， VRRP 数据使用特殊的虚拟源 MAC 地址发送数据而不是自身网卡的 MAC 地址， VRRP 运行时只有 MASTER 路由器定时发送 VRRP 通告信息，表示 MASTER 工作正常以及虚拟路由器 IP(组)， BACKUP 只接收 VRRP 数据，不发送数据，如果一定时间内没有接收到 MASTER 的通告信息，各 BACKUP 将宣告自己成为 MASTER，发送通告信息，重新进行 MASTER 选举状态。

上一篇文章讲解了从零配置Nginx+Keepalived高可用，本文将讲解通过docker方式配置。

nginx配置
先添加nginx的配置文件，如下：nginx.conf

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    gzip  on;
   
    upstream  api  {
        server   61.135.169.121:80 max_fails=1; #baidu.com
        server   42.236.9.70:80 max_fails=1;  #so.com
    }

    server {
        listen  80;
        location / {
        proxy_next_upstream error timeout http_500 http_502 http_504;
                proxy_read_timeout 10s;
                proxy_pass        http://api;
                proxy_set_header   Host             $host;
                proxy_set_header   X-Real-IP        $remote_addr;
                proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
    }
    }
}

此配置文件配置nginx自动跳转baidu和360搜索。

nginx检测脚本
该脚本被keepalived用来检测nginx是否在运行，若nginx停止则也停止keepalived。
nginx_check.sh

#!/bin/bash  
A=`ps -C nginx -no-header |wc -l`
if [ $A -eq 1 ];then
        pkill keepalived
fi

keepalived配置文件
该配置文件配置2个虚拟实例，状态均为BACKUP，非抢占式，即谁都可以成为MASTER而不会被另一个抢占。
keepalived.conf

global_defs {
    #这里邮件配置只对内网
    notification_email {
      #xxx@qq.com #邮件报警
    }
    #notification_email_from xxx@163.com #指定发件人
    #smtp_server smtp.163.com   #指定smtp服务器地址
    #smtp_connect_timeout 30    #指定smtp连接超时时间
    router_id LVS_DEVEL  #负载均衡标识，在局域网内应该是唯一的。
}

vrrp_script chk_nginx {
    script "/etc/keepalived/nginx_check.sh"
    #每2秒检测一次nginx的运行状态
    interval 2
    #失败一次，将自己的优先级调整为-20
    weight  -20
}

# virtual_ipaddress vip
# vrrp-虚拟路由冗余协议
# vrrp_instance 用来定义对外提供服务的VIP区域及其相关属性
vrrp_instance VI_1 {
    state BACKUP #指定该keepalived节点的初始状态
    interface enp0s31f6 #vrrp实例绑定的接口，用于发送VRRP包
    virtual_router_id 51 #取值在0-255之间，用来区分多个instance的VRRP组播， 同一网段中该值不能重复，并且同一个vrrp实例使用唯一的标识
    priority 149 #指定优先级，优先级高的将成为MASTER
    nopreempt #设置为不抢占。默认是抢占的
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    ###采用单播通信，避免同一个局域网中多个keepalived组之间的相互影响
    unicast_src_ip 172.19.165.231  ##本机ip
    unicast_peer { #采用单播的方式发送VRRP通告，指定单播邻居的IP地址
        172.19.165.222
    }
    virtual_ipaddress { #指定VIP地址
        172.19.165.251
    }
    #nginx存活状态检测脚本
    track_script {
        chk_nginx
    }
    notify "/container/service/keepalived/assets/notify.sh"
}
vrrp_instance VI_2 {
    state BACKUP
    interface enp0s31f6
    virtual_router_id 52
    priority 98
    nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    ###采用单播通信，避免同一个局域网中多个keepalived组之间的相互影响
    unicast_src_ip 172.19.165.231  ##本机ip
    unicast_peer { #采用单播的方式发送VRRP通告，指定单播邻居的IP地址
        172.19.165.222
    }
    virtual_ipaddress {
         172.19.165.252
    }
    #nginx存活状态检测脚本
    track_script {
        chk_nginx
    }
    notify "/container/service/keepalived/assets/notify.sh"
}

注意，chk_nginx中的路径是指容器中路径。

因为要部署在2台机器上，配置文件是需要略为改变，如下：
机器172.19.165.222：

interface 当前机器的网卡
priority 150
unicast_src_ip 本机IP 也可以不设
unicast_peer 需要通告的IP
机器172.19.165.231：
interface 当前机器的网卡
priority 149
unicast_src_ip 本机IP 也可以不设
unicast_peer 需要通告的IP
可以看到配置变化不大。

Dockerfile
配置keepalived镜像的Dockerfile文件

FROM osixia/keepalived
MAINTAINER yxx

ADD ./nginx_check.sh /etc/keepalived/nginx_check.sh
RUN chmod +x /etc/keepalived/nginx_check.sh

ADD ./keepalived.conf /container/service/keepalived/assets/keepalived.conf

该文件是基于另一个镜像配置的，添加本地文件到容器对应的位置，以便脚本调用。

docker-compse配置
下面配置docker-compose去执行一连串的任务，这正是我们需要的，如启动nginx--创建keepalived镜像并启动等等
docker-compose.yml

version: '2'

services:
  keepalived:
    build: ./
    depends_on:
      - nginx
    network_mode: "host"
    cap_drop:
      - NET_ADMIN
    privileged: true
    #volumes: 
      #- "./send_mail.sh:/etc/keepalived/send_mail.sh"
    restart: on-failure:3
  nginx:
    image: nginx:1.15.5
    privileged: true  #root权限
    ports:
      - "80:80" #本机：容器
    volumes: 
      - "./nginx.conf:/etc/nginx/nginx.conf"
    restart: on-failure:3

该配置文件配置2个service且keepalived依赖nginx，启动keepalived容器时会自动编译创建镜像。
注意，这里并没有支持发送外网报警邮件。
网上发报警邮件资料多是非docker方式，这个在前一篇文章已说明，若要让send_mail.sh能在docker中执行需要安装一些依赖。
也有发送邮件的docker镜像，感兴趣的可以研究下。

docker-smtp

启动服务
所有配置都完成了，就可以启动docker服务了

# 第一次执行会下载镜像并build
# docker-compose up
# 若修改了配置文件，需要重新build
# docker-compose up --build
# 以后台模式运行
# docker-compose up -d

高可用检查
当机器222启动完毕，可以看到它先是进入BACKUP状态，然后发现没有MASTER则再把自己变为MASTER。

keepalived_1  | Mon Nov  5 02:50:50 2018: VRRP_Instance(VI_2) Entering BACKUP STATE
keepalived_1  | Mon Nov  5 02:50:50 2018: Opening script file /container/service/keepalived/assets/notify.sh
keepalived_1  | Mon Nov  5 02:50:50 2018: VRRP sockpool: [ifindex(2), proto(112), unicast(1), fd(10,11)]
keepalived_1  | Ok, i'm just a backup, great.
keepalived_1  | Ok, i'm just a backup, great.
keepalived_1  | Mon Nov  5 02:50:50 2018: VRRP_Script(chk_nginx) succeeded
keepalived_1  | Mon Nov  5 02:51:13 2018: VRRP_Instance(VI_1) Transition to MASTER STATE
keepalived_1  | Mon Nov  5 02:51:13 2018: VRRP_Instance(VI_2) Transition to MASTER STATE
keepalived_1  | Mon Nov  5 02:51:14 2018: VRRP_Instance(VI_1) Entering MASTER STATE
keepalived_1  | Mon Nov  5 02:51:14 2018: VRRP_Instance(VI_1) setting protocol VIPs.
keepalived_1  | Mon Nov  5 02:51:14 2018: Sending gratuitous ARP on enp0s31f6 for 172.19.165.251
keepalived_1  | Mon Nov  5 02:51:14 2018: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on enp0s31f6 for 172.19.165.251
keepalived_1  | Mon Nov  5 02:51:14 2018: Sending gratuitous ARP on enp0s31f6 for 172.19.165.251
keepalived_1  | Mon Nov  5 02:51:14 2018: Sending gratuitous ARP on enp0s31f6 for 172.19.165.251
keepalived_1  | Mon Nov  5 02:51:14 2018: Sending gratuitous ARP on enp0s31f6 for 172.19.165.251
keepalived_1  | Mon Nov  5 02:51:14 2018: Sending gratuitous ARP on enp0s31f6 for 172.19.165.251
keepalived_1  | Mon Nov  5 02:51:14 2018: Opening script file /container/service/keepalived/assets/notify.sh
keepalived_1  | I'm the MASTER! Whup whup.
keepalived_1  | Mon Nov  5 02:51:14 2018: VRRP_Instance(VI_2) Entering MASTER STATE
keepalived_1  | Mon Nov  5 02:51:14 2018: VRRP_Instance(VI_2) setting protocol VIPs.
keepalived_1  | Mon Nov  5 02:51:14 2018: Sending gratuitous ARP on enp0s31f6 for 172.19.165.252
keepalived_1  | Mon Nov  5 02:51:14 2018: VRRP_Instance(VI_2) Sending/queueing gratuitous ARPs on enp0s31f6 for 172.19.165.252
keepalived_1  | Mon Nov  5 02:51:14 2018: Sending gratuitous ARP on enp0s31f6 for 172.19.165.252
keepalived_1  | Mon Nov  5 02:51:14 2018: Sending gratuitous ARP on enp0s31f6 for 172.19.165.252
keepalived_1  | Mon Nov  5 02:51:14 2018: Sending gratuitous ARP on enp0s31f6 for 172.19.165.252
keepalived_1  | Mon Nov  5 02:51:14 2018: Sending gratuitous ARP on enp0s31f6 for 172.19.165.252
keepalived_1  | Mon Nov  5 02:51:14 2018: Opening script file /container/service/keepalived/assets/notify.sh
keepalived_1  | I'm the MASTER! Whup whup.
keepalived_1  | Mon Nov  5 02:51:19 2018: Sending gratuitous ARP on enp0s31f6 for 172.19.165.251
keepalived_1  | Mon Nov  5 02:51:19 2018: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on enp0s31f6 for 172.19.165.251
keepalived_1  | Mon Nov  5 02:51:19 2018: Sending gratuitous ARP on enp0s31f6 for 172.19.165.251
keepalived_1  | Mon Nov  5 02:51:19 2018: Sending gratuitous ARP on enp0s31f6 for 172.19.165.251
keepalived_1  | Mon Nov  5 02:51:19 2018: Sending gratuitous ARP on enp0s31f6 for 172.19.165.251
keepalived_1  | Mon Nov  5 02:51:19 2018: Sending gratuitous ARP on enp0s31f6 for 172.19.165.251
keepalived_1  | Mon Nov  5 02:51:19 2018: Sending gratuitous ARP on enp0s31f6 for 172.19.165.252

再启动231机器，发现它只是进入BACKUP状态，因为MASTER已经存在。
通过命令ip addr查看虚拟IP绑定到了222机器，231并没有绑定，当输入虚拟IP就能自动切换到222机器上。
222机器

2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether e0:d5:5e:28:59:34 brd ff:ff:ff:ff:ff:ff
    inet 172.19.165.222/24 brd 172.19.165.255 scope global noprefixroute enp0s31f6
       valid_lft forever preferred_lft forever
    inet 172.19.165.251/32 scope global enp0s31f6
       valid_lft forever preferred_lft forever
    inet 172.19.165.252/32 scope global enp0s31f6
       valid_lft forever preferred_lft forever
    inet6 fe80::5354:6434:1fc6:2947/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

231机器

2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether e0:d5:5e:26:5d:97 brd ff:ff:ff:ff:ff:ff
    inet 172.19.165.231/24 brd 172.19.165.255 scope global noprefixroute enp0s31f6
       valid_lft forever preferred_lft forever
    inet6 fe80::63da:2a60:d37d:d5da/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

当关掉222机器上的服务，再次查看虚拟IP的绑定情况，发现绑定已从222移231上。
美中不足是无报警邮件，我会继续研究。
至此，高可用完毕。

Nginx+Keepalived高可用之Docker

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

java高级开发