我们常常遇到这样一些应用环境：客户端通过访问nginx，nginx再通过反向代理将后端web server运行的业务反馈给客户端，但是如果nginx server出现宕机或者其他bug导致业务不能正常运营，所以为了不影响业务正常运营，我们引进了keepalive+nginx组合，来保证整个系统的高可用。


Keepalive+Nginx双机热备

如图所示为整体的拓扑图：


keepalived内网安装 keepalived部署_keepalived内网安装

一.部署前说明：

(1)系统版本: centos 6.6（64位）

(2)角色及ip相关信息:

角色名称

网络ip信息

客户端(CIP)

10.58.137.203/24

Master_DR

eth0:172.51.96.105/24 &&  eth1:192.168.0.105/24

Backup_DR

eth0:172.51.96.119/24 &&  eth1:192.168.0.119/24

VIP

172.51.96.175/32

(3)相关中间件信息

keepalive版本信息: keepalived-1.2.15

nginx版本信息:   nginx1.6.0  （提供http服务）


二.部署操作：

负载均衡器上配置操作


nginx部署：

分别在Master_DR和backup_DR上安装nginx，操作如下：

1.1相关系统依赖包安装检查准备

1.2.1 检查系统自带nginx是否安装

rpm -qa | grep nginx

如有安装，请使用以下命令卸载相关程序 

yum remove nginx -y

1.2.2 安装编译nginx需要的依赖包



yum install gcc openssl-devel pcre-devel zlib-devel -y



2.1.1到nginx官网下载nginx软件，如下所示：

cd ~
wget http://nginx.org/download/nginx-1.8.0.tar.gz

2.1.2 添加运行nginx服务账号

groupadd -r nginx 
useradd -r -g nginx -s /bin/false -M nginx

2.1.3 解压编译安装nginx

cd ~
tar -zxf /root/nginx-1.8.0.tar.gz -C /usr/local/src
./configure \
 --prefix=/usr/local/nginx \
 --sbin-path=/usr/local/nginx/sbin/nginx \
 --conf-path=/etc/nginx/nginx.conf  \
 --error-log-path=/var/log/nginx/error.log \
 --http-log-path=/var/log/nginx/access.log  \
 --pid-path=/var/run/nginx/nginx.pid  \
 --lock-path=/var/lock/nginx.lock \
 --user=nignx \
 --group=nginx \
 --with-http_ssl_module \
 --with-http_stub_status_module \
 --with-http_gunzip_module      \
 --with-http_gzip_static_module \
 --http-client-body-temp-path=/var/tmp/nginx/client/ \
 --http-proxy-temp-path=/var/tmp/nginx/proxy/ \
 --http-fastcgi-temp-path=/var/tmp/nginx/fastcgi/ \
 --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi/ \
 --http-scgi-temp-path=/var/tmp/nginx/scgi/ \
 --with-pcre 
 make && make install

2.1.4 使用sed修改nginx的配置文件，指定nginx的运行用户

sed  '3 iuser nginx;' -i /etc/nginx/nginx.conf

修改后，nginx主配置文件内容如下

keepalived内网安装 keepalived部署_运维_02

注意：不修改nginx的运行用户的话，会报如下错误，如下图所示，

keepalived内网安装 keepalived部署_运维_03

2.1.5 启动nginx服务




/usr/local/nginx/sbin/nginx



下面就可访问nginx的主页面了，如下：

keepalived内网安装 keepalived部署_ViewUI_04

                                                           

至此，nginx部署就基本OK了！ 

注：关于nginx服务方式启动脚本示例请访问：http://blief.blog.51cto.com/6170059/1690492 


Keepalive部署

(1)分别在Master_DR和backup_DR上安装keepalive，操作如下：

1. 安装Keepalive所需要的相关依赖包:



# yum install openssl-devel popt-devel libnl-devel kernel-devel  -y



2. 编译安装keepalive


1.1 keepalived的源码获取 

keepalived源码包我们可以到keepalived的官网:http://www.keepalived.org/去下载，相关说明文档亦可在其官网查看，比如keepalived的使用，相关配置说明，这里演示的版本为:1.2.15



# cd ~
# wget http://www.keepalived.org/software/keepalived-1.2.15.tar.gz



1.2 编译安装keepalived

<--编译安装keepalived-->




# ln -s /usr/src/kernels/2.6.32-573.18.1.el6.x86_64/ /usr/src/linux
# tar zxvf keepalived-1.2.15.tar.gz -C /usr/local/src
# cd /usr/local/src/keepalived-1.2.15/
# ./configure \
  --prefix=/usr/local/keepalived \
  --with-kernel-dir=/usr/src/kernels/2.6.32-573.18.1.el6.x86_64
# make  make install



<--对keepalived进行相关路径优化调整-->

<---拷贝keepalived相关启动命令--->
# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/ 
# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/ 

<---将keepalived启动脚本添加到系统服务--->
# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
# chkconfig --add keepalived 
# chkconfig --level 2345 keepalived on

<---创建keepalived相关配置文件--->
# mkdir -p /etc/keepalived 
# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived

(3)分别配置Master_DR以及Backup_DR上的keepalive实例，如下所示:

1. Lvs_master_dr配置代码示例（主调度器）




vim /etc/keepalived/keepalived.conf



内容如下



! Configuration File for keepalived global_defs {    notification_email {        admin@bluemobi.cn    }    notification_email_from  lvs_admin@bluemobi.cn    smtp_server 127.0.0.1    smtp_connect_timeout 30    router_id DR_MASTER } vrrp_script check_nginx {                              #表示创建一个脚本check_nginx    script "/etc/keepalived/scripts/check_nginx.sh"     #引用的脚本路径    interval 3                                          #表示检测时间的间隔为3s } vrrp_instance http {     state BACKUP     interface eth0     lvs sync daemon interface eth0   #类似HA心跳检测的端口     dont_track_primary     nopreempt                        #不抢占master     track_interface {                #表示需要检测的网卡     eth0     eth1     }     mcast_src_ip 172.51.96.105       #表示设置组播的源地址     garp_master_delay 6     virtual_router_id 60     priority 110     advert_int 1     authentication {     auth_type PASS     autp_pass 1234     }     virtual_ipaddress {     172.51.96.175/24 brd  172.51.96.255 dev eth0 label eth0:1     }     virtual_routes {     172.51.96.175/24 dev eth0     }     track_script {     check_nginx      }     notify_master /etc/keepalived/scripts/state_master.sh     notify_backup /etc/keepalived/scripts/state_backup.sh     notify_fault  /etc/keepalived/scripts/state_fault.sh }




2 Lvs_backup_dr配置示例（备调度器）




vim /etc/keepalived/keepalived.conf



内容如下



! Configuration File for keepalived global_defs {    notification_email {        admin@bluemobi.cn    }    notification_email_from  lvs_admin@bluemobi.cn    smtp_server 127.0.0.1    smtp_connect_timeout 30    router_id DR_BACKUP } vrrp_script check_nginx {    script "/etc/keepalived/scripts/check_nginx.sh"    interval 3 } vrrp_instance http {     state BACKUP     interface eth0     lvs sync daemon interface eth0     dont_track_primary     track_interface {     eth0     eth1     }     mcast_src_ip 172.51.96.119     garp_master_delay 6     virtual_router_id 60     priority 105     advert_int 1     authentication {     auth_type PASS     autp_pass 1234     }     virtual_ipaddress {     172.51.96.175/24 brd  172.51.96.255 dev eth0 label eth0:1     }     virtual_routes {     172.51.96.175/24 dev eth0     }          track_script {     check_nginx      }     notify_master /etc/keepalived/scripts/state_master.sh     notify_backup /etc/keepalived/scripts/state_backup.sh     notify_fault  /etc/keepalived/scripts/state_fault.sh }



3.分别在主调度server和备调度server编写以下脚本，如下：

i 当调度器为切换master server时，记录切换时间日志

vim /etc/keepalived/scripts/state_master.sh 

代码如下:



#!/bin/bash echo -e  >> $LOGFILE host=CN-SH-DR01      #设置当前的主机名 LOGFILE="/var/log/keepalived-state.log" echo "[Master]" >> $LOGFILE date >> $LOGFILE echo "The ${host}  Starting to become master server...." >> $LOGFILE 2>&1 echo "Please run the “ipvsadm -Ln”  check the keepalived state ..." >> $LOGFILE echo ".........................................................................!">> $LOGFILE echo >>$LOGFILE




ii 当调度器为切换backup server时，记录切换时间日志

vim /etc/keepalived/scripts/state_backup.sh 

代码如下:



#!/bin/bash echo -e >> $LOGFILE host=CN-SH-DR01     #设置当前的主机名 LOGFILE="/var/log/keepalived-state.log" echo "[Backup]" >> $LOGFILE date >> $LOGFILE echo "The ${host}  Starting to become Backup server...." >> $LOGFILE 2>&1 echo "Please run the “ipvsadm -Ln”  check the state ..." >> $LOGFILE echo "........................................................................!">> $LOGFILE echo  >> $LOGFILE



iii  当调度器出现错误时，记录错误时间日志

vim /etc/keepalived/scripts/state_fault.sh 

代码如下:



#!/bin/bash echo -e >> $LOGFILE host=CN-SH-DR01      #设置当前的主机名 LOGFILE="/var/log/keepalived-state.log" echo "[fault errot ]" >> $LOGFILE date >> $LOGFILE echo "The ${host}  is fault error...." >> $LOGFILE 2>&1 echo "Please check the server state ..." >> $LOGFILE echo "........................................................................!">> $LOGFILE echo  >> $LOGFILE



iiii 服务状态健康监测脚本,比如当nginx不可用时，及时切换到backup调度器上

vim /etc/keepalived/scripts/check_nginx.sh 



#!/bin/bash #nginx="/usr/local/nginx/sbin/nginx" PID=`ps -C nginx --no-heading|wc -l` if [ "${PID}" = "0" ];     then          /etc/init.d/nginx start     sleep 3     LOCK=`ps -C nginx --no-heading|wc -l`     if [ "${LOCK}" = "0" ];     then         /etc/init.d/keepalived stop     fi fi



(4)依次重启master_dr,backup-dr上keepalive服务


# service keepalived restart


三.测试验证：

在两台调度器创建相关测试页，如下：

master-dr主调度服务器的测试页面



[root@master-dr www]# curl 172.51.96.105 this is master server  [root@master-dr www]#



backup-dr主调度服务器的测试页面

[root@backup-dr htdocs]# curl 127.0.0.1



this is backup server  [root@backup-dr htdocs]#



通过messages查看vip抢夺情况，如下所示:

keepalived内网安装 keepalived部署_前端_05


述上发现vip已经被master-dr抢夺，通过ifconfig看到master-dr已经存在vip地址，如下

keepalived内网安装 keepalived部署_keepalived内网安装_06


在客户端通过:http://vip 就可以访问到页面，此时访问的是master-dr的页面：

keepalived内网安装 keepalived部署_开发工具_07


将nginx的启动命令:/usr/local/nginx/sbin/nginx命名为：/usr/local/nginx/sbin/nginx1,再关闭nginx来模仿nginx服务故障

keepalived内网安装 keepalived部署_前端_08


分别查看master-dr和backup-dr的keepalive日志：

keepalived内网安装 keepalived部署_ViewUI_09

keepalived内网安装 keepalived部署_前端_10


 此时vip已经由backup-dr接管了，因为master-dr上nginx服务异常，而keepalive会定时触发引用的检查脚本“check_nginx”检查nginx状态，发现nginx可用就停止了keepalive服务，从而使vip顺利的飘逸到backup-dr上；

 查看keepalive-state.log,可以看到master-dr和backup-dr会记录每个状态的的信息：同时日志记录脚本也会记录相关信息：



[root@master-dr sbin]# tail -10  /var/log/keepalived-state.log 
The CN-SH-DR01  Starting to become Backup server....
Please run the “ipvsadm -Ln”  check the state ...
...............................................................................!

[fault errot ]
Fri Mar 11 15:28:48 CST 2016
The CN-SH-DR01  is fault error....
Please check the server state ...
...............................................................................!
 
  
[root@backup-dr htdocs]# tail -10 /var/log/keepalived-state.log 
[Backup]
Fri Mar 11 14:42:56 CST 2016
The CN-SH-DR02  Starting to become Backup server....
Please run the “ipvsadm -Ln”  check the state ...
...............................................................................!
[Master]
Fri Mar 11 15:06:58 CST 2016
The CN-SH-DR02  Starting to become master server....
Please run the “ipvsadm -Ln”  check the state ...
...............................................................................!




再次访问http://vip 发现页面为backup-dr的页面：

keepalived内网安装 keepalived部署_前端_11


到这里，整个keepalive+nginx双机热备就部署完成了


总结:关于keepalive+nginx双机热备适合很多web前端高可用集群应用，相对于keepalive+lvs案例应用，配置起来更方便，更简单，与它同样的经典应用案例：keepalive+haproxy也是一个不错的双机热备方案，比如在很多反向代理场景，往往需要根据业务应用来选择这两个应用组合！




转载于:https://blog.51cto.com/blief/1749783