nginx和keepalived实现nginx高可用

Keepalived介绍

Keepalived:它是一个高性能的服务器高可用或热备解决方案，Keepalived主要来防止服务器单点故障的发生问题，可以通过其与Nginx的配合实现web服务端的高可用。

Keepalived以VRRP协议为实现基础，用VRRP协议来实现高可用性，VRRP协议将两台或多台路由器设备虚拟成一个设备，对外提供虚拟路由器IP（一个或多个），如下图所示。

keepalived nginx高可用 nginx和keepalived实现nginx高可用_IP


\这张图的意思是，我们使用keepalived来管理两台设备的Nginx，并虚拟出一个IP，外界请求直接访问虚拟IP而不是真正的Nginx，让虚拟IP去访问提供服务的Nginx，然后再由Nginx去访问tomcat。

高可用是指同一时间提供服务的只有一台设备，提供服务的设备挂掉之后，备份服务器便开始提供服务。

实验环境

主机名

IP地址

角色

系统版本

web1

192.168.100.33

tomacat

red-hat7

web2

192.168.100.115

tomcat

red-hat7

keepalived-master

192.168.100.116keepalived nginx

red-hat7

keepalived-backup

192.168.100.169

keepalived nginx

red-hat7

关闭防火墙和selinux

keepalived nginx高可用 nginx和keepalived实现nginx高可用_keepalived nginx高可用_02

在web服务器上安装tomcat

  • java环境安装
    安装jdk 环境
[root@web1 ~]# yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel
  • 下载tomcat
[root@web1 ~]# cd /usr/src/
[root@web1 src]# wget https://archive.apache.org/dist/tomcat/tomcat-9/v9.0.8/bin/apache-tomcat-9.0.8.tar.gz
  • 解压部署tomcat
[root@web1 src]# tar xf apache-tomcat-9.0.8.tar.gz  -C /usr/local/
[root@web1 src]# cd /usr/local/
[root@web1 local]# ln -s apache-tomcat-9.0.8/ tomcat
  • 启动tomcat
[root@web1 local]# /usr/local/tomcat/bin/catalina.sh start
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:        /usr
Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Tomcat started.
[root@web1 local]# ss -anpt
State       Recv-Q Send-Q     Local Address:Port                    Peer Address:Port              
LISTEN      0      128                    *:22                                 *:*                   users:(("sshd",pid=1006,fd=3))
LISTEN      0      100            127.0.0.1:25                                 *:*                   users:(("master",pid=1100,fd=13))
ESTAB       0      0         192.168.100.33:22                     192.168.100.1:49959               users:(("sshd",pid=1213,fd=3))
LISTEN      0      100                   :::8080                              :::*                   users:(("java",pid=4015,fd=48))
LISTEN      0      128                   :::22                                :::*                   users:(("sshd",pid=1006,fd=4))
LISTEN      0      100                  ::1:25                                :::*                   users:(("master",pid=1100,fd=14))
LISTEN      0      1       ::ffff:127.0.0.1:8005                              :::*                   users:(("java",pid=4015,fd=75))
LISTEN      0      100                   :::8009                              :::*
  • 设置web2的测试网页
[root@web2 tomcat]# cd /usr/local/tomcat/webapps/ROOT/
[root@web2 ROOT]# cat index.jsp 
<%@ page language="java" import="java.util.*" pageEncoding="utf-8"%>
<HTML>
    <HEAD>
        <TITLE>JSP Test Page</TITLE>
    </HEAD>
    <BODY>
      <%
        Random rand = new Random();
        out.println("<h1>Random number:</h1>");
        out.println(rand.nextInt(99)+100);
      %>
</BODY>
</HTML>
  • 重启web2
[root@web2 ROOT]# /usr/local/tomcat/bin/startup.sh stop
[root@web2 ROOT]# /usr/local/tomcat/bin/startup.sh start

\访问效果

keepalived nginx高可用 nginx和keepalived实现nginx高可用_IP_03


keepalived nginx高可用 nginx和keepalived实现nginx高可用_keepalived nginx高可用_04

在keepalived-master和keepalived-backup上部署nginx反向代理和负载均衡

  • 配置keepalived-master
  • 下载nginx，并启动
[root@keepalived-master ~]# cd /usr/src/
[root@keepalived-master src]# wget http://nginx.org/download/nginx-1.14.0.tar.gz
[root@keepalived-master nginx-1.14.0]# yum -y install pcre-devel openssl openssl-devel gd-devel gcc
[root@keepalived-master nginx-1.14.0]# yum -y groups mark install 'Development Tools'
[root@keepalived-master nginx-1.14.0]# groupadd -r nginx
[root@keepalived-master nginx-1.14.0]# useradd -r -M -s /sbin/nologin -g nginx nginx
[root@keepalived-master nginx-1.14.0]# mkdir -p /var/log/nginx
[root@keepalived-master nginx-1.14.0]# chown -R nginx.nginx /var/log/nginx
[root@keepalived-master nginx-1.14.0]# ./configure \
>  --prefix=/usr/local/nginx \
>  --user=nginx \
>  --group=nginx \
>  --with-debug \
>  --with-http_ssl_module \
>  --with-http_realip_module \
>  --with-http_image_filter_module \
>  --with-http_gunzip_module \
>  --with-http_gzip_static_module \
>  --with-http_stub_status_module \
>  --http-log-path=/var/log/nginx/access.log \
>  --error-log-path=/var/log/nginx/error.log
[root@keepalived-master nginx-1.14.0]# echo "export PATH=/usr/local/nginx/sbin:$PATH" > /etc/profile.d/nginx.sh
[root@keepalived-master nginx-1.14.0]# . /etc/profile.d/nginx.sh
[root@keepalived-master nginx-1.14.0]# nginx
  • 在web上配置负载均衡和反向代理
upstream dsb {
        server 192.168.100.33:8080 weight=2;
        server 192.168.100.115:8080 weight=1;
    }

    server {
        listen       80;
        server_name  localhost;


        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
            root   html;
            index  index.jsp index.html index.htm;
            proxy_pass http://dsb;
        }

\浏览器上测试

keepalived nginx高可用 nginx和keepalived实现nginx高可用_nginx_05


keepalived nginx高可用 nginx和keepalived实现nginx高可用_nginx_06


\web2与web1的配置一样

测试如下

keepalived nginx高可用 nginx和keepalived实现nginx高可用_keepalived nginx高可用_07

keepalived nginx高可用 nginx和keepalived实现nginx高可用_keepalived nginx高可用_08

部署keepalived

keepalived-master 和 keepalived-backup 上做相同的部署

  • 下载keepalived 所需要的开发包
[root@keepalived-master conf]# yum install popt-devel  openssl-devel  -y
  • 编译安装keepalived
[root@keepalived-master conf]# cd /usr/src/
[root@keepalived-master src]# wget http://www.keepalived.org/software/keepalived-2.0.8.tar.gz
[root@keepalived-master src]# tar xf keepalived-2.0.8.tar.gz
[root@keepalived-master src]# cd keepalived-2.0.8/
[root@keepalived-master keepalived-2.0.8]# ./configure --prefix=/usr/local/keepalived
[root@keepalived-master keepalived-2.0.8]#  make && make install
  • 使用keepalived 的默认安装路径，现在是/usr/local/，复制默认配置文件到默认路径
[root@keepalived-master keepalived]# mkdir /etc/keepalived
[root@keepalived-master keepalived]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
  • 复制keepalived 的服务脚本到默认的位置
[root@keepalived-master keepalived]# cp /usr/src/keepalived-2.0.8/keepalived/etc/init.d/keepalived /etc/init.d/
[root@keepalived-master keepalived]# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[root@keepalived-master keepalived]# ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin/
  • 设置keep alived的开机自启动
[root@keepalived-master sbin]# chkconfig keepalived on
Note: Forwarding request to 'systemctl enable keepalived.service'.
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.

配置keepalived 的配置文件

  • Master 配置文件
[root@keepalived-master sbin]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc          //建议使用监控邮件 SMTP 
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id keepalived-master           //本节点的标识，建议使用hostname
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_script chk_nginx {                                   //检测nginx的脚本路径
        script "/etc/keepalived/nginx_check.sh"
        interval 2                                                //检测时间间隔
        weiht -20                                               //如果条件成立权重减20
}

vrrp_instance VI_1 {                                        //  VI_1 为虚拟路由标识符，自己定义
    state MASTER                                            //主节点为master ，被节点为backup
    interface eth32                                             //绑定虚拟IP的网络接口，与本机的网卡相同
    virtual_router_id 10                                      //虚拟路由的ID号，与备节点一致
    priority 100                                                     //节点优先级，范围0~254，master必须比backup要高
    advert_int 1                                                   //组播信息放送间隔，两个接点必须设置一致
    authentication {                                               //设置的验证信息
        auth_type PASS
        auth_pass 1111                                           //两个节点必须一致
    }
        track_script{                                  //定时执行脚本配置（vrrp_script chk_nginx）的名称要一样
        chk_nginx
    }
    virtual_ipaddress {
        192.168.100.200/24                                       虚拟IP，又称为VIP 两节点必须一致
    }
}
  • Backup 配置文件
[root@keepalived-backup keepalived-2.0.8]# vim /etc/keepalived/keepalived.conf
global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id keepalived-backup
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}
vrrp_script chk_nginx {
        script "/etc/keepalived/nginx_check.sh"
        interval 2
        weight  -20
}

vrrp_instance VI_1 {
    state MASTER
    interface ens32
    virtual_router_id 10
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
        track_script{
        chk_nginx
    }
    virtual_ipaddress {
        192.168.100.200/24
    }
}

编写Nginx状态检测脚本

编写 Nginx 状态检测脚本 /etc/keepalived/nginx_check.sh (已在 keepalived.conf 中配置)脚本要求：如果 nginx 停止运行，尝试启动，如果无法启动则杀死本机的 keepalived 进程， keepalied将虚拟 ip 绑定到 BACKUP 机器上

[root@keepalived-master ~]# cat /etc/keepalived/nginx_check.sh
#!/bin/bash

A=`ps -C nginx --no-header |wc -l`
if [ $A -eq 0 ];then
        /usr/local/nginx/sbin/nginx
        sleep 2
        if [ `ps -C nginx --no-header | wc -l` -eq 0 ];then
            killall keeplived
        fi
fi

\此处如果killall没有命令，需要安装yum -y install psmisc

  • 保存后给执行权限
[root@keepalived-master ~]# chmod  +x /etc/keepalived/nginx_check.sh
  • 启动keepalived
[root@keepalived-master ~]# service keepalived start
Starting keepalived (via systemctl):                       [  OK  ]
  • 查看keepalived-master 的ip
[root@keepalived-master ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:05:94:b0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.116/24 brd 192.168.100.255 scope global ens32
       valid_lft forever preferred_lft forever
    inet 192.168.100.200/24 scope global ens32
       valid_lft forever preferred_lft forever
    inet6 fe80::13e4:67f8:1586:dd18/64 scope link 
       valid_lft forever preferred_lft forever
  • 查看keepalived-backup 的ip
[root@keepalived-backup keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:b9:a5:63 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.169/24 brd 192.168.100.255 scope global ens32
       valid_lft forever preferred_lft forever
    inet6 fe80::bb22:4f4e:8e9:25fd/64 scope link 
       valid_lft forever preferred_lft forever

//如果两太服务器上都出现虚拟地址，代表配置文件有误

验证，访问虚拟IP

keepalived nginx高可用 nginx和keepalived实现nginx高可用_keepalived nginx高可用_09

  • 停止keepalived-master 服务,在keepalived-backup 上实现了,ip地址间的漂移
[root@keepalived-master ~]# service keepalived stop
Stopping keepalived (via systemctl):                       [  OK  ]
[root@keepalived-backup keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:b9:a5:63 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.169/24 brd 192.168.100.255 scope global ens32
       valid_lft forever preferred_lft forever
    inet 192.168.100.200/24 scope global secondary ens32
       valid_lft forever preferred_lft forever
    inet6 fe80::bb22:4f4e:8e9:25fd/64 scope link 
       valid_lft forever preferred_lft forever

\keepalived-master 服务恢复后，master会抢占ip地址

[root@keepalived-master ~]# service keepalived start
Starting keepalived (via systemctl):                       [  OK  ]
[root@keepalived-backup keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:b9:a5:63 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.169/24 brd 192.168.100.255 scope global ens32
       valid_lft forever preferred_lft forever
    inet6 fe80::bb22:4f4e:8e9:25fd/64 scope link 
       valid_lft forever preferred_lft forever

\将nginx的配置文件故意写错后，重启

[root@keepalived-master ~]# vim /usr/local/nginx/conf/nginx.conf
#user  nobody;
worker_processes  1
[root@keepalived-master ~]# nginx -t
nginx: [emerg] directive "worker_processes" is not terminated by ";" in /usr/local/nginx/conf/nginx.conf:12
nginx: configuration file /usr/local/nginx/conf/nginx.conf test failed
[root@keepalived-master ~]# reboot
[root@keepalived-backup keepalived]# ip a    //地址在backup上偏移了
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:b9:a5:63 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.169/24 brd 192.168.100.255 scope global ens32
       valid_lft forever preferred_lft forever
    inet 192.168.100.200/24 scope global secondary ens32
       valid_lft forever preferred_lft forever
    inet6 fe80::bb22:4f4e:8e9:25fd/64 scope link 
       valid_lft forever preferred_lft forever
  • 网站还是能正常访问