使用keepalived实现HA Cluster集群,下面以示例来演示keepalived的使用和keepalived的相关知识


HA Cluster配置前提:

    1、本机的主机名,要与hostname(uname-n)获得的名称保持一致;

     CentOS6: /etc/sysconfig/network

       CentOS7: hostnamectl set-hostname HOSTNAME

     各节点要能互相解析主机名;一般建议通过hosts文件进行解析;

     修改主机名为centos7centos6.1.com,然后修改hosts文件,能让两边能相互解析到

   2、各节点时间同步;ntpdate  172.16.249.195把这个主机当成时间服务器,然后进行时间同步

    设置ntp时间同步服务器

    修改/etc/ntp.conf文件

    内容如下:

        driftfile /var/lib/ntp/drift

        restrict 127.0.0.1

        restrict ::1

        restrict 172.16.0.0 mask 255.255.0.0nomodify notrap

        server 127.127.1.0

        fudge 127.127.1.0 stratum 8

        includefile /etc/ntp/crypto/pw

        keys /etc/ntp/keys

        disable monitor

 

    3、确保iptablesselinux不会成为服务阻碍;

        nopreempt:非抢占模式;默认为抢占模式;

 

 在调度器上,首先安装keepalived程序,然后把时间同步成一样的

ntpdate 172.16.249.195

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee

检查iptablesselinux是否开启了

然后在CentOS7中进行实验

yum install  keepalived 

 

用两台机器一台CentOS7,一台CentOS6看是否有问题

CentOS7

修改位置文件(/etc/keepalived/keepalived.conf

首先把“virtual_server 192.168.200.100 443 {”这一行以下的全部注释掉

然后修改内容如下

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_02

为了完整,把这后面的信息补全

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_03

然后复制这个文件到另一个主机

scp keepalived.conf 172.16.249.147:/etc/keepalived/

然后稍微修改下

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_04

这样就可以了,这样就可以启动测试了

systemctl start keepalived.service;ssh172.16.249.147 'service  keepalivedstart'  因为我使用的是一个centos7和一个centos6,所以前后的启动命令不一样,如果使用的是同样的就不需这样了

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_05

使用  ps   aux  查看keepalived是否启动

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_06

然后在centos7上使用ifconfig进行查看,可以看到地址加上来了

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_07

然后我们把centos7下的keepalived关闭,然后查看地址

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_08

地址没有了,去centos6上查看会发现地址在6上看,这就是高可用了,就是当一台负载均衡器下线时,另一台能立即启用,实现高可用

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_09

 

没有日志信息,我们自己定义一个

首先修改/etc/sysconfig/keepalived这个文件,

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_10

然后修改/etc/rsyslog.conf这个文件

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_11

然后使用systemctl restart rsyslog.service重启日志服务

另一个主机同样修改重启日志服务使用(service rsyslog restart

然后重启keepalived服务查看日志就有了

 

然后做两个虚拟路由,即双主模式

修改keepalived的配置文件,增加下面一段

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_12

然后另一台主机修改keepalived的配置文件为下面一段

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_13

然后重启服务就可以查看了(如果因为网络环境比较复杂的话,等的时间可能要长一点)

 

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_14

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_15

如果停掉一台的keepalived服务,另一台会包括所有的地址

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_16

 

这样停服务在生产中,是不行的,下面来添加脚本实现动态切换

修改配置文件,添加下面配置

这一段要加在vrrp_instance 字段外面

 

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_17

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_18

 

另一个主机也做这样的配置

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_19

这就可以启用了

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_20

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_21


然后在172.16.249.195的/etc/keepalived目录下创建一个down文件,然后查看地址

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_22

另一台主机上就有两个地址了

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_23


定义邮件的发送,需要自行定义,需要自己写一个脚本

脚本内容:

#!/bin/bash

    #Author: MageEdu <linuxedu@foxmail.com>

    #description: An example of notify script

    #

    vip=172.16.100.88

    contact='root@localhost'

    notify(){

        mailsubject="`hostname` to be $1: $vipfloating"

        mailbody="`date '+%F %H:%M:%S'`: vrrptransition, `hostname` changed to be $1"

        echo $mailbody | mail -s"$mailsubject" $contact

    }

    case"$1" in

       master)

        notify master

        exit 0

        ;;

      backup)

        notify backup

        exit 0

        ;;

      fault)

        notify fault

        exit 0

        ;;

      *)

        echo 'Usage: `basename $0`{master|backup|fault}'

        exit 1

        ;;

    esac

/etc/keepalived/目录下创建一个名为notify.sh的脚本

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_24

给这个脚本一个执行权限chmod +x notify.sh

 

配置文件中vrrp_instance VI_1这一段里写入下面三行

notify_master"/etc/keepalived/notify.sh master"

notify_backup"/etc/keepalived/notify.sh backup"

notify_fault"/etc/keepalived/notify.shfault"

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_25 

验证下,

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_26

然后把这个脚本复制到另一个负载均衡器上,然后修改配置文件

scp -rp notify.sh172.16.249.147:/etc/keepalived/

这样邮件的发送就弄好了

另一个虚拟路由就不再演示了,跟这个步骤一样

 


我们加入lvsnginx来实现高可用的负载均衡器的实现(做的是双主模型)


下面先以lvs做负载均衡器,进行高可用负载均衡,以lvsDR模式为例,规划如下


Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_27

首先两个RS打开web服务,配好测试页,访问测试以下,没问题了,

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_28 Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_29

 

然后配置第一个Director(负载均衡器),安装ipvsadm,配置lvs规则(如果不会配置可以上网搜索,或者查看本人的lvs的博客),首先测试是能正常工作的

首先加入VIP

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_30

用其他主机ping测试一下,这个地址通不通,通了就没问题了,就差lvs规则了,

 

然后配置两个RS,首先配置arp_ignorearp_annoce,然后配置VIP和路由

写一个脚本实现一些功能,然后直接在两台RS上运行一下,就可以了

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_31

然后 bash  set.sh  start  然后cat看一下是否成功了;把这个脚本复制给另一台RS执行一次

然后添加地址和路由

ifconfig lo:0 172.16.249.18 netmask255.255.255.255 broadcast 172.16.249.18

route add -host 172.16.249.18 dev lo:0

ifconfig lo:1 172.16.249.19 netmask255.255.255.255 broadcast 172.16.249.19

route add -host 172.16.249.19 dev lo:1

 

另一台RS也添加地址和路由

ifconfig lo:0 172.16.249.18 netmask255.255.255.255 broadcast 172.16.249.18

route add -host 172.16.249.18 dev lo:0

ifconfig lo:1 172.16.249.19 netmask255.255.255.255 broadcast 172.16.249.19

route add -host 172.16.249.19 dev lo:1

因为做的是两个虚拟路由器的即双主的,就是通过DNS在前面分流,一部分走172.16.249.18这个地址一部分走172.16.249.19,所以RS要配置两个VIP

             

然后RS的配置就做完了,然后回到Director上写入ipvs规则

ipvsadm -A -t 172.16.249.18:80 -s rr

ipvsadm -a -t 172.16.249.18:80 -r172.16.249.116 -g

ipvsadm -a -t 172.16.249.18:80 -r172.16.249.123 -g

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_32

然后测试是否能正常轮调

如果用物理机的浏览器访问不轮调,是网络的影响,开一个虚拟机进行测试,如果没有问题,就证明lvsdr模式的负载均衡做好了

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_33

另一台lvs服务器也做一下这个规则,然后测试一下,证明lvs的负载均衡没有问题了

然后我们把lvs服务器上的规则清掉,地址删除

然后在两台lvs上做sorry  server 启动httpd服务,然后写一个页面,然后访问


Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_34


Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_35

测试没问题,就可以了

 

下面做keepalived高可用的负载均衡器(安装keepalived

直接做成两台虚拟路由器的,

修改第一台lvs主机的keepalived的配置文件如下

global_defs {

  notification_email {

         root@localhost

   }

  notification_email_from kaadmin@localhost

  smtp_server 127.0.0.1

  smtp_connect_timeout 30

  router_id centos7

}

vrrp_script chk_schedown {

   script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0"

   interval 1

   weight -2

    }

 

vrrp_instance VI_1 {

   state MASTER

   interface eth0

   virtual_router_id 51

   priority 100

   advert_int 1

   authentication {

       auth_type PASS

       auth_pass 5b03429a7cf5

    }

   virtual_ipaddress {

         172.16.249.18/16dev eth0 label eth0:0

    }

   track_script {

       chk_schedown

    }

   notify_master "/etc/keepalived/notify.sh master"

   notify_backup "/etc/keepalived/notify.sh backup"

   notify_fault "/etc/keepalived/notify.sh fault

}

 

vrrp_instance VI_2 {

   state BACKUP

   interface eth0

   virtual_router_id 61

   priority 99

   advert_int 1

   authentication {

       auth_type PASS

       auth_pass 20b662c6c06d

    }

   virtual_ipaddress {

         172.16.249.19/16dev eth0 label eth0:1

    }

}

 

virtual_server 172.16.249.18 80 {

   delay_loop 6

   lb_algo wrr

   lb_kind DR

   nat_mask 255.255.0.0

   protocol TCP

 

   real_server 172.16.249.116 80 {

       weight 1

       HTTP_GET {

           url {

              path /

               status_code 200

           }

           connect_timeout 3

           nb_get_retry 3

           delay_before_retry 3

       }

    }

   real_server 172.16.249.123 80 {

       weight 2

       HTTP_GET {

           url {

              path /

               status_code 200

           }

           connect_timeout 3

           nb_get_retry 3

           delay_before_retry 3

       }

    }

}

 

virtual_server 172.16.249.19 80 {

   delay_loop 6

   lb_algo wrr

   lb_kind DR

   nat_mask 255.255.0.0

   protocol TCP

 

   real_server 172.16.249.116 80 {

       weight 1

       HTTP_GET {

           url {

              path /

               status_code 200

           }

           connect_timeout 3

           nb_get_retry 3

           delay_before_retry 3

       }

    }

   real_server 172.16.249.123 80 {

       weight 2

       HTTP_GET {

           url {

              path /

               status_code 200

           }

           connect_timeout 3

           nb_get_retry 3

           delay_before_retry 3

       }

    }

}

然后配置另一台lvs主机中的keepalived如下:

global_defs {

  notification_email {

         root@localhost

   }

  notification_email_from kaadmin@localhost

  smtp_server 127.0.0.1

  smtp_connect_timeout 30

  router_id centos6.1.com

}

 

vrrp_script chk_schedown {

   script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0"

   interval 1

   weight -2

    }

 

vrrp_instance VI_1 {

   state BACKUP

   interface eth0

   virtual_router_id 51

   priority 99

   advert_int 1

   authentication {

       auth_type PASS

       auth_pass 5b03429a7cf5

    }

   virtual_ipaddress {

         172.16.249.18/16dev eth0 label eth0:0

    }

  track_script {

       chk_schedown

    }

   notify_master "/etc/keepalived/notify.sh master"

   notify_backup "/etc/keepalived/notify.sh backup"

   notify_fault "/etc/keepalived/notify.sh fault"

 

}

 

vrrp_instance VI_2 {

   state MASTER

   interface eth0

   virtual_router_id 61

   priority 100

   advert_int 1

   authentication {

       auth_type PASS

       auth_pass 20b662c6c06d

    }

   virtual_ipaddress {

       172.16.249.19/16 dev eth0 label eth0:1

    }

}

 

virtual_server 172.16.249.18 80 {

   delay_loop 6

   lb_algo wrr

   lb_kind DR

   nat_mask 255.255.0.0

   protocol TCP

 

   real_server 172.16.249.116 80 {

       weight 1

       HTTP_GET {

           url {

              path /

               status_code 200

           }

           connect_timeout 3

           nb_get_retry 3

           delay_before_retry 3

       }

    }

   real_server 172.16.249.123 80 {

       weight 2

       HTTP_GET {

           url {

              path /

               status_code 200

           }

            connect_timeout 3

           nb_get_retry 3

           delay_before_retry 3

       }

    }

}

 

virtual_server 172.16.249.19 80 {

   delay_loop 6

   lb_algo wrr

   lb_kind DR

   nat_mask 255.255.0.0

   protocol TCP

 

   real_server 172.16.249.116 80 {

       weight 1

       HTTP_GET {

           url {

              path /

               status_code 200

           }

           connect_timeout 3

           nb_get_retry 3

           delay_before_retry 3

       }

    }

    real_server 172.16.249.123 80 {

       weight 2

       HTTP_GET {

           url {

              path /

               status_code 200

           }

           connect_timeout 3

           nb_get_retry 3

           delay_before_retry 3

       }

    }

}

这只在第一个虚拟路由器中做脚本,第二个没有做,要想做了,跟第一个的一样

然后就可以启动keepalived服务了

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_36

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_37

然后访问测试一下

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_38

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_39

可以看到两个地址都能负载均衡


下面让一台lvskeepalived下线

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_40

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_41

这样两个地址都在一台lvs上了,访问测试下

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_42

可以看到也没有问题,

如果把一台RS下线,所有请求就都在一个RS上了;如果RS都下线,sorry  server就可以发挥作用了。(这里不再演示了)

这样高可用的lvs的负载均衡就做好了

 

 

下面做下nginx的高可用的负载均衡,就是用nginx做负载均衡器

使用nginx做负载均衡器,架构如下:

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_43

安装nginx,然后启动起来,配置成负载均衡服务器

修改/etc/nginx/nginx.conf

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_44

另一台负载均衡器同样这样配置

 

然后修改这一台nginx的keepalived的配置文件

global_defs {

  notification_email {

         root@localhost

   }

  notification_email_from kaadmin@localhost

  smtp_server 127.0.0.1

  smtp_connect_timeout 30

  router_id centos7

}

vrrp_script chk_nginx {

   script "killall -0 nginx &> /dev/null"

   interval 1

   weight -10

    }

 

vrrp_instance VI_1 {

   state MASTER

   interface eth0

   virtual_router_id 51

   priority 100

    advert_int 1

   authentication {

       auth_type PASS

       auth_pass 5b03429a7cf5

    }

   virtual_ipaddress {

         172.16.249.18/16dev eth0 label eth0:0

    }

   track_script {

          chk_nginx

    }

   notify_master "/etc/keepalived/notify.sh master"

   notify_backup "/etc/keepalived/notify.sh backup"

   notify_fault "/etc/keepalived/notify.sh fault

}

 

另一台nginxkeepalived配置文件为

global_defs {

  notification_email {

         root@localhost

   }

  notification_email_from kaadmin@localhost

  smtp_server 127.0.0.1

  smtp_connect_timeout 30

  router_id centos6.1.com

}

 

vrrp_script chk_nginx {

   script "killall -0 nginx &> /dev/null"

   interval 1

   weight -10

    }

 

vrrp_instance VI_1 {

   state BACKUP

   interface eth0

   virtual_router_id 51

   priority 99

   advert_int 1

   authentication {

       auth_type PASS

       auth_pass 5b03429a7cf5

    }

   virtual_ipaddress {

         172.16.249.18/16dev eth0 label eth0:0

    }

  track_script {

         chk_nginx

    }

   notify_master "/etc/keepalived/notify.sh master"

   notify_backup "/etc/keepalived/notify.sh backup"

   notify_fault "/etc/keepalived/notify.sh fault"

 

}

然后启动keepalived

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_45

可以看到地址配好了,测试下

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_46Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_47


可以看到做好了,然后把一台nginxnginx服务停掉,看另一台能否上线

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_48Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_49

访问也没有问题,这样nginx的高可用就做好了,

 

做双主模式的,只需要在keepalived配置文件里加入下面这些内容就行了

第一台(172.16.249.195

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_50

第二台(172.16.249.147

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_51

然后重启keepalived服务

查看ip

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_52Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_53


访问测试,

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_54

让一台的nginx服务停掉

Keepalived的相关应用,使用keepalived实现nginx和lvs的高可用负载均衡器_keepalived+nginx和kee_55

可以看到能正常切换,这样nginx高可用的双主模式就做好了