zabbix 通过keepalived实现高可用 keepalived怎么实现高可用

转载

jack 2024-05-10 03:45:39

1、keepalived软件介绍

2、keepalived的安装及VRRP的实现

3、总结

1、keepalived软件介绍

keepalived是由c语言编写的一个路径选择软件，是IPVS的一个扩展性项目，为IPVS提供高可用性（故障转移）特性，它的高可用性是通过VRRP协议实现的，并实现了对负载均衡服务器池中的real server进行健康状态检测，当real server不可用时，自身实现了故障的隔离，这弥补了IPVS不能对real server服务器进行健康检测的不足，这也是keepalived运用最广泛的场景，当然keepalived不只限于实现IPVS的高可用。

1.1、keepalived软件架构

下部分是内核空间，有IPVS和NETLINK两个部分组成，IPVS提供IP虚拟网络，而NETLINK提供高级的路由功能和其他相关的网络功能。上部分是用户空间，由这些组件来完成具体的功能，在核心组件中：

1、WatchDog：实现对healthchecking和VRRP进程的监控，如果子进程非法停止，父进程会重启子进程。

2、 Checkers 负责real server的 healthchecking，是 keepalived 最主要的功能，它实时对real server进行测试，判断其是否存活后实现对lvs规则的添加或删除。可利用OSI模型中的第四、五、七层来进行测试。healthchecking进程由父进程监控一个独立的进程运行。

3、 VRRP Stack 负责负载均衡器之间的失败切换( FailOver)，由父进程监控一个独立的进程运行。

4、 IPVS wrapper 用来发送设定的规则到内核 ipvs 代码。

5、Netlink Reflector 用来设定 vrrp 的 vip 地址等。

为了keepalived的健状性和稳定性，keepalived启动后运行了三个守护进程，一个父进程，两个child进程，父进程用于监控两个child进程，两个child进程中一个是vrrp child，一个是healthchecking child。

2、keepalived的安装及VRRP的实现

VRRP(Virtual Router Redundancy Protocol)全称为“虚拟路由冗余协议”，对VRRP这里不做过多的阐述，可自行google了解，简单说来就是一种能在多个运行VRRP的设备上使用一个虚拟的IP地址对外提供服务，这个虚拟的IP地址绑定在这一组运行了VRRP组设备中的MASTER节点上，当这个节点发生故障时，其他的BACKUP节点中的一个能够接管出现故障的设备，使服务不会中断。运行VRRP协议的设备一般是专业的路由设备，比如华为、锐捷、CISCO的路由产品，而keepalived这个软件把VRRP协议在linux主机上得以实现，使linux主机也具备了高可用的特性。下边让我们来看一下keepalived是怎样实现VRRP的，准确的说应该是VRRP v2版本。

2.1、keepalived的安装

在现在主流的centos系统中，keepalived已被收录进yum源，可直接用yum进行安装，只是得到的版本不是最新的稳定版，此处以目前最稳定版的编译安装方式作介绍。目前最稳定的是"keepalived-1.2.16.tar.gz"。

[root@nod1 software]# tar xf keepalived-1.2.16.tar.gz
[root@nod1 software]# cd keepalived-1.2.16
[root@nod1 keepalived-1.2.16]# ls
AUTHOR  ChangeLog  configure.in  COPYING  genhash  install-sh  keepalived.spec.in  Makefile.in  TODO
bin     configure  CONTRIBUTORS  doc      INSTALL  keepalived  lib                 README       VERSION

在编译时可能会有决依赖关系，一般要安装以下这几个包： libnl-devel openssl-devel

[root@nod1 keepalived-1.2.16]# ./configure --prefix=/usr/local/keepalived
编译完成后会出现如下信息：
Keepalived configuration
------------------------
Keepalived version       : 1.2.16
Compiler                 : gcc
Compiler flags           : -g -O2 -DFALLBACK_LIBNL1
Extra Lib                : -lssl -lcrypto -lcrypt  -lnl  
Use IPVS Framework       : Yes
IPVS sync daemon support : Yes
IPVS use libnl           : Yes
fwmark socket support    : Yes
Use VRRP Framework       : Yes
Use VRRP VMAC            : Yes
SNMP support             : No
SHA1 support             : No
Use Debug flags          : No
[root@nod1 keepalived-1.2.16]# make && make install
[root@nod1 keepalived]# pwd
/usr/local/keepalived
[root@nod1 keepalived]# ll
total 16
drwxr-xr-x 2 root root 4096 May 26 08:32 bin
drwxr-xr-x 5 root root 4096 May 26 08:32 etc
drwxr-xr-x 2 root root 4096 May 26 08:32 sbin
drwxr-xr-x 3 root root 4096 May 26 08:32 share
[root@nod1 keepalived-1.2.16]# /usr/local/keepalived/sbin/keepalived -v
Keepalived v1.2.16 (05/26,2015)

安装好后，先来测试一下看keepalived是否能正常启动：

[root@nod1 keepalived]# /usr/local/keepalived/sbin/keepalived -D
[root@nod1 keepalived]# ps aux | grep keepalived
root      8130  0.0  0.2  44844  1032 ?        Ss   08:43   0:00 /usr/local/keepalived/sbin/keepalived -D
root      8131  0.1  0.4  47072  2384 ?        S    08:43   0:00 /usr/local/keepalived/sbin/keepalived -D
root      8132  2.3  0.3  46948  1560 ?        S    08:43   0:00 /usr/local/keepalived/sbin/keepalived -D
root      8143  0.0  0.1 103236   860 pts/0    S+   08:43   0:00 grep keepalived
#出现了三个守护进程，用以下命令可看出三个进程之间的关联：
[root@nod1 keepalived]# pstree | grep keepalived
     |-keepalived---2*[keepalived]

2.2、VRRP的实现

[root@nod1 keepalived]# pwd
/usr/local/keepalived
[root@nod1 keepalived]# ls
bin  etc  sbin  share
[root@nod1 keepalived]# tree etc -L 3
etc
├── keepalived
│   ├── keepalived.conf
│   └── samples
│       ├── client.pem
│       ├── dh1024.pem
│       ├── keepalived.conf.fwmark
│       ├── keepalived.conf.HTTP_GET.port
│       ├── keepalived.conf.inhibit
│       ├── keepalived.conf.IPv6
│       ├── keepalived.conf.misc_check
│       ├── keepalived.conf.misc_check_arg
│       ├── keepalived.conf.quorum
│       ├── keepalived.conf.sample
│       ├── keepalived.conf.SMTP_CHECK
│       ├── keepalived.conf.SSL_GET
│       ├── keepalived.conf.status_code
│       ├── keepalived.conf.track_interface
│       ├── keepalived.conf.virtualhost
│       ├── keepalived.conf.virtual_server_group
│       ├── keepalived.conf.vrrp
│       ├── keepalived.conf.vrrp.localcheck
│       ├── keepalived.conf.vrrp.lvs_syncd
│       ├── keepalived.conf.vrrp.routes
│       ├── keepalived.conf.vrrp.scripts
│       ├── keepalived.conf.vrrp.static_ipaddress
│       ├── keepalived.conf.vrrp.sync
│       ├── root.pem
│       └── sample.misccheck.smbcheck.sh
├── rc.d
│   └── init.d
│       └── keepalived
└── sysconfig
    └── keepalived

从上边的输出信息中可知在etc这个目录下有个keepalived.conf配置文件，在samples目录下有一堆的配置样例，在etc/rc.d/init.d/下还有一个启动脚本。

下边来测试keepalived软件的VRRP的实现，我这里有两个节点，一个ip为192.168.0.200，另一个ip为192.168.0.201，两个节点都以编译安装好keepalived软件。

一个完善的keepalived.conf的配置文件一般分为三个配置块，一是全局定义块，二是vrrp实例定义块，三是虚拟服务器定义块，在各配置段中还有一些了配置段，而有些配置段不是必须的。如果只是想实现vrrp，那配置文件（192、168.0.200节点上）中只保留以下内容即可：

global_defs {
   router_id LVS_VRRP_1
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 123
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.0.222
    }
}

注释：

在全局定义块中“router_id”是表示运行vrrp主机的惟一标识，在vrrp实例块中“state MASTER”表示主机运行后的角色是MASTER。

“ virtual_router_id 123”表示虚拟路由标识，表示在"VI_1"这个vrrp实例中的一个唯一标识，在主节点和备节点中这个号码是相同的，且在整个VRRP中也是唯一的，号码取值范围为0-255。

“ priority 150 ”表示vrrp实例中的优先级，数字越大，优先级越高。

在备节点（192.168.0.201）上的配置如下：

[root@nod2 ~]# vim /usr/local/keepalived/etc/keepalived/keepalived.conf
global_defs {
   router_id LVS_VRRP_2    !与主节点不同
}
vrrp_instance VI_1 {
    state BACKUP    !与主节点不同
    interface eth0
    virtual_router_id 123
    priority 140     ! 小于主节点优先级
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.0.222
    }
}

备节点上一般有三处与主节点不同。

两个节点的配置文件都准备好后，就可以启动keepalived了，如下：

[root@nod2 ~]# /usr/local/keepalived/sbin/keepalived -D -f /usr/local/keepalived/etc/keepalived/keepalived.conf

选项“-D”：表示把启动信息打印到日志记录，即/var/log/messages

选项“-f”：表示读取的配置文件，默认时keepalived会去读取/etc/keepalived/keepalived.conf文件

[root@nod1 ~]# /usr/local/keepalived/sbin/keepalived -D -f /usr/local/keepalived/etc/keepalived/keepalived.conf

接下来验证keepalived是否正常工作，因为在启动时用了“-D”选项，所以我们去查看/var/log/messages文件即可：

下边是nod1上的日志输出信息：

May 26 09:06:15 nod1 Keepalived[8229]: Starting Keepalived v1.2.16 (05/26,2015)
May 26 09:06:15 nod1 Keepalived[8230]: Starting Healthcheck child process, pid=8231
May 26 09:06:15 nod1 Keepalived_vrrp[8232]: Netlink reflector reports IP 192.168.0.200 added
May 26 09:06:15 nod1 Keepalived_vrrp[8232]: Netlink reflector reports IP fe80::20c:29ff:fe92:a73d added
May 26 09:06:15 nod1 Keepalived_vrrp[8232]: Netlink reflector reports IP fe80::20c:29ff:fe92:a747 added
May 26 09:06:15 nod1 Keepalived_vrrp[8232]: Registering Kernel netlink reflector
May 26 09:06:15 nod1 Keepalived_vrrp[8232]: Registering Kernel netlink command channel
May 26 09:06:15 nod1 Keepalived_vrrp[8232]: Registering gratuitous ARP shared channel
May 26 09:06:15 nod1 Keepalived[8230]: Starting VRRP child process, pid=8232
May 26 09:06:15 nod1 Keepalived_vrrp[8232]: Opening file '/usr/local/keepalived/etc/keepalived/keepalived.conf'.
May 26 09:06:15 nod1 Keepalived_vrrp[8232]: Configuration is using : 61889 Bytes
May 26 09:06:15 nod1 Keepalived_vrrp[8232]: Using LinkWatch kernel netlink reflector...
May 26 09:06:16 nod1 Keepalived_vrrp[8232]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
May 26 09:06:16 nod1 Keepalived_vrrp[8232]: VRRP_Instance(VI_1) Transition to MASTER STATE
May 26 09:06:16 nod1 Keepalived_vrrp[8232]: VRRP_Instance(VI_1) Received lower prio advert, forcing new election
May 26 09:06:17 nod1 Keepalived_vrrp[8232]: VRRP_Instance(VI_1) Entering MASTER STATE
.....
May 26 09:06:17 nod1 Keepalived_vrrp[8232]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.0.222

从日志输出可知，nod1已成为了MASTER，且把192.168.0.222这个虚拟地址配置在了eth0上。

以下是nod2上的日志输出信息：

May 26 09:05:32 nod2 Keepalived[2819]: Starting Keepalived v1.2.16 (05/26,2015)
May 26 09:05:32 nod2 Keepalived[2820]: Starting Healthcheck child process, pid=2821
May 26 09:05:32 nod2 Keepalived[2820]: Starting VRRP child process, pid=2822
May 26 09:05:32 nod2 Keepalived_healthcheckers[2821]: Netlink reflector reports IP 192.168.0.201 added
May 26 09:05:32 nod2 Keepalived_healthcheckers[2821]: Netlink reflector reports IP fe80::20c:29ff:fec6:f77a added
May 26 09:05:32 nod2 Keepalived_healthcheckers[2821]: Registering Kernel netlink reflector
May 26 09:05:32 nod2 Keepalived_healthcheckers[2821]: Registering Kernel netlink command channel
May 26 09:05:32 nod2 Keepalived_vrrp[2822]: Netlink reflector reports IP 192.168.0.201 added
May 26 09:05:32 nod2 Keepalived_vrrp[2822]: Netlink reflector reports IP fe80::20c:29ff:fec6:f77a added
May 26 09:05:32 nod2 Keepalived_vrrp[2822]: Registering Kernel netlink reflector
May 26 09:05:32 nod2 Keepalived_vrrp[2822]: Registering Kernel netlink command channel
May 26 09:05:32 nod2 Keepalived_vrrp[2822]: Registering gratuitous ARP shared channel
May 26 09:05:32 nod2 Keepalived_healthcheckers[2821]: Opening file '/usr/local/keepalived/etc/keepalived/keepalived.conf'.
May 26 09:05:32 nod2 Keepalived_healthcheckers[2821]: Configuration is using : 6120 Bytes
May 26 09:05:32 nod2 Keepalived_vrrp[2822]: Opening file '/usr/local/keepalived/etc/keepalived/keepalived.conf'.
May 26 09:05:32 nod2 Keepalived_vrrp[2822]: Configuration is using : 61761 Bytes
May 26 09:05:32 nod2 Keepalived_vrrp[2822]: Using LinkWatch kernel netlink reflector...
May 26 09:05:32 nod2 Keepalived_healthcheckers[2821]: Using LinkWatch kernel netlink reflector...
May 26 09:05:32 nod2 Keepalived_vrrp[2822]: VRRP_Instance(VI_1) Entering BACKUP STATE
May 26 09:05:32 nod2 Keepalived_vrrp[2822]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
May 26 09:05:36 nod2 Keepalived_vrrp[2822]: VRRP_Instance(VI_1) Transition to MASTER STATE
May 26 09:05:37 nod2 Keepalived_vrrp[2822]: VRRP_Instance(VI_1) Entering MASTER STATE
May 26 09:05:37 nod2 Keepalived_vrrp[2822]: VRRP_Instance(VI_1) setting protocol VIPs.
May 26 09:05:37 nod2 Keepalived_healthcheckers[2821]: Netlink reflector reports IP 192.168.0.222 added
May 26 09:05:37 nod2 Keepalived_vrrp[2822]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.0.222
May 26 09:05:39 nod2 ntpd[1086]: Listen normally on 7 eth0 192.168.0.222 UDP 123
May 26 09:05:39 nod2 ntpd[1086]: peers refreshed
May 26 09:05:42 nod2 Keepalived_vrrp[2822]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.0.222
May 26 09:06:41 nod2 Keepalived_vrrp[2822]: VRRP_Instance(VI_1) Received higher prio advert
May 26 09:06:41 nod2 Keepalived_vrrp[2822]: VRRP_Instance(VI_1) Entering BACKUP STATE
May 26 09:06:41 nod2 Keepalived_vrrp[2822]: VRRP_Instance(VI_1) removing protocol VIPs.

因我是先启动的nod2节点，再启动的nod1节点，所以在nod2的日志输出中看到它先把自己设定为了MASTER角色，然后当nod1启动后，nod2接收到了一个优先级比自己高的通知，nod2再把自己降级为BACKUP。

再去nod1上查看一相网卡上是否配置了虚拟IP，记住keepalived配置的虚拟IP，用ifconfig命令是无法查看的：

[root@nod1 keepalived]# ip add list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:0c:29:92:a7:3d brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.200/24 brd 192.168.0.255 scope global eth0
    inet 192.168.0.222/32 scope global eth0
    inet6 fe80::20c:29ff:fe92:a73d/64 scope link
       valid_lft forever preferred_lft forever

最后再来验证一下VRRP是否能实现高可用，在nod1节点上停止keepalived进程：

[root@nod1 keepalived]# killall keepalived
[root@nod1 keepalived]# ip add list  #虚拟ip地址已被释放
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:0c:29:92:a7:3d brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.200/24 brd 192.168.0.255 scope global eth0
    inet6 fe80::20c:29ff:fe92:a73d/64 scope link
       valid_lft forever preferred_lft forever
[root@nod2 ~]# ip add list  #nod2上被配置了虚拟ip
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:0c:29:c6:f7:7a brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.201/24 brd 192.168.0.255 scope global eth0
    inet 192.168.0.222/32 scope global eth0
    inet6 fe80::20c:29ff:fec6:f77a/64 scope link
       valid_lft forever preferred_lft forever

以上测试证明了keepalived利用VRRP实现了高可用的功能。

2.3、keepalived.conf配置段详解

上边已说到过keepalived.conf一般由全局配置段、vrrp实例配置段、虚拟服务配置段组成，而各配置段中还可以嵌套另外的配置段，而有些配置段又不是必须的。虚拟服务配置是专门针对LVS设计的，如果你所在的环境不是让LVS具有高可用性，那虚拟服务配置段就可以不要，如果你想让keepalived每一次状态的改变（MASTER-BACKUP间的转变）都以邮件的方式通知管理员，那可增加关于邮件报警相关的配置段，总之，keepalived.conf的配置十分灵活，下边对常用的配置进行说明。

下边以一个实例说明：

!全局配置段
global_defs {
   notification_email {
       admin@test.com
   }
   notification_email_from admin@test.com
   smtp_server smtp.163.com
       #定义所使用的smtp服务器
   smtp_connect_timeout 30
   router_id NGINX_NUM1
     #vrrp路由的唯一标识，在各服务器上运行vrrp的主机上应该是唯一的
}
! vrrp的脚本检测配置段，以后边vrrp实例中调用
vrrp_script chk_nginx {
    script "killall -0 nginx"  #定义了一个检测nginx进程是否存在的块
    interval 1  #检测的间隔时间
    weight -5   #如果nginx进程不存在，刚把相应的vrrp实例的优先级减去5 
    fall 2      #如果两次检测nginx进程都不在，则认为nginx不可用   
    rise 1      #如果1次检测到nginx进程，则认为nginx可用
        
}
! vrrp实例配置段
vrrp_instance VI_1 {
    state MASTER   #MASTER或BACKUP一定要大写
    interface eth0
    virtual_router_id 10
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 111111
    }
    virtual_ipaddress {
        192.168.0.222
    }
    track_script {
        chk_nginx  #这里调用了上边定义的chk_nginx
    }
    notify_master "/bin/sh /etc/keepalived/scripts/notify.sh master"
    #这里定义当vrrp转变为master角色时所要进行的操作，这里执行了一个脚本，脚本的内容就是向管理员发送了邮件
    notify_backup "/bin/sh /etc/keepalived/scripts/notify.sh backup"
  #这里定义当vrrp转变为backup角色时所要进行的操作
    notify_fault "/bin/sh /etc/keepalived/scripts/notify.sh fault"
}
 
!lvs虚拟服务配置段
virtual_server 192.168.0.222 80 {
    delay_loop 6
    lb_algo rr
     #定义调度算法
    lb_kind DR  #定义lvs的模型
    nat_mask 255.255.255.0
    persistence_timeout 50
    protocol TCP
   
    sorry_server 127.0.0.1 80
    real_server 192.168.0.202 80 {
   #定义一个real server
        weight 1
        HTTP_GET {
   #定义real server的检测机制，这里是七层检测机制，如果后端的real server不是运行的http服务，那可用以TCP_CHECK
            url {
              path /
              status_code 200
            }
            connect_timeout 3
   #健康检测时的超时时间
            nb_get_retry 3
        #重试次数
            delay_before_retry 3
   #延迟重试，如果一次检测中不成功，则延迟此值后再重试
        }
    }
    real_server 192.168.0.203 80 {
        weight 1
        HTTP_GET {
            url {
              path /
              status_code 200
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }

从这个配置文件中也可看出keepalived不仅可以为LVS提供高可用，也可以给nginx做高可用，只是keepalived和LVS结合时自身带了健康检测的机制，这也是healthcheck子进程所在的意义，而对非LVS做高可用时则需要借助vrrp_script这个配置段和相应的脚本来实现。

3、总结

至此，keepalived的软件架构，安装及配置文件的结构进行了简单说明，通过此博文可知，keepalived在与LVS结合时是最佳选择，因为keepalived自带了对real server的健康检测机制，这正好弥补了LVS的不足，而要知道的是keepalived不仅仅适用与LVS结合实现高可用，在其他需要高可用的环境keepalived依然可用，但总结起来keepalived更适用与作为负载均衡调度器应用类的高可用方案，比如nginx作为反向代理时，haproxy等这样的应用，在接下来的博客中将会以两个实例来说明利用keepalived实现nginx和lvs的高可用。

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。