1、集群类型
LB：Load Balance 负载均衡 LVS/HAProxy/nginx（http/upstream, stream/upstream） HA：High Availability 高可用集群 数据库、Zookeeper、Redis KeepAlived 通用的高可用集群 SPoF: Single Point of Failure，解决单点故障 HPC：High Performance Computing 高性能集群 https://www.top500.org
2、Keepalived 介绍
vrrp 协议的软件实现，原生设计目的为了高可用 ipvs服务 官网：http://keepalived.org/ 功能： 基于vrrp协议完成地址流动 为vip地址所在的节点生成ipvs规则(在配置文件中预先定义) 为ipvs集群的各RS做健康状态检测 基于脚本调用接口完成脚本中定义的功能，进而影响集群事务，以此支持nginx、haproxy等服务
用户空间核心组件： vrrp stack：VIP消息通告 checkers：监测 Real Server system call：实现 vrrp 协议状态转换时调用脚本的功能 SMTP：邮件组件 IPVS wrapper：生成 IPVS 规则 Netlink Reflector：网络接口 WatchDog：监控进程
控制组件：提供keepalived.conf 的解析器，完成Keepalived配置
IO复用器：针对网络目的而优化的自己的线程抽象
内存管理组件：为某些通用的内存管理功能（例如分配，重新分配，发布等）提供访问权限
各节点时间必须同步：ntp, chrony
关闭防火墙及SELinux
各节点之间可通过主机名互相通信：非必须
建议使用/etc/hosts文件实现：非必须
各节点之间的root用户可以基于密钥认证的ssh服务完成互相通信：非必须
2-1、Keepalived 相关文件
软件包名：keepalived 主程序文件：/usr/sbin/keepalived 主配置文件：/etc/keepalived/keepalived.conf 配置文件示例：/usr/share/doc/keepalived/ Unit File：/lib/systemd/system/keepalived.service Unit File的环境配置文件： /etc/sysconfig/keepalived CentOS /etc/default/keepalived Ubuntu
注意：CentOS 7 上有 bug，可能有下面情况出现
systemctl restart keepalived #新配置可能无法生效
systemctl stop keepalived;systemctl start keepalived #无法停止进程，需要 kill停止
3、Keepalived 安装
3-1、包安装
[root@ka1 ~]#vim /etc/hosts
10.0.0.101 ka1.wang.org
10.0.0.102 ka2.wang.org
[root@ka1 ~]#hostnamectl set-hostname ka1.wang.org
[root@ka1 ~]#apt install chrony
[root@ka1 ~]#chronyc sources
[root@ka1 ~]#apt install keepalived
##默认没有配置文件无法启动
[root@ka1 ~]#systemctl status keepalived.service
#利用范例生成配置文件
[root@ka1 ~]#dpkg -L keepalived
[root@ka1 ~]#cp /usr/share/doc/keepalived/samples/keepalived.conf.sample /etc/keepalived/
[root@ka1 ~]#cd /etc/keepalived/
[root@ka1 keepalived]#mv keepalived.conf.sample keepalived.conf
[root@ka1 keepalived]#vim keepalived.conf
global_defs {
router_id ka1.wang.org
}
vrrp_instance ka1 {
interface eth0
virtual_router_id 66
priority 100
advert_int 1
virtual_ipaddress {
10.0.0.66
}
}
[root@ka1 keepalived]#systemctl restart keepalived.service
[root@ka1 keepalived]#systemctl status keepalived.service
3-2、编译安装
3-2-1、安装依赖包
[root@ka2 ~]#apt -y install make gcc ipvsadm build-essential pkg-config automake autoconf libipset-dev libnl-3-dev libnl-genl-3-dev libssl-dev libxtables-dev libip4tc-dev libip6tc-dev libipset-dev libmagic-dev libsnmp-dev libglib2.0-dev libpcre2-dev libnftnl-dev libmnl-dev libsystemd-dev
#红帽系列依赖包
]# yum install gcc curl openssl-devel libnl3-devel net-snmp-devel
3-2-2、下载解压
[root@ka2 ~]#wget https://keepalived.org/software/keepalived-2.2.7.tar.gz
[root@ka2 ~]#tar xf keepalived-2.2.7.tar.gz -C /usr/local/src/
3-2-3、编译安装
[root@ka2 ~]#cd /usr/local/src/
[root@ka2 src]#cd keepalived-2.2.7
[root@ka2 keepalived-2.2.7]#./configure --prefix=/usr/local/keepalived --disable-fwmark
#选项--disable-fwmark 可用于禁用iptables规则,可防止VIP无法访问,无此选项默认会启用iptables规则
[root@ka2 keepalived-2.2.7]#make && make install
[root@ka2 keepalived-2.2.7]#cd
[root@ka2 ~]#/usr/local/keepalived/sbin/keepalived -v
Keepalived v2.2.7 (01/16,2022)
#自动生成keepalived.service文件
[root@ka2 ~]#ll /usr/lib/systemd/system/keepalived.service
-rw-r--r-- 1 root root 567 10月 26 19:56 /usr/lib/systemd/system/keepalived.service
#默认无法启动，缺少配置文件，需要拷贝自行创建
[root@ka2 ~]#mkdir /etc/keepalived
[root@ka2 ~]#cp /usr/local/src/keepalived-2.2.7/doc/samples/keepalived.conf.sample /etc/keepalived/keepalived.conf
[root@ka2 ~]#ll /etc/keepalived/keepalived.conf
-rw-r--r-- 1 root root 910 10月 26 20:05 /etc/keepalived/keepalived.conf
[root@ka2 ~]#systemctl restart keepalived.service
[root@ka2 ~]#systemctl status keepalived.service
4、KeepAlived 配置说明
4-1、 配置文件组成部分
/etc/keepalived/keepalived.conf
配置文件组成
- GLOBAL CONFIGURATION Global definitions：定义邮件配置，route_id，vrrp配置，多播地址等
- VRRP CONFIGURATION VRRP instance(s)：定义每个vrrp虚拟路由器
- LVS CONFIGURATION Virtual server group(s) Virtual server(s)：LVS集群的VS和RS
man keepalived.conf
4-2、全局配置
#/etc/keepalived/keepalived.conf
global_defs {
notification_email {
root@localhost #keepalived 发生故障切换时邮件发送的目标邮箱，可以按行区分写多个
root@wang.org
}
notification_email_from keepalived@localhost #发邮件的地址
smtp_server 127.0.0.1 #邮件服务器地址
smtp_connect_timeout 30 #邮件服务器连接timeout
router_id ka1.wang.org #每个keepalived主机唯一标识，建议使用当前主机名，如果多节点重名可能会影响切换脚本执行
vrrp_skip_check_adv_addr #对所有通告报文都检查，会比较消耗性能，启用此配置后，如果收到的通告报文和上一个报文是同一个路由器，则跳过检查，默认值为全检查
vrrp_strict #严格遵守VRRP协议,启用此项后以下状况将无法启动服务:1.无VIP地址 2.配置了单播邻居 3.在VRRP版本2中有IPv6地址，开启动此项并且没有配置vrrp_iptables时会自动开启iptables防火墙规则，默认导致VIP无法访问,建议不加此项配置
vrrp_garp_interval 0 #gratuitous ARP messages 报文发送延迟，0表示不延迟
vrrp_gna_interval 0 #unsolicited NA messages （不请自来）消息发送延迟
vrrp_mcast_group4 224.0.0.18 #指定组播IP地址范围：224.0.0.0到239.255.255.255,默认值：224.0.0.18
vrrp_iptables #此项和vrrp_strict同时开启时，则不会添加防火墙规则,如果无配置vrrp_strict项,则无需启用此项配置，注意：新版加此项仍有iptables规则
}
4-3、配置虚拟路由器
vrrp_instance <STRING> { #<String>为vrrp的实例名,一般为业务名称配置参数
......
}
#配置参数：
state MASTER|BACKUP #当前节点在此虚拟路由器上的初始状态，状态为MASTER或者BACKUP
interface IFACE_NAME #绑定为当前虚拟路由器使用的物理接口，如：eth0,bond0,br0,可以和VIP不在一个网卡
virtual_router_id VRID #每个虚拟路由器唯一标识，范围：0-255，每个虚拟路由器此值必须唯一，否则服务无法启动，同属一个虚拟路由器的多个keepalived节点必须相同,务必要确认在同一网络中此值必须唯一
priority 100 #当前物理节点在此虚拟路由器的优先级，范围：1-254，每个keepalived主机节点此值不同
advert_int 1 #vrrp通告的时间间隔，默认1s
authentication { #认证机制
auth_type AH|PASS #AH为IPSEC认证(不推荐),PASS为简单密码(建议使用)
auth_pass <PASSWORD> #预共享密钥，仅前8位有效，同一个虚拟路由器的多个keepalived节点必须一样
}
virtual_ipaddress { #虚拟IP,生产环境可能指定几十上百个VIP地址
<IPADDR>/<MASK> brd <IPADDR> dev <STRING> scope <SCOPE> label <LABEL>
192.168.200.100 #指定VIP，不指定网卡，默认为eth0,注意：不指定/prefix,默认为/32
192.168.200.101/24 dev eth1 #指定VIP的网卡，建议和interface指令指定的网卡不在一个网卡
192.168.200.102/24 dev eth2 label eth2:1 # 指定VIP的网卡label
}
track_interface { #配置监控网络接口，一旦出现故障，则转为FAULT状态实现地址转移
eth0
eth1
...
}4-4、启用 Keepalived 日志功能
[root@ka2 ~]#vim /usr/local/keepalived/etc/sysconfig/keepalived #编译安装路径
KEEPALIVED_OPTIONS="-D -S 6"
[root@ka2 ~]#vim /etc/rsyslog.d/keepalived.conf
local6.* /var/log/keepalived.log
[root@ka2 ~]#systemctl restart rsyslog.service keepalived.service
4-5、实现 Keepalived 独立子配置文件
当生产环境复杂时， /etc/keepalived/keepalived.conf 文件中保存所有集群的配置会导致内容过多，不易管理 可以将不同集群的配置，比如：不同集群的VIP配置放在独立的子配置文件中 利用include 指令可以实现包含子配置文件
格式:
include /path/file
[root@ka2 ~]#vim /etc/keepalived/keepalived.conf
global_defs {
router_id ka2.wang.org
}
include /etc/keepalived/conf.d/*.conf
[root@ka2 ~]#mkdir /etc/keepalived/conf.d
[root@ka2 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance ka2.wang.org {
interface eth0
virtual_router_id 66
priority 80
advert_int 1
virtual_ipaddress {
10.0.0.66
}
}
[root@ka2 ~]#systemctl restart keepalived.service
5、Keepalived 实现 VRRP
5-1、实现master/slave的 Keepalived 单主架构
5-1-1、MASTER配置
[root@ka1 ~]#vim /etc/keepalived/keepalived.conf
global_defs {
router_id ka1.wang.org
}
include /etc/keepalived/conf.d/*.conf
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance ka1.wang.org {
state MASTER #在另一个结点上为BACKUP
interface eth0
virtual_router_id 66 #每个虚拟路由器必须唯一，同属一个虚拟路由器的多个keepalived节点必须相同
priority 100 #在另一个结点上为80(BACKUP优先级比master低)
advert_int 1
authentication { #预共享密钥认证，同一个虚拟路由器的keepalived节点必须一样
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
}
[root@ka1 ~]#systemctl restart keepalived.service
5-1-2、BACKUP配置
#配置文件和master基本一致，只需修改三行
[root@ka2 ~]#vim /etc/keepalived/keepalived.conf
global_defs {
router_id ka2.wang.org #id不一样
}
include /etc/keepalived/conf.d/*.conf
[root@ka2 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance ka2.wang.org {
stats BACKUP #此处为backup
interface eth0
virtual_router_id 66
priority 80 #优先级低于master
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
}
5-1-3、脑裂
主备节点同时拥有VIP，此时为脑裂理象
注意：脑裂现象原因
- 心跳线故障
- 防火墙错误配置
- Keepalived 配置错误
5-2、 抢占模式和非抢占模式
5-2-1、非抢占模式 nopreempt
默认为抢占模式 preempt，即当高优先级的主机恢复在线后，会抢占低先级的主机的master角色，造成网络抖动，建议设置为非抢占模式 nopreempt ，即高优先级主机恢复后，并不会抢占低优先级主机的master 角色 注意: 非抢占模式下,如果原主机down机, VIP迁移至的新主机, 后续新主机也发生down时,VIP还会迁移回修复好的原主机 但如果新主机的服务down掉（keepalived服务正常），原主机也不会接管VIP，仍会由新主机拥有VIP即非抢占式模式，只是适合当主节点宕机，切换到从节点的一次性的高可用性，后续即使当原主节点修复好，仍无法再次起到高可用
注意：要关闭 VIP抢占，必须将各 Keepalived 服务器 state 配置为 BACKUP
#ka1:
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance ka1.wang.org {
state BACKUP #都为BACKUP
interface eth0
virtual_router_id 66
priority 100 #优先级高
nopreempt #添加此行，设为nopreempt
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
}
[root@ka1 ~]#systemctl restart keepalived.service
#ka2:
[root@ka2 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance ka2.wang.org {
stats BACKUP #都为BACKUP
interface eth0
virtual_router_id 66
priority 80 #优先级低
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
}
#ka2，不需要配置nopreempt
#注意：如果ka2主机也是非抢占式，会导致ka1即使优先级降低于ka2，VIP也不会切换至ka2
5-3、抢占延迟模式 preempt_delay
抢占延迟模式，即优先级高的主机恢复后，不会立即抢回VIP，而是延迟一段时间（默认300s）再抢回VIP
preempt_delay # #指定抢占延迟时间为#s，默认延迟300s[root@ka1 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance ka1.wang.org {
state BACKUP
interface eth0
virtual_router_id 66
priority 100
# nopreempt
preempt_delay 60 #抢占延迟模式，默认延迟300s
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
}
5-4、VIP 单播配置
默认keepalived主机之间利用多播相互通告消息，会造成网络拥塞，可以替换成单播，减少网络流量
另外：有些公有云不支持多播，可以单播实现
注意：启用 vrrp_strict 时，不能启用单播
#在所有节点vrrp_instance语句块中设置对方主机的IP，建议设置为专用于对应心跳线网络的地址，而非使用业务网络
unicast_src_ip <IPADDR> #指定发送单播的源IP
unicast_peer {
<IPADDR> #指定接收单播的对方目标主机IP
......
}
#master
[root@ka1 ~]#vim /etc/keepalived/keepalived.conf
global_defs {
router_id ka1.wang.org
vrrp_mcast_group4 239.0.0.0 #单播优先于多播，多播配不配均可，如果不配，默认是224.0.0.18
}
include /etc/keepalived/conf.d/*.conf
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance ka1.wang.org {
state MASTER
interface eth0
virtual_router_id 66
priority 100
# nopreempt
preempt_delay 60
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.101 #本机IP
unicast_peer { #指向对方主机IP
10.0.0.102 #如果有多个keepalived,再加其它节点的IP
# 10.0.0.103
}
}
#backup
[root@ka2 ~]#vim /etc/keepalived/keepalived.conf
global_defs {
router_id ka2.wang.org
vrrp_mcast_group4 239.0.0.0
}
include /etc/keepalived/conf.d/*.conf
[root@ka2 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance ka2.wang.org {
stats BACKUP
interface eth0
virtual_router_id 66
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.102
unicast_peer {
10.0.0.101
}
}
5-5、 Keepalived 通知脚本配置
当keepalived的状态变化时，可以自动触发脚本的执行，比如：发邮件通知用户 默认以用户keepalived_script身份执行脚本，如果此用户不存在，以root执行脚本
可以用下面指令指定脚本执行用户的身份
global_defs {
......
script_user <USER>
......
}5-5-1、通知脚本类型
#当前节点成为主节点时触发的脚本
notify_master <STRING>|<QUOTED-STRING>
#当前节点转为备节点时触发的脚本
notify_backup <STRING>|<QUOTED-STRING>
#当前节点转为“失败”状态时触发的脚本
notify_fault <STRING>|<QUOTED-STRING>
#通用格式的通知触发机制，一个脚本可完成以上三种状态的转换时的通知
notify <STRING>|<QUOTED-STRING>
#当停止VRRP时触发的脚本
notify_stop <STRING>|<QUOTED-STRING>
5-5-2、脚本的调用方法
#在 vrrp_instance VI_1 语句块的末尾加下面行
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
[root@ka1 keepalived]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance ka1.wang.org {
state MASTER
interface eth0
virtual_router_id 66
priority 100
# nopreempt
preempt_delay 60
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.101
unicast_peer {
10.0.0.102
}
notify_master "/etc/keepalived/notify.sh master" #脚本路径
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
[root@ka1 keepalived]#chmod +x notify.sh
contact='965509771@qq.com' #qq邮箱有问题，建议用163邮箱
email_send='965509771@qq.com'
email_passwd='xenddddmynztddzbceb'
email_smtp_server='smtp.qq.com'
notify() {
if [[ $1 =~ ^(master|backup|fault)$ ]];then
mailsubject="$(hostname) to be $1, vip floating"
mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
send_email "$contact" "$mailsubject" "$mailbody"
else
echo "Usage: $(basename $0) {master|backup|fault}"
exit 1
fi
}
notify $1
#模拟master故障
[root@ka1 keepalived]#killall keepalived
#邮箱收到邮件：
2022-10-26 21:41:33: vrrp transition, ka2.wang.org changed to be master
5-6、实现 Master/Master 的 Keepalived 双主架构
master/slave的单主架构，同一时间只有一个Keepalived对外提供服务，此主机繁忙，而另一台主机却很空闲，利用率低下，可以使用master/master的双主架构，解决此问题。 Master/Master 的双主架构： 即将两个或以上VIP分别运行在不同的keepalived服务器，以实现服务器并行提供web访问的目的，提高服务器资源利用率
#ka1：
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 66
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.101
unicast_peer {
10.0.0.102
}
}
vrrp_instance VI_2 { #添加 VI_2 实例
state BACKUP
interface eth0
virtual_router_id 88
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.88/24 dev eth0 label eth0:2
}
unicast_src_ip 10.0.0.101
unicast_peer {
10.0.0.102
}
}
[root@ka1 ~]#systemctl restart keepalived.service
[root@ka1 ~]#hostname -I
10.0.0.101 10.0.0.66
#ka2:
[root@ka2 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance VI_1 {
stats BACKUP
interface eth0
virtual_router_id 66
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.102
unicast_peer {
10.0.0.101
}
}
vrrp_instance VI_2 {
state MASTER
interface eth0
virtual_router_id 88
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.88/24 dev eth0 label eth0:2
}
unicast_src_ip 10.0.0.102
unicast_peer {
10.0.0.101
}
}
[root@ka2 ~]#systemctl restart keepalived.service
[root@ka2 ~]#hostname -I
10.0.0.102 10.0.0.88
5-7、三个节点的三主三从架构实现
#ka1：
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 66
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.101
unicast_peer {
10.0.0.102
}
}
vrrp_instance VI_2 {
state BACKUP
interface eth0
virtual_router_id 99
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.99/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.101
unicast_peer {
10.0.0.103
}
}
[root@ka1 ~]#systemctl restart keepalived.service
#ka2:
[root@ka2 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance VI_1 {
stats MASTER
interface eth0
virtual_router_id 88
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.88/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.102
unicast_peer {
10.0.0.103
}
}
vrrp_instance VI_2 {
state BACKUP
interface eth0
virtual_router_id 66
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.102
unicast_peer {
10.0.0.101
}
}
[root@ka2 ~]#systemctl restart keepalived.service
#ka3:
[root@ka3 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance VI_1 {
stats MASTER
interface eth0
virtual_router_id 99
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.99/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.103
unicast_peer {
10.0.0.101
}
}
vrrp_instance VI_2 {
stats BACKUP
interface eth0
virtual_router_id 88
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.88/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.103
unicast_peer {
10.0.0.102
}
}
[root@ka3 ~]#systemctl restart keepalived.service
#测试
[root@ka1 ~]#systemctl stop keepalived.service
[root@ka2 ~]#hostname -I
10.0.0.102 10.0.0.88 10.0.0.66
[root@ka1 ~]#systemctl start keepalived.service
[root@ka1 ~]#hostname -I
10.0.0.101 10.0.0.66
[root@ka2 ~]#systemctl stop keepalived.service
[root@ka3 ~]#hostname -I
10.0.0.103 10.0.0.99 10.0.0.88
[root@ka3 ~]#systemctl stop keepalived.service
[root@ka1 ~]#hostname -I
10.0.0.101 10.0.0.66 10.0.0.99
5-8、三个节点的三主六从架构实现
#ka1:
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 66
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.101
unicast_peer {
10.0.0.102
10.0.0.103
}
}
vrrp_instance VI_2 {
state BACKUP
interface eth0
virtual_router_id 88
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.88/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.101
unicast_peer {
10.0.0.102
10.0.0.103
}
}
vrrp_instance VI_3 {
state BACKUP
interface eth0
virtual_router_id 99
priority 60
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.99/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.101
unicast_peer {
10.0.0.102
10.0.0.103
}
}
[root@ka1 ~]#systemctl restart keepalived.service
[root@ka1 ~]#hostname -I
10.0.0.101 10.0.0.66
#ka2:
[root@ka2 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance VI_1 {
stats MASTER
interface eth0
virtual_router_id 88
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.88/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.102
unicast_peer {
10.0.0.101
10.0.0.103
}
}
vrrp_instance VI_2 {
state BACKUP
interface eth0
virtual_router_id 99
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.99/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.102
unicast_peer {
10.0.0.101
10.0.0.103
}
}
vrrp_instance VI_3 {
state BACKUP
interface eth0
virtual_router_id 66
priority 60
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.102
unicast_peer {
10.0.0.101
10.0.0.103
}
}
[root@ka2 ~]#systemctl restart keepalived.service
[root@ka2 ~]#hostname -I
10.0.0.102 10.0.0.88
#ka3:
[root@ka3 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance VI_1 {
stats MASTER
interface eth0
virtual_router_id 99
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.99/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.103
unicast_peer {
10.0.0.101
10.0.0.102
}
}
vrrp_instance VI_2 {
stats BACKUP
interface eth0
virtual_router_id 66
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.103
unicast_peer {
10.0.0.102
10.0.0.101
}
}
vrrp_instance VI_3 {
stats BACKUP
interface eth0
virtual_router_id 88
priority 60
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.88/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.103
unicast_peer {
10.0.0.102
10.0.0.101
}
}
[root@ka3 ~]#systemctl restart keepalived.service
[root@ka3 ~]#hostname -I
10.0.0.103 10.0.0.99
#测试：
[root@ka1 ~]#systemctl stop keepalived.service
[root@ka3 ~]#hostname -I
10.0.0.103 10.0.0.99 10.0.0.66
[root@ka2 ~]#systemctl stop keepalived.service
[root@ka3 ~]#hostname -I
10.0.0.103 10.0.0.99 10.0.0.66 10.0.0.88
5-9、同步组
LVS NAT 模型VIP和DIP需要同步，需要同步组
vrrp_sync_group VG_1 {
group {
VI_1 # name of vrrp_instance (below)
VI_2 # One for each moveable IP
}
}
vrrp_instance VI_1 {
eth0
vip
}
vrrp_instance VI_2 {
eth1
dip
}6、实现 IPVS 的高可用性
6-1、虚拟服务器配置结构
每一个虚拟服务器即一个IPVS集群 可以通过下面语法实现
virtual_server IP port {
...
real_server {
...
}
real_server {
...
}
...
}6-2、Virtual Server （虚拟服务器）的定义格式
virtual_server IP port #定义虚拟主机IP地址及其端口
virtual_server fwmark int #ipvs的防火墙打标，实现基于防火墙的负载均衡集群
virtual_server group string #使用虚拟服务器组
6-3、虚拟服务器组
将多个虚拟服务器定义成一个组，统一对外服务，如：http和https定义成一个虚拟服务器组
#参考文档：/usr/share/doc/keepalived/keepalived.conf.virtual_server_group
virtual_server_group <STRING> {
# Virtual IP Address and Port
<IPADDR> <PORT>
<IPADDR> <PORT>
...
# <IPADDR RANGE> has the form
# XXX.YYY.ZZZ.WWW-VVV eg 192.168.200.1-10
# range includes both .1 and .10 address
<IPADDR RANGE> <PORT># VIP range VPORT
<IPADDR RANGE> <PORT>
...
# Firewall Mark (fwmark)
fwmark <INTEGER>
fwmark <INTEGER>
...
}
6-4、虚拟服务器配置
virtual_server IP port { #VIP和PORT
delay_loop <INT> #检查后端服务器的时间间隔
lb_algo rr|wrr|lc|wlc|lblc|sh|dh #定义调度方法
lb_kind NAT|DR|TUN #集群的类型,注意要大写
persistence_timeout <INT> #持久连接时长
protocol TCP|UDP|SCTP #指定服务协议,一般为TCP
sorry_server <IPADDR> <PORT> #所有RS故障时，备用服务器地址
real_server <IPADDR> <PORT> { #RS的IP和PORT
weight <INT> #RS权重
notify_up <STRING>|<QUOTED-STRING> #RS上线通知脚本
notify_down <STRING>|<QUOTED-STRING> #RS下线通知脚本
HTTP_GET|SSL_GET|TCP_CHECK|SMTP_CHECK|MISC_CHECK { ... } #定义当前主机健康状态检测方法
}
}
#注意:括号必须分行写,两个括号写在同一行,如: }} 会出错6-5、应用层监测
应用层检测：HTTP_GET|SSL_GET
HTTP_GET|SSL_GET {
url {
path <URL_PATH> #定义要监控的URL
status_code <INT> #判断上述检测机制为健康状态的响应码，一般为 200
}
connect_timeout <INTEGER> #客户端请求的超时时长, 相当于haproxy的timeout server
nb_get_retry <INT> #重试次数
delay_before_retry <INT> #重试之前的延迟时长
connect_ip <IP ADDRESS> #向当前RS哪个IP地址发起健康状态检测请求
connect_port <PORT> #向当前RS的哪个PORT发起健康状态检测请求
bindto <IP ADDRESS> #向当前RS发出健康状态检测请求时使用的源地址
bind_port <PORT> #向当前RS发出健康状态检测请求时使用的源端口
}virtual_server 10.0.0.66 80 {
lb_algo rr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80
real_server 10.0.0.28 80 {
weight 1
HTTP_GET {
url {
path /monitor.html
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 10.0.0.38 80 {
weight 1
HTTP_GET {
uri {
path /monitor.html
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
}
6-6、TCP监测
传输层检测：TCP_CHECK
TCP_CHECK {
connect_ip <IP ADDRESS> #向当前RS的哪个IP地址发起健康状态检测请求
connect_port <PORT> #向当前RS的哪个PORT发起健康状态检测请求
bindto <IP ADDRESS> #发出健康状态检测请求时使用的源地址
bind_port <PORT> #发出健康状态检测请求时使用的源端口
connect_timeout <INTEGER> #客户端请求的超时时长, 等于haproxy的timeout server
}virtual_server 10.0.0.66 80 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
# persistence_timeout 1230
sorry_server 127.0.0.1 80
real_server 10.0.0.28 80 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 10.0.0.38 80 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
6-7、实现单主的 LVS-DR 模式
6-7-1、准备web服务器并使用脚本绑定VIP至web服务器lo网卡
#两台web服务器执行：
[root@rocky8 ~]#cat lvs_dr_rs.sh
#!/bin/bash
vip=10.0.0.66
mask='255.255.255.255'
dev=lo:1
#rpm -q httpd &> /dev/null || yum -y install httpd &>/dev/null
#service httpd start &> /dev/null && echo "The httpd Server is Ready!"
#echo "`hostname -I`" > /var/www/html/index.html
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig $dev $vip netmask $mask #broadcast $vip up
#route add -host $vip dev $dev
echo "The RS Server is Ready!"
;;
stop)
ifconfig $dev down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "The RS Server is Canceled!"
;;
*)
echo "Usage: $(basename $0) start|stop"
exit 1
;;
esac
[root@rocky8 ~]#bash lvs_dr_rs.sh start
#ka1:
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 66
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.101
unicast_peer {
10.0.0.102
}
}
virtual_server 10.0.0.66 80 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
# persistence_timeout 1230
sorry_server 127.0.0.1 80
real_server 10.0.0.28 80 {
weight 1
HTTP_GET {
url {
path /monitor.html
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 10.0.0.38 80 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
#ka2:
[root@ka2 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 66
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.102
unicast_peer {
10.0.0.101
}
}
virtual_server 10.0.0.66 80 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
# persistence_timeout 1230
sorry_server 127.0.0.1 80
real_server 10.0.0.28 80 {
weight 1
HTTP_GET {
url {
path /monitor.html
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 10.0.0.38 80 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
6-7-2、实现双主的 LVS-DR 模式
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 66
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.101
unicast_peer {
10.0.0.102
}
}
vrrp_instance VI_2 {
state BACKUP
interface eth0
virtual_router_id 88
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.88/24 dev eth0 label eth0:2
}
unicast_src_ip 10.0.0.101
unicast_peer {
10.0.0.102
}
}
virtual_server 10.0.0.66 80 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
# persistence_timeout 1230
sorry_server 127.0.0.1 80
real_server 10.0.0.28 80 {
weight 1
HTTP_GET {
url {
path /monitor.html
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 10.0.0.38 80 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
virtual_server 10.0.0.88 80 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
# persistence_timeout 1230
sorry_server 127.0.0.1 80
real_server 10.0.0.48 80 {
weight 1
HTTP_GET {
url {
path /monitor.html
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 10.0.0.58 80 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
#ka2:
[root@ka2 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 66
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:1
}
unicast_src_ip 10.0.0.102
unicast_peer {
10.0.0.101
}
}
vrrp_instance VI_2{
state BACKUP
interface eth0
virtual_router_id 88
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.88/24 dev eth0 label eth0:2
}
unicast_src_ip 10.0.0.102
unicast_peer {
10.0.0.101
}
}
virtual_server 10.0.0.66 80 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
# persistence_timeout 1230
sorry_server 127.0.0.1 80
real_server 10.0.0.28 80 {
weight 1
HTTP_GET {
url {
path /monitor.html
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 10.0.0.38 80 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
virtual_server 10.0.0.88 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
# persistence_timeout 1230
sorry_server 127.0.0.1 80
real_server 10.0.0.48 80 {
weight 1
HTTP_GET {
url {
path /monitor.html
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 10.0.0.58 80 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
6-8、双主分别实现httpd和mysql服务的调度
#keepalived1:
[root@ka1 ~]#vim /etc/keepalived/conf.d/web.conf
vrrp_instance web1 {
state MASTER
interface eth0
virtual_router_id 66
priority 100
advert_int 1
authentication {
auth_TYPE pass
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:66
}
unicast_src_ip 10.0.0.101
unicast_peer {
10.0.0.102
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
[root@ka1 ~]#vim /etc/keepalived/conf.d/lvs_web.conf
virtual_server 10.0.0.66 80 {
delay_loop 3
lb_algo rr
lb_kind DR
protocol TCP
sorry server 10.0.0.28 80
real_server 10.0.0.48 80 {
weight 1
HTTP_GET {
url {
path /monitor.html
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 10.0.0.58 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
=============================================================================
[root@ka1 ~]#vim /etc/keepalived/conf.d/mysql.conf
vrrp_instance mysql {
state BACKUP
interface eth0
virtual_router_id 88
priority 80
advert_int 1
authentication {
auth_TYPE pass
auth_PASS 123456
}
virtual_ipaddress {
10.0.0.88/24 dev eth0 label eth0:2
}
unicast_src_ip 10.0.0.101
unicast_peer {
10.0.0.102
}
}
[root@ka1 ~]#vim /etc/keepalived/conf.d/lvs_mysql.conf
virtual_server 10.0.0.88 3306 {
delay_loop 3
lb_algo rr
lb_kind DR
protocol TCP
real_server 10.0.0.48 3306 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
real_server 10.0.0.58 3306 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
}
[root@ka1 ~]#systemctl restart keepalived.service
[root@ka1 ~]#hostname -I
10.0.0.101 10.0.0.66
#keepalived2：
[root@ka2 ~]#vim /etc/keepalived/conf.d/web.conf
vrrp_instance web1 {
state BACKUP
interface eth0
virtual_router_id 66
priority 80
advert_int 1
authentication {
auth_TYPE pass
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:66
}
unicast_src_ip 10.0.0.102
unicast_peer {
10.0.0.101
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
[root@ka2 ~]#vim /etc/keepalived/conf.d/lvs_web.conf
virtual_server 10.0.0.66 80 {
delay_loop 3
lb_algo rr
lb_kind DR
protocol TCP
sorry server 10.0.0.28 80
real_server 10.0.0.48 80 {
weight 1
HTTP_GET {
url {
path /monitor.html
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 10.0.0.58 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
=============================================================================
[root@ka2 ~]#vim /etc/keepalived/conf.d/mysql.conf
vrrp_instance mysql {
state MASTER
interface eth0
virtual_router_id 88
priority 100
advert_int 1
authentication {
auth_TYPE pass
auth_PASS 123456
}
virtual_ipaddress {
10.0.0.88/24 dev eth0 label eth0:2
}
unicast_src_ip 10.0.0.102
unicast_peer {
10.0.0.101
}
}
[root@ka2 ~]#vim /etc/keepalived/conf.d/lvs_mysql.conf
virtual_server 10.0.0.88 3306 {
delay_loop 3
lb_algo rr
lb_kind DR
protocol TCP
real_server 10.0.0.48 3306 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
real_server 10.0.0.58 3306 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 3306
}
}
}
[root@ka2 ~]#systemctl restart keepalived.service
[root@ka2 ~]#hostname -I
10.0.0.102 10.0.0.88
#测试：
[root@ka3 ~]#while true; do mysql -utest -p123456 -h10.0.0.88 -e 'select @@hostname';curl 10.0.0.66;sleep 1;done
mysql: [Warning] Using a password on the command line interface can be insecure.
+------------------+
| @@hostname |
+------------------+
| mysql48.wang.org |
+------------------+
10.0.0.48
mysql: [Warning] Using a password on the command line interface can be insecure.
+------------------+
| @@hostname |
+------------------+
| mysql58.wang.org |
+------------------+
10.0.0.58
mysql: [Warning] Using a password on the command line interface can be insecure.
+------------------+
| @@hostname |
+------------------+
| mysql48.wang.org |
+------------------+
10.0.0.48
mysql: [Warning] Using a password on the command line interface can be insecure.
+------------------+
| @@hostname |
+------------------+
| mysql58.wang.org |
+------------------+
10.0.0.58
6-9、实现单主的 LVS-DR 模式，利用FWM绑定成多个服务为一个集群服务
#两个keepalived节点都执行（10.0.0.101、10.0.0.102）:
[root@ka1 ~]#iptables -t mangle -A PREROUTING -d 10.0.0.66 -p tcp -m multiport --dports 80,443 -j MARK --set-mark 6
[root@ka1 ~]#vim /etc/keepalived/conf.d/web.conf
vrrp_instance web1 {
state MASTER #在另一个节点为BACKUP
interface eth0
virtual_router_id 66
priority 100 #在另一个节点为80
advert_int 1
authentication {
auth_TYPE pass
auth_pass 123456
}
virtual_ipaddress {
10.0.0.66/24 dev eth0 label eth0:66
}
unicast_src_ip 10.0.0.101 #另一个节点源地址是102
unicast_peer {
10.0.0.102 #另一个节点是101
}
track_interface {
eth0
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
[root@ka1 ~]#vim /etc/keepalived/conf.d/fwmark_web.conf
virtual_server fwmark 6 { #fwmark 6 和iptables定义的名称一样
delay_loop 3
lb_algo rr
lb_kind DR
protocol TCP
sorry server 127.0.0.1 80
real_server 10.0.0.48 80 { #注意端口必须指定,只用于健康性检查,而非通信
weight 1
HTTP_GET {
url {
path /monitor.html
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 10.0.0.58 80 {
weight 1
HTTP_GET {
url {
path /monitor.html
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
[root@ka1 ~]#systemctl restart keepalived.service
#web服务器（10.0.0.48、10.0.0.58）：
[root@mysql48 ~]# bash lvs_dr_rs.sh start
#制作ssl证书，并且开启nginx443端口
[root@mysql48 ~]# ls /etc/nginx/conf.d/certs/
ca.crt ca.srl www.wang.org.crt www.wang.org.key
ca.key certs.sh www.wang.org.csr www.wang.org.pem
#测试访问：
[root@ka3 ~]#curl -k https://10.0.0.66
10.0.0.48
[root@ka3 ~]#curl -k https://10.0.0.66
10.0.0.58
[root@ka3 ~]#curl 10.0.0.66
10.0.0.48
[root@ka3 ~]#curl 10.0.0.66
10.0.0.58
7、基于 VRRP Script 实现其它应用的高可用性
keepalived利用 VRRP Script 技术，可以调用外部的辅助脚本进行资源监控，并根据监控的结果实现优先动态调整，从而实现其它应用的高可用性功能
参考配置文件：
/usr/share/doc/keepalived/keepalived.conf.vrrp.localcheck
7-1、 VRRP Script 配置
分两步实现：
- 定义脚本 vrrp_script：自定义资源监控脚本，vrrp实例根据脚本返回值，公共定义，可被多个实例调用，定义在vrrp实例之外的独立配置块，一般放在global_defs设置块之后,是和global_defs平级的语句块 通常此脚本用于监控指定应用的状态。一旦发现应用的状态异常，则触发对MASTER节点的权重减至低于SLAVE节点，从而实现 VIP 切换到 SLAVE 节点 当 keepalived_script 用户存在时,会以此用户身份运行脚本,否则默认以root运行脚本 注意: 此定义脚本的语句块一定要放在下面调用此语句vrrp_instance语句块的前面
vrrp_script <SCRIPT_NAME> {
script <STRING>|<QUOTED-STRING> #此脚本返回值为非0时，会触发下面OPTIONS执行
OPTIONS
}- 调用脚本 track_script：调用vrrp_script定义的脚本去监控资源，定义在VRRP实例之内，调用事先定义的vrrp_script
track_script {
SCRIPT_NAME_1
SCRIPT_NAME_2
}7-1-1、定义 VRRP script
vrrp_script <SCRIPT_NAME> { #定义一个检测脚本，在global_defs 之外配置
script <STRING>|<QUOTED-STRING> #shell命令或脚本路径
interval <INTEGER> #间隔时间，单位为秒，默认1秒
timeout <INTEGER> #超时时间
weight <INTEGER:-254..254> #默认为0,如果设置此值为负数，当上面脚本返回值为非0时，会将此值与本节点权重相加可以降低本节点权重，即表示fall. 如果是正数，当脚本返回值为0，会将此值与本节点权重相加可以提高本节点权重，即表示 rise.通常使用负值
fall <INTEGER> #执行脚本连续几次都失败,则转换为失败，建议设为2以上
rise <INTEGER> #执行脚本连续几次都成功，把服务器从失败标记为成功
user USERNAME [GROUPNAME] #执行监测脚本的用户或组
init_fail #设置默认标记为失败状态，监测成功之后再转换为成功状态
}7-1-2、调用 VRRP script
vrrp_instance VI_1 {
...
track_script {
<SCRIPT_NAME>
}
}7-2、利用脚本实现主从角色切换
[root@ka1 ~]#vim /etc/keepalived/keepalived.conf
global_defs {
router_id ka1.wang.org
}
vrrp_script check_down {
script "[ ! -f /etc/keepalived/down ] " #/etc/keepalived/down存在时返回非0，触发权重-30
interval 1
weight -30
fall 3
rise 2
timeout 2
}
include /etc/keepalived/conf.d/*.conf
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 33
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.33/24 dev eth0 label eth0:1
}
track_interface {
eth0
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
track_script {
check_down #调用前面定义的脚本
}
}
#测试
[root@ka1 ~]#touch /etc/keepalived/down
[root@ka3 ~]#tcpdump -i eth0 -nn host 224.0.0.18
6:04:07.101894 IP 10.0.0.101 > 224.0.0.18: VRRPv2, Advertisement, vrid 33, prio 70, authtype simple, intvl 1s, length 20
16:04:08.103307 IP 10.0.0.101 > 224.0.0.18: VRRPv2, Advertisement, vrid 33, prio 70, authtype simple, intvl 1s, length 20
16:04:08.788557 IP 10.0.0.102 > 224.0.0.18: VRRPv2, Advertisement, vrid 33, prio 80, authtype simple, intvl 1s, length 20
16:04:09.790807 IP 10.0.0.102 > 224.0.0.18: VRRPv2, Advertisement, vrid 33, prio 80, authtype simple, intvl 1s, length 20
16:04:10.792118 IP 10.0.0.102 > 224.0.0.18: VRRPv2, Advertisement, vrid 33, prio 80, authtype simple,
[root@ka1 ~]#rm -f /etc/keepalived/down
[root@ka3 ~]#tcpdump -i eth0 -nn host 224.0.0.18
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:05:38.489491 IP 10.0.0.101 > 224.0.0.18: VRRPv2, Advertisement, vrid 33, prio 100, authtype simple, intvl 1s, length 20
16:05:39.490135 IP 10.0.0.101 > 224.0.0.18: VRRPv2, Advertisement, vrid 33, prio 100, authtype simple, intvl 1s, length 20
7-3、实现单主模式的 Nginx 反向代理的高可用
#在两个节点都配置nginx反向代理
[root@ka1 ~]#vim /etc/nginx/conf.d/www.li.org.conf
upstream li {
server 10.0.0.28:80 weight=1;
server 10.0.0.38:80 weight=1;
}
server {
listen 80;
server_name www.li.org;
location / {
proxy_pass http://li;
}
}
#keepalived节点都执行
[root@ka1 ~]#vim /etc/keepalived/keepalived.conf
global_defs {
router_id ka1.wang.org
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 1
weight -30
fall 3
rise 2
timeout 2
}
include /etc/keepalived/conf.d/*.conf
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.li.org.conf
vrrp_instance VI_1 {
state MASTER #在另一个节点为BACKUP
interface eth0
virtual_router_id 44
priority 100 #在另一个节点为80
advert_int 1
authentication {
auth_TYPE pass
auth_pass 123456
}
virtual_ipaddress {
10.0.0.44/24 dev eth0 label eth0:4
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
track_script {
check_nginx
}
unicast_src_ip 10.0.0.101 #另一节点102
unicast_peer {
10.0.0.102 #另一节点101
}
}
[root@ka1 ~]#vim /etc/keepalived/check_nginx.sh
/usr/bin/killall -0 nginx || systemctl restart nginx #自动重启服务
[root@ka1 ~]#chmod +x /etc/keepalived/check_nginx.sh
#测试
[root@ka1 keepalived]#systemctl stop nginx
[root@ka1 keepalived]#systemctl status nginx #自动重启服务
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2022-10-27 16:43:23 CST; 26s ago
7-4、实现双主模式 Nginx 反向代理的高可用
#在两个节点都配置nginx反向代理
[root@ka1 ~]#vim /etc/nginx/conf.d/www.li.org.conf
upstream li {
server 10.0.0.28:80 weight=1;
server 10.0.0.38:80 weight=1;
}
upstream wang {
server 10.0.0.48:80 weight=1;
server 10.0.0.58:80 weight=1;
}
server {
listen 80;
server_name www.li.org;
location / {
proxy_pass http://li;
}
}
server {
listen 80;
server_name www.wang.org;
location / {
proxy_pass http://wang;
}
}
# keepalived所有节点执行：
[root@ka1 ~]#vim /etc/keepalived/keepalived.conf
global_defs {
router_id ka1.wang.org
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 1
weight -30
fall 3
rise 2
timeout 2
}
include /etc/keepalived/conf.d/*.conf
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.li.org.conf
vrrp_instance VI_1 {
state MASTER #在另一个节点为BACKUP
interface eth0
virtual_router_id 44
priority 100 #在另一个节点为80
advert_int 1
authentication {
auth_TYPE pass
auth_pass 123456
}
virtual_ipaddress {
10.0.0.44/24 dev eth0 label eth0:4
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
track_script {
check_nginx
}
unicast_src_ip 10.0.0.101 #在另一个节点为102
unicast_peer {
10.0.0.102 #在另一个节点为101
}
}
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance VI_2 {
state BACKUP #在另一个节点为MASTER
interface eth0
virtual_router_id 33
priority 80 #在另一个节点为100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.33/24 dev eth0 label eth0:1
}
track_interface {
eth0
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
track_script {
check_down
}
unicast_src_ip 10.0.0.101 #在另一个节点为102
unicast_peer {
10.0.0.102 #在另一个节点为101
}
track_script {
check_nginx
}
}
[root@ka1 ~]#systemctl restart keepalived.service
#测试：
[root@ka1 ~]#systemctl stop nginx
[root@ka1 ~]#systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2022-10-27 16:59:53 CST; 4s ago
[root@ka3 ~]#curl www.wang.org
10.0.0.48
[root@ka3 ~]#curl www.wang.org
10.0.0.58
[root@ka3 ~]#curl www.li.org
welcome to 8.222222
[root@ka3 ~]#curl www.li.org
welcome to 8.333333333
7-5、实现 HAProxy 高可用
#两个节点都配置haprxoy（10.0.0.101、10.0.0.102）：
[root@ka1 ~]#vim /etc/haproxy/haproxy.cfg
listen wang_http
bind 10.0.0.33:80
server web1 10.0.0.48:80 check
server web2 10.0.0.58:80 check
listen li_http
bind 10.0.0.44:80
server web3 10.0.0.28:80 check
server web4 10.0.0.38:80 check
listen stats
mode http
bind 10.0.0.101:9999
stats enable
log global
stats uri /haproxy-status
stats auth admin:123456
#在两个ka1和ka2两个节点启用内核参数
[root@ka1 ~]#vim /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind = 1
[root@ka1 ~]#vim /etc/keepalived/keepalived.conf
global_defs {
router_id ka1.wang.org
}
vrrp_script check_haproxy {
script "/etc/keepalived/check_haproxy.sh"
interval 1
weight -30
fall 3
rise 2
timeout 2
}
include /etc/keepalived/conf.d/*.conf
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.li.org.conf
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 44
priority 100
advert_int 1
authentication {
auth_TYPE pass
auth_pass 123456
}
virtual_ipaddress {
10.0.0.44/24 dev eth0 label eth0:4
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
track_script {
check_haproxy
}
unicast_src_ip 10.0.0.101
unicast_peer {
10.0.0.102
}
}
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.wang.org.conf
vrrp_instance VI_2 {
state BACKUP
interface eth0
virtual_router_id 33
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.33/24 dev eth0 label eth0:1
}
track_interface {
eth0
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
unicast_src_ip 10.0.0.101
unicast_peer {
10.0.0.102
}
track_script {
check_haproxy
}
}
7-6、实现 MySQL 双主模式的高可用
[root@ka1 ~]#vim /etc/keepalived/keepalived.conf
global_defs {
router_id ka1.wang.org
}
vrrp_script check_mysql {
script "/etc/keepalived/check_mysql.sh" #只需在第一个节点上实现脚本
interval 1
weight -30
fall 3
rise 2
timeout 2
}
include /etc/keepalived/conf.d/*.conf
[root@ka1 ~]#vim /etc/keepalived/conf.d/mysql.conf
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 55
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.6/24 dev eth0 label eth0:6
}
track_interface {
eth0
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
track_script {
check_mysql #只需在第一个节点上实现脚本
}
}
[root@ka1 ~]#vim /etc/keepalived/check_mysql.sh
slave_is=( $(mysql -uadmin -p123456 -h10.0.0.102 -e "show slave status\G" | grep "Slave_.*_Running:" | awk 'print $2') )
if [ "${slave_is[0]}" = "Yes" -a "${slave_is[1]}" = "Yes" ];then
exit 0
else
exit 1
fi
7-7、实现 Zabbix Server 的高可用
#keepalived所有节点执行（10.0.0.101、10.0.0.102）：
[root@ka1 ~]#bash install_zabbix_server5.0.sh #脚本详见：https://blog.51cto.com/dayu/5801993
#Zabbix Server 使用下面相同的配置
[root@ka1 ~]#vim /etc/zabbix/zabbix_server.conf
SourceIP=10.0.0.77
[root@ka1 ~]#systemctl restart zabbix-server.service
#keepalived相关配置
#指定检测脚本
[root@ka1 ~]#vim /etc/keepalived/keepalived.conf
global_defs {
router_id ka1.wang.org
}
vrrp_script check_zabbix_server {
script "/usr/bin/killall -0 zabbix-server"
interval 1
weight -30
fall 3
rise 2
timeout 2
}
include /etc/keepalived/conf.d/*.conf
[root@ka1 ~]#vim /etc/keepalived/conf.d/zabbix.conf
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 66
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.77/24 dev eth0 label eth0:7
}
notify_master "systemctl start zabbix-server"
notify_backup "systemctl stop zabbix-server"
track_script {
check_zabbix_server
}
}
[root@ka1 ~]#systemctl restart keepalived.service
#keepalived2:(10.0.0.102)
[root@ka2 ~]#vim /etc/keepalived/conf.d/zabbix.conf
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 66
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.77/24 dev eth0 label eth0:7
}
notify_master "systemctl start zabbix-server"
notify_backup "systemctl stop zabbix-server"
}
#注意：脚本check_zabbix_server 在ka2节点不能启用,否则会导致ka2节点也降低优先级,从而切换失败
[root@ka2 ~]#systemctl restart keepalived.service
总结：
1、实例名称不能一样；
2、state 后边的master，backup不能一样
3、同一虚拟路由id要一样
4、优先级不能一样
5、尽量不要用组播，如果有多个实例一定要用单播，要不然会有不轮询或者vip飘不过去的问题
