VRRP:Virtual Router Redundancy Protocol

虚拟路由冗余协议,解决静态网关单点风险

  • 物理层:路由器、三层交换机
  • 软件层:keepalived

VRRP 网络层硬件实现

https://support.huawei.com/enterprise/zh/doc/EDOC1000141382/19258d72/basicconcepts-of-vrrp

54-Keepalive-基于VRRP技术的集群VIP漂移以及各种生产常见案例_VRRP

VRRP 相关术语

  • 虚拟路由器:Virtual Router
  • 虚拟路由器标识:VRID(0-255),唯一标识虚拟路由器
  • VIP:Virtual IP
  • VMAC:Virutal MAC (00-00-5e-00-01-VRID)
  • 物理路由器:
  • master:主设备
  • backup:备用设备
  • priority:优先级

VRRP 相关技术

  • 通告:心跳,优先级等;周期性
  • 工作方式:抢占式,非抢占式
  • 安全认证:
  • 无认证
  • 简单字符认证:预共享密钥
  • MD5
  • 工作模式:
  • 主/备:单虚拟路由器
  • 主/主:主/备(虚拟路由器1),备/主(虚拟路由器2)

Keepalived 架构

54-Keepalive-基于VRRP技术的集群VIP漂移以及各种生产常见案例_通用高可用解决方案_02

  • vrrp 协议的软件实现,原生设计目的为了高可用 ipvs服务
  • keepalived 是高可用集群的通用无状态应用解决方案
  • 功能:
  • 基于vrrp协议完成地址流动
  • 为vip地址所在的节点生成ipvs规则(在配置文件中预先定义)
  • 为ipvs集群的各RS做健康状态检测
  • 基于脚本调用接口完成脚本中定义的功能,进而影响集群事务,以此支持nginx、haproxy等服务

Keepalived 环境准备

54-Keepalive-基于VRRP技术的集群VIP漂移以及各种生产常见案例_keepalived_03

包安装

[root@ubuntu2204 ~]#apt list keepalived
正在列表... 完成
keepalived/jammy 1:2.2.4-0.2build1 amd64
[root@ubuntu2204 ~]#apt update;apt -y install keepalived
#默认缺少配置,服务无法启动,提示/etc/keepalived/keepalived.conf 不存在
[root@ubuntu2204 ~]#systemctl status keepalived.service
○ keepalived.service - Keepalive Daemon (LVS and VRRP)
Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor
preset: enabled)
Active: inactive (dead)
Condition: start condition failed at Thu 2023-01-12 15:28:42 CST; 4s ago
└─ ConditionFileNotEmpty=/etc/keepalived/keepalived.conf was not
met
112 15:26:47 ubuntu2204.wang.org systemd[1]: Condition check resulted in
Keepalive Daemon (LVS and VRRP) being skipped.
112 15:28:42 ubuntu2204.wang.org systemd[1]: Condition check resulted in
Keepalive Daemon (LVS and VRRP) being skipped.
#利用模板生成生配置文件
[root@ubuntu2204 ~]#cp /usr/share/doc/keepalived/samples/keepalived.conf.sample
/etc/keepalived/keepalived.conf
[root@ubuntu2204 ~]#systemctl start keepalived.service
[root@ubuntu2204 ~]#systemctl status keepalived.service
● keepalived.service - Keepalive Daemon (LVS and VRRP)
Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor
preset: enabled)
Active: active (running) since Thu 2023-01-12 15:29:54 CST; 1s ago
Main PID: 2174 (keepalived)
Tasks: 3 (limit: 2196)
Memory: 4.5M
CPU: 40ms
CGroup: /system.slice/keepalived.service
├─2174 /usr/sbin/keepalived --dont-fork
├─2175 /usr/sbin/keepalived --dont-fork
└─2176 /usr/sbin/keepalived --dont-fork

案例:实现master/slave的 Keepalived 单主架构

配置主从vrrp虚拟路由设备,同时加设心跳检测专用网卡

#202 主设备
[root@ka1 ~]#cat /etc/keepalived/keepalived.conf
global_defs {
router_id ka1.mooreyxia.org #每个keepalived主机唯一标识,建议使用当前主机名
vrrp_mcast_group4 224.1.1.1 #指定组播IP地址
}

vrrp_instance VI_1 { #vrrp的实例名,一般为业务名称
state MASTER #BACKUP#当前节点在此虚拟路由器上的初始状态,状态为MASTER或者BACKUP,当priority相同时,先启动的节点优先获取VIP
interface eth1 #绑定为当前VRRP虚拟路由器使用的物理接口做集群内心跳检测.注意:如果和虚拟IP一样,有脑裂现象
virtual_router_id 88 #每个虚拟路由器唯一标识,同一集群内公用
priority 100 #当前物理节点在此虚拟路由器的优先级
advert_int 1 #vrrp通告的时间间隔,默认1s
authentication { #认证机制
auth_type PASS #AH为IPSEC认证(不推荐),PASS为简单密码(建议使用)
auth_pass 123456 #预共享密钥,仅前8位有效,同一个虚拟路由器的多个keepalived节点必须一样
}
virtual_ipaddress { #虚拟IP,生产环境可能指定几十上百个VIP地址
10.0.0.200/24 dev eth0 lable eth0:1 #指定VIP的网卡,建议和interface指令指定的网卡不在一个网卡 #指定VIP的网卡label
}
[root@ka1 ~]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:f6:07:67 brd ff:ff:ff:ff:ff:ff
altname enp2s1
altname ens33
inet 10.0.0.202/24 brd 10.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet 10.0.0.200/24 scope global secondary eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fef6:767/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:f6:07:71 brd ff:ff:ff:ff:ff:ff
altname enp2s5
altname ens37
inet 192.168.10.202/24 brd 192.168.10.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fef6:771/64 scope link
valid_lft forever preferred_lft forever

[root@ka1 ~]#systemctl restart keepalived.service
[root@ka1 ~]#hostname -I
10.0.0.202 10.0.0.200 192.168.10.202

#203 从设备
[root@ka2 ~]#cat /etc/keepalived/keepalived.conf
global_defs {
router_id ka2.mooreyxia.org
vrrp_mcast_group4 224.1.1.1
}

vrrp_instance VI_1 {
state BACKUP
interface eth1
virtual_router_id 88
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.200/24 dev eth0 lable eth0:1
}
}
[root@ka2 ~]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:bb:ef:c8 brd ff:ff:ff:ff:ff:ff
altname enp2s1
altname ens33
inet 10.0.0.203/24 brd 10.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:febb:efc8/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:bb:ef:d2 brd ff:ff:ff:ff:ff:ff
altname enp2s5
altname ens37
inet 192.168.10.203/24 brd 192.168.10.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:febb:efd2/64 scope link
valid_lft forever preferred_lft forever
[root@ka2 ~]#systemctl restart keepalived.service
[root@ka2 ~]#hostname -I
10.0.0.203 192.168.10.203

#测试 - 抓包查看数据发送
[root@ka2 ~]#tcpdump -i eth1 -nn host 224.1.1.1
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:47:55.724918 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20
20:47:56.726320 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20
20:47:57.726704 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20

#停止202主设备服务
[root@ka1 ~]#systemctl stop keepalived.service
#数据包内容发生变化,从设备上线 注意prio值变化 100 --> 0 --> 80
[root@ka2 ~]#tcpdump -i eth1 -nn host 224.1.1.1
20:48:55.768656 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20
20:48:56.768491 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20
20:48:57.567124 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 0, authtype simple, intvl 1s, length 20
20:48:58.255613 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20
20:48:59.257236 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20
20:49:00.257956 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20
20:49:01.258241 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20

#此时主设备上的vrrp ip地址消失,在从设备上出现
[root@ka1 ~]#hostname -I
10.0.0.202 192.168.10.202
[root@ka2 ~]#hostname -I
10.0.0.203 10.0.0.200 192.168.10.203

#恢复主设备服务
[root@ka1 ~]#systemctl restart keepalived.service
#数据包内容发生变化,从设备上线 注意prio值变化 80 --> 100
[root@ka2 ~]#tcpdump -i eth1 -nn host 224.1.1.1
20:53:06.452921 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20
20:53:07.453781 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20
20:53:08.454422 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20
20:53:09.454788 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20
20:53:10.455557 IP 192.168.10.203 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 80, authtype simple, intvl 1s, length 20
20:53:10.522302 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20
20:53:11.524605 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20
20:53:12.525119 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20
20:53:13.525798 IP 192.168.10.202 > 224.1.1.1: VRRPv2, Advertisement, vrid 88, prio 100, authtype simple, intvl 1s, length 20

#此时主设备上的vrrp ip地址恢复,在从设备上vrrp ip地址消失
[root@ka1 ~]#hostname -I
10.0.0.202 10.0.0.200 192.168.10.202
[root@ka2 ~]#hostname -I
10.0.0.203 192.168.10.203

查看心跳线间的传输协议

*虽然集群间通信密码没有加密,还是可以被看到,有大于无

54-Keepalive-基于VRRP技术的集群VIP漂移以及各种生产常见案例_keepalived_04

启用 Keepalived 日志功能

案例:

[root@ka2 ~]#cat /lib/systemd/system/keepalived.service |grep EnvironmentFile
EnvironmentFile=-/apps/keepalived/etc/sysconfig/keepalived
[root@ka2 ~]#vim /apps/keepalived/etc/sysconfig/keepalived
[root@ka2 ~]#cat /apps/keepalived/etc/sysconfig/keepalived
# Options for keepalived. See `keepalived --help' output and keepalived(8) and
# keepalived.conf(5) man pages for a list of all options. Here are the most
# common ones :
#
# --vrrp -P Only run with VRRP subsystem.
# --check -C Only run with Health-checker subsystem.
# --dont-release-vrrp -V Dont remove VRRP VIPs & VROUTEs on daemon stop.
# --dont-release-ipvs -I Dont remove IPVS topology on daemon stop.
# --dump-conf -d Dump the configuration data.
# --log-detail -D Detailed log messages.
# --log-facility -S 0-7 Set local syslog facility (default=LOG_DAEMON)
#

KEEPALIVED_OPTIONS="-D -S 6"

[root@ka2 ~]#vim /etc/rsyslog.
rsyslog.conf rsyslog.d/
[root@ka2 ~]#vim /etc/rsyslog.conf
[root@ka2 ~]#cat /etc/rsyslog.conf |grep local6
local6.* /var/log/keepalived.log
[root@ka2 ~]#systemctl restart keepalived rsyslog
#观察日志
[root@ka2 ~]#tail -f /var/log/keepalived.log
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: Registering Kernel netlink command channel
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: (/etc/keepalived/keepalived.conf: Line 17) Unknown configuration entry 'lable' for ip address - ignoring
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: (/etc/keepalived/keepalived.conf: Line 17) Unknown configuration entry 'eth0:1' for ip address - ignoring
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: Assigned address 192.168.10.203 for interface eth1
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: Assigned address fe80::20c:29ff:febb:efd2 for interface eth1
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: Registering gratuitous ARP shared channel
Jan 20 11:46:33 ka2 Keepalived[1467]: Startup complete
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: (VI_1) removing VIPs.
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: (VI_1) Entering BACKUP STATE (init)
Jan 20 11:46:33 ka2 Keepalived_vrrp[1468]: VRRP sockpool: [ifindex( 3), family(IPv4), proto(112), fd(12,13) multicast, address(224.1.1.1)]

实现 Keepalived 独立子配置文件

  • 当生产环境复杂时, /etc/keepalived/keepalived.conf 文件中保存所有集群的配置会导致内容过多,不易管理,可以将不同集群的配置,比如:不同集群的VIP配置放在独立的子配置文件中

案例:

[root@ka1 ~]#vim /etc/keepalived/keepalived.conf
[root@ka1 ~]#cat /etc/keepalived/keepalived.conf
global_defs {
router_id ka1.mooreyxia.org
vrrp_mcast_group4 224.1.1.1
}

include /etc/keepalived/conf.d/*.conf


[root@ka1 ~]#mkdir /etc/keepalived/conf.d/
[root@ka1 ~]#vim /etc/keepalived/conf.d/mooreyxia.org.conf
[root@ka1 ~]#cat /etc/keepalived/conf.d/mooreyxia.org.conf
vrrp_instance VI_1 {
state MASTER
interface eth1
virtual_router_id 88
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.200/24 dev eth0 lable eth0:1
}

[root@ka1 ~]#systemctl restart keepalived
[root@ka1 ~]#hostname -I
10.0.0.202 10.0.0.200 192.168.10.202

脑裂

主备节点同时拥有同一个VIP,此时为脑裂理象

脑裂现象原因

  • 心跳线故障
  • 防火墙错误配置
  • Keepalived 配置错误
  • interface错误,virtual_router_id不一致,密码不一致,优先级相同

范例: 发现脑裂

#在从节点服务器执行iptables -A INPUT -s 主服务心跳网卡IP -j DROP 进行模拟
[root@ka2 ~]#iptables -A INPUT -s 192.168.10.202 -j DROP
#主从节点都出现vrrp地址
[root@ka1 ~]#hostname -I
10.0.0.202 10.0.0.200 192.168.10.202
[root@ka2 ~]#hostname -I
10.0.0.203 10.0.0.200 192.168.10.203

#arping 发现两个MAC对外广播vrrp地址
[root@rocky8 ~]#arping 10.0.0.200
ARPING 10.0.0.200 from 10.0.0.8 eth0
Unicast reply from 10.0.0.200 [00:0C:29:BB:EF:C8] 1.081ms
Unicast reply from 10.0.0.200 [00:0C:29:F6:07:67] 1.104ms

非抢占模式 nopreempt

  • 默认为抢占模式 preempt,即当高优先级的主机恢复在线后,会抢占低先级的主机的master角色,造成网络抖动建议设置为非抢占模式 nopreempt ,即高优先级主机恢复后,并不会抢占低优先级主机的master 角色
  • 注意: 非抢占模式下,如果原主机down机, VIP迁移至的新主机, 后续新主机也发生down((keepalived服务down))时,VIP还会迁移回修复好的原主机
  • 但如果新主机的服务down掉(keepalived服务正常),原主机也不会接管VIP,仍会由新主机拥有VIP即非抢占式模式,只是适合当主节点宕机,切换到从节点的一次性的高可用性,后续即使当原主节点修复好,仍无法再次起到高用功能

注意:要关闭 VIP抢占,必须将各 Keepalived 服务器 state 配置为 BACKUP

#ka1
[root@ka1 ~]#vim /etc/keepalived/conf.d/mooreyxia.org.conf
[root@ka1 ~]#cat /etc/keepalived/conf.d/mooreyxia.org.conf
vrrp_instance VI_1 {
state BACKUP #都为BACKUP
interface eth1
virtual_router_id 88
priority 100
advert_int 1
nopreempt #添加此行,设为nopreempt
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.200/24 dev eth0 lable eth0:1
}
[root@ka1 ~]#systemctl restart keepalived

#ka2 - 从服务器不需要添加nopreempt
[root@ka2 ~]#cat /etc/keepalived/conf.d/mooreyxia.org.conf
vrrp_instance VI_1 {
state BACKUP
interface eth1
virtual_router_id 88
priority 80 #优先级低
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.200/24 dev eth0 lable eth0:1
}
}

#测试
#正常情况
[root@ka1 ~]#hostname -I
10.0.0.202 10.0.0.200 192.168.10.202
[root@ka2 ~]#hostname -I
10.0.0.203 192.168.10.203
#停止ka1服务,主服务vrrp漂移至从服务
[root@ka1 ~]#systemctl stop keepalived
[root@ka1 ~]#hostname -I
10.0.0.202 192.168.10.202
[root@ka2 ~]#hostname -I
10.0.0.203 10.0.0.200 192.168.10.203
#修复ka1服务,vrrp未漂移
[root@ka1 ~]#systemctl start keepalived
[root@ka1 ~]#hostname -I
10.0.0.202 192.168.10.202
[root@ka2 ~]#hostname -I
10.0.0.203 10.0.0.200 192.168.10.203

#如果此时从节点服务dwon机,vrrp依然会漂移至主节点
[root@ka2 ~]#systemctl stop keepalived
[root@ka2 ~]#hostname -I
10.0.0.203 192.168.10.203
[root@ka1 ~]#hostname -I
10.0.0.202 10.0.0.200 192.168.10.202

抢占延迟模式 preempt_delay

  • 抢占延迟模式,即优先级高的主机恢复后,不会立即抢回VIP,而是延迟一段时间(默认300s)再抢回VIP
#更改主节点抢占模式
[root@ka1 ~]#vim /etc/keepalived/conf.d/mooreyxia.org.conf
[root@ka1 ~]#cat /etc/keepalived/conf.d/mooreyxia.org.conf
vrrp_instance VI_1 {
state BACKUP #都为BACKUP
interface eth1
virtual_router_id 88
priority 100
advert_int 1
preempt_delay 60 #添加此行,设为延迟10秒再抢占
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.200/24 dev eth0 lable eth0:1
}
[root@ka1 ~]#systemctl restart keepalived

VIP 单播配置

  • 默认keepalived主机之间利用多播相互通告消息,会造成网络拥塞,可以替换成单播,减少网络流量
  • 另外:有些公有云不支持多播,可以利用单播实现

注意:启用 vrrp_strict 时,不能启用单播

案例:

#设置心跳线网络为单播模式
#主节点
[root@ka1 ~]#vim /etc/keepalived/conf.d/mooreyxia.org.conf
[root@ka1 ~]#cat /etc/keepalived/conf.d/mooreyxia.org.conf
vrrp_instance VI_1 {
state BACKUP
interface eth1
virtual_router_id 88
priority 100
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.200/24 dev eth0 lable eth0:1
}
unicast_src_ip 192.168.10.202 #本机IP #单播优先于多播
unicast_peer{
192.168.10.203 #如果有多个keepalived,再加其它节点的IP
}
}
[root@ka1 ~]#systemctl restart keepalived
#从节点
[root@ka2 ~]#vim /etc/keepalived/conf.d/mooreyxia.org.conf
[root@ka2 ~]#cat /etc/keepalived/conf.d/mooreyxia.org.conf
vrrp_instance VI_1 {
state BACKUP
interface eth1
virtual_router_id 88
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.200/24 dev eth0 lable eth0:1
}
unicast_src_ip 192.168.10.203
unicast_peer{
192.168.10.202
}
}

[root@ka2 ~]#systemctl restart keepalived

Keepalived 通知脚本配置

  • 通知脚本类型
#当前节点成为主节点时触发的脚本
notify_master <STRING>|<QUOTED-STRING>

#当前节点转为备节点时触发的脚本
notify_backup <STRING>|<QUOTED-STRING>

#当前节点转为“失败”状态时触发的脚本
notify_fault <STRING>|<QUOTED-STRING>

#通用格式的通知触发机制,一个脚本可完成以上三种状态的转换时的通知
notify <STRING>|<QUOTED-STRING>

#当停止VRRP时触发的脚本
notify_stop <STRING>|<QUOTED-STRING>

#脚本的调用方法 - 在 vrrp_instance VI_1 语句块的末尾加下面行
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
  • 实现 Keepalived 状态切换的通知脚本

案例:

#在所有 keepalived节点配置如下
[root@ka1 ~]#cat /etc/keepalived/notify.sh
#!/bin/bash
contact='mooreyxia@gmail.com'
email_send='xxxxx@qq.com'
email_passwd='xxxxx'
email_smtp_server='smtp.qq.com'
. /etc/os-release
msg_error() {
echo -e "\033[1;31m$1\033[0m"
}
msg_info() {
echo -e "\033[1;32m$1\033[0m"
}
msg_warn() {
echo -e "\033[1;33m$1\033[0m"
}
color () {
RES_COL=60
MOVE_TO_COL="echo -en \\033[${RES_COL}G"
SETCOLOR_SUCCESS="echo -en \\033[1;32m"
SETCOLOR_FAILURE="echo -en \\033[1;31m"
SETCOLOR_WARNING="echo -en \\033[1;33m"
SETCOLOR_NORMAL="echo -en \E[0m"
echo -n "$1" && $MOVE_TO_COL
echo -n "["
if [ $2 = "success" -o $2 = "0" ] ;then
${SETCOLOR_SUCCESS}
echo -n $" OK "
elif [ $2 = "failure" -o $2 = "1" ] ;then
${SETCOLOR_FAILURE}
echo -n $"FAILED"
else
${SETCOLOR_WARNING}
echo -n $"WARNING"
fi
${SETCOLOR_NORMAL}
echo -n "]"
echo
}
install_sendemail () {
if [[ $ID =~ rhel|centos|rocky ]];then
rpm -q sendemail &> /dev/null || yum install -y sendemail
elif [ $ID = 'ubuntu' ];then
dpkg -l |grep -q sendemail || { apt update; apt install -y libiosocket-
ssl-perl libnet-ssleay-perl sendemail ; }
else
color "不支持此操作系统,退出!" 1
exit
fi
}
send_email () {
local email_receive="$1"
local email_subject="$2"
local email_message="$3"
sendemail -f $email_send -t $email_receive -u $email_subject -m
$email_message -s $email_smtp_server -o message-charset=utf-8 -o tls=yes -xu
$email_send -xp $email_passwd
[ $? -eq 0 ] && color "邮件发送成功!" 0 || color "邮件发送失败!" 1
}
notify() {
if [[ $1 =~ ^(master|backup|fault)$ ]];then
mailsubject="$(hostname) to be $1, vip floating"
mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be
$1"
send_email "$contact" "$mailsubject" "$mailbody"
else
echo "Usage: $(basename $0) {master|backup|fault}"
exit 1
fi
}
install_sendemail
notify $1
[root@ka1 ~]#chmod a+x /etc/keepalived/notify.sh
[root@ka1 ~]#vim /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
......
virtual_ipaddress {
10.0.0.200/24 dev eth0 label eth0:1
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
#模拟master故障
[root@ka1 ~]#killall keepalived

实现 Master/Master 的 Keepalived 双主架构

54-Keepalive-基于VRRP技术的集群VIP漂移以及各种生产常见案例_keepalived_05

  • 将两个或以上VIP分别运行在不同的keepalived服务器,以实现服务器并行提供web访问的目的,提高服务器资源利用率
  • 注意:多主模型在发生VIP迁移时会使另一个服务器负载加大,注意承压阈值
#在拥有了一个vip的基础上再添加一组vip,与上一个vip设置主从关系相反
#ka1
[root@ka1 ~]#vim /etc/keepalived/conf.d/mooreyxia.com.conf
[root@ka1 ~]#cat /etc/keepalived/conf.d/mooreyxia.com.conf
vrrp_instance VI_2 {
state BACKUP
interface eth1
virtual_router_id 89
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 654321
}
virtual_ipaddress {
10.0.0.100/24 dev eth0 lable eth0:2
}
unicast_src_ip 192.168.10.202
unicast_peer{
192.168.10.203
}
}
[root@ka1 ~]#systemctl restart keepalived
#ka2
[root@ka2 ~]#vim /etc/keepalived/conf.d/mooreyxia.com.conf
[root@ka2 ~]#cat /etc/keepalived/conf.d/mooreyxia.com.conf
vrrp_instance VI_2 {
state MASTER
interface eth1
virtual_router_id 89
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 654321
}
virtual_ipaddress {
10.0.0.100/24 dev eth0 lable eth0:2
}
unicast_src_ip 192.168.10.203
unicast_peer{
192.168.10.202
}
}
[root@ka2 ~]#systemctl restart keepalived

#测试
[root@ka1 conf.d]#hostname -I
10.0.0.202 10.0.0.200 192.168.10.202
[root@ka2 conf.d]#hostname -I
10.0.0.203 10.0.0.100 192.168.10.203

实现多主模型架构

  • 三个节点的三主六从架构实现
  • 实际上超出三个节点的情况时每个服务器一主两从也足够应对
  • 注意:多主模型在发生VIP迁移时会使另一个服务器负载加大,注意承压阈值

54-Keepalive-基于VRRP技术的集群VIP漂移以及各种生产常见案例_VIP漂移_06

案例:

#第一个节点ka1配置:
virtual_router_id 1 , Vrrp instance 1 , MASTER,优先级100
virtual_router_id 2 , Vrrp instance 2 , BACKUP,优先级80
virtual_router_id 3 , Vrrp instance 3 , BACKUP,优先级60
#第二个节点ka2配置:
virtual_router_id 1 , Vrrp instance 1 , BACKUP,优先级60
virtual_router_id 2 , Vrrp instance 2 , MASTER,优先级100
virtual_router_id 3 , Vrrp instance 3 , BACKUP,优先级80
#第三个节点ka3配置:
virtual_router_id 1 , Vrrp instance 1 , BACKUP,优先级80
virtual_router_id 2 , Vrrp instance 2 , BACKUP,优先级60
virtual_router_id 3 , Vrrp instance 3 , MASTER,优先级100

同步组

  • LVS NAT 模型VIP和DIP需要同步,需要同步组

案例:

vrrp_sync_group VG_1 {
group {
VI_1 # name of vrrp_instance (below)
VI_2 # One for each moveable IP
}
}
vrrp_instance VI_1 {
eth0
vip
}
vrrp_instance VI_2 {
eth1
dip
}

实现单主的 LVS-DR 模式

  • 多主LVS-DR模型的实现也可参照,更改keepalived为互为主从策略即可

虚拟服务器配置

virtual_server IP port { #VIP和PORT
delay_loop <INT> #检查后端服务器的时间间隔
lb_algo rr|wrr|lc|wlc|lblc|sh|dh #定义调度方法
lb_kind NAT|DR|TUN #集群的类型,注意要大写
persistence_timeout <INT> #持久连接时长
protocol TCP|UDP|SCTP #指定服务协议,一般为TCP
sorry_server <IPADDR> <PORT> #所有RS故障时,备用服务器地址
real_server <IPADDR> <PORT> { #RS的IP和PORT
weight <INT> #RS权重
notify_up <STRING>|<QUOTED-STRING> #RS上线通知脚本
notify_down <STRING>|<QUOTED-STRING> #RS下线通知脚本
HTTP_GET|SSL_GET|TCP_CHECK|SMTP_CHECK|MISC_CHECK { ... } #定义当前主机健康状态检测方法
}
}
#注意:括号必须分行写,两个括号写在同一行,如: }} 会出错

案例:

#准备两台后端RS主机
[root@rocky8 ~]#curl 10.0.0.204
web01 www.mooreyxia.org 10.0.0.204
[root@rocky8 ~]#curl 10.0.0.28
web02 www.mooreyxia.org 10.0.0.28

#使用脚本绑定VIP至web服务器lo网卡
[root@web01 ~]#cat lvs_dr_rs.sh
#!/bin/bash
#Author:mooreyxia
#Date:2023-01-20
vip=10.0.0.200
mask='255.255.255.255'
dev=lo:1

#rpm -q httpd &> /dev/null || yum -y install httpd &>/dev/null
#service httpd start &> /dev/null && echo "The httpd Server is Ready!"
#echo "`hostname -I`" > /var/www/html/index.html

case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig $dev $vip netmask $mask #broadcast $vip up
#route add -host $vip dev $dev
echo "The RS Server is Ready!"
;;
stop)
ifconfig $dev down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "The RS Server is Canceled!"
;;
*)
echo "Usage: $(basename $0) start|stop"
exit 1
;;
esac

#web01
[root@web01 ~]#bash lvs_dr_rs.sh start
The RS Server is Ready!
[root@web01 ~]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 10.0.0.200/32 scope global lo:1
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:e3:99:33 brd ff:ff:ff:ff:ff:ff
altname enp2s1
altname ens33
inet 10.0.0.204/24 brd 10.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fee3:9933/64 scope link
valid_lft forever preferred_lft forever

#web02
[root@web02 ~]#bash lvs_dr_rs.sh start
The RS Server is Ready!
[root@web02 ~]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 10.0.0.200/32 scope global lo:1
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:0c:29:d3:0d:20 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.28/24 brd 10.0.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fed3:d20/64 scope link
valid_lft forever preferred_lft forever

配置keepalived

#ka1节点的配置
[root@ka1 conf.d]#vim /etc/keepalived/conf.d/mooreyxia.org.conf
[root@ka1 conf.d]#cat /etc/keepalived/conf.d/mooreyxia.org.conf
vrrp_instance VI_1 {
state MASTER
interface eth1
virtual_router_id 88
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.200/24 dev eth0 lable eth0:1
}
unicast_src_ip 192.168.10.202
unicast_peer{
192.168.10.203
}
}

virtual_server 10.0.0.200 80 {
delay_loop 3
lb_algo wrr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80
real_server 10.0.0.204 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 10.0.0.28 80 {
weight 2
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
[root@ka1 conf.d]#systemctl restart keepalived
[root@ka1 conf.d]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:f6:07:67 brd ff:ff:ff:ff:ff:ff
altname enp2s1
altname ens33
inet 10.0.0.202/24 brd 10.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet 10.0.0.200/24 scope global secondary eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fef6:767/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:f6:07:71 brd ff:ff:ff:ff:ff:ff
altname enp2s5
altname ens37
inet 192.168.10.202/24 brd 192.168.10.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fef6:771/64 scope link
valid_lft forever preferred_lft forever


#ka2节点的配置
[root@ka2 conf.d]#vim mooreyxia.org.conf
[root@ka2 conf.d]#cat mooreyxia.org.conf
vrrp_instance VI_1 {
state BACKUP
interface eth1
virtual_router_id 88
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.200/24 dev eth0 lable eth0:1
}
unicast_src_ip 192.168.10.203
unicast_peer{
192.168.10.202
}
}

virtual_server 10.0.0.200 80 {
delay_loop 3
lb_algo wrr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80
real_server 10.0.0.204 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 10.0.0.28 80 {
weight 2
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
[root@ka2 conf.d]#systemctl restart keepalived

#观察lvs调度模型
[root@ka1 conf.d]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.0.200:80 wrr
-> 10.0.0.28:80 Route 2 0 0
-> 10.0.0.204:80 Route 1 0 0

#测试 - 权重策略成功
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web01 www.mooreyxia.org 10.0.0.204
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web01 www.mooreyxia.org 10.0.0.204
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28

#测试健康检查功能
#关闭后端服务后会自动调度到另一台服务上
[root@web01 ~]#systemctl stop nginx
#调度策略自动撤下204
[root@ka1 conf.d]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.0.200:80 wrr
-> 10.0.0.28:80 Route 2 0 0
#访问调度到28
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
#再度恢复204服务
[root@web01 ~]#systemctl start nginx
#调度策略自动加入204
[root@ka1 conf.d]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.0.200:80 wrr
-> 10.0.0.28:80 Route 2 0 4
-> 10.0.0.204:80 Route 1 0 0
#访问恢复调度策略
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web01 www.mooreyxia.org 10.0.0.204
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl www.mooreyxia.org
web01 www.mooreyxia.org 10.0.0.204

#当后端服务都失效,keepalived会将自身加入策略组
[root@web01 ~]#systemctl stop nginx
[root@web02 ~]#systemctl stop nginx
[root@ka1 conf.d]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.0.0.200:80 wrr
-> 127.0.0.1:80 Route 1 0 1

#本机服务可以做个失效页面或者做服务转发 --> 比如本机安装nginx 做失效html提醒

实现单主的 LVS-DR 模式,利用FWM绑定成多个服务为一个集群服务

  • 将http 80和https 443绑定到一个集群服务上
#在单主的 LVS-DR 模式的基础上配置后端服务https认证
[root@web01 conf.d]#vim certificate.sh
[root@web01 conf.d]#cat certificate.sh
#!/bin/bash

SITE_NAME=www.mooreyxia.org
CA_SUBJECT="/O=mooreyxia/CN=ca.mooreyxia.org"
SUBJECT="/C=CN/ST=huanggang/L=hubei/O=mooreyxia/CN=$SITE_NAME"
SERIAL=34
EXPIRE=202002
FILE=$SITE_NAME

openssl req -x509 -newkey rsa:2048 -subj $CA_SUBJECT -keyout ca.key -nodes -days 202002 -out ca.crt

openssl req -newkey rsa:2048 -nodes -keyout ${FILE}.key -subj $SUBJECT -out ${FILE}.csr

openssl x509 -req -in ${FILE}.csr -CA ca.crt -CAkey ca.key -set_serial $SERIAL -days $EXPIRE -out ${FILE}.crt

chmod 600 ${FILE}.key ca.key

[root@web01 conf.d]#bash certificate.sh
.+....+......+..+.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.........+......+.........+..+....+.....+...+.......+...+......+..+...+......+....+.........+.....+......+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+...+..+.........+....+..+.............+..+...+.+.........+...+...+.....+.+...+...........+.+.....+...+..........+...+.........+.......................+...+............+..........+...+...+.....+....+..+...+......+......+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
..+...+......+.......+.....+.........+......+...+....+..+..........+...+..+...+...+.......+.....+.+...........+.+..+.+......+.....+.+..+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+..+...+...+....+...+.....+.+...............+..+...+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.....+.........+......+......+...+..........+.....+....+...+..+...+............+..........+...+...+......+.....+........................+............+....+...+...........+......+.+..............+.........+.......+........+......+...+.+......+.....+.+.....+...+.+......+.........+..............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
...+.+.....+...+.+......+...+...+..+......+......+.+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...+..........+..+.+..................+..+.......+.................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...........+......+.........+....+..+.+.........+...+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
Certificate request self-signature ok
subject=C = CN, ST = huanggang, L = hubei, O = mooreyxia, CN = www.mooreyxia.org
[root@web01 conf.d]#ls
ca.crt ca.key certificate.sh www.mooreyxia.org.crt www.mooreyxia.org.csr www.mooreyxia.org.key
#合并ca.crt文件及www.mooreyxia.org.crt
[root@web01 conf.d]#mv www.mooreyxia.org.crt www.mooreyxia.org.pem
[root@web01 conf.d]#ls
ca.crt ca.key certificate.sh www.mooreyxia.org.crt www.mooreyxia.org.csr www.mooreyxia.org.key www.mooreyxia.org.pem

#nginx启用https
[root@web01 conf.d]#vim /etc/nginx/sites-enabled/default
[root@web01 conf.d]#cat /etc/nginx/sites-enabled/default
...
server {
listen 80 default_server;
listen [::]:80 default_server;

# SSL configuration
#
listen 443 ssl default_server; --> 开放443
# listen [::]:443 ssl default_server;
#
# Note: You should disable gzip for SSL traffic.
# See: https://bugs.debian.org/773332
#
# Read up on ssl_ciphers to ensure a secure configuration.
# See: https://bugs.debian.org/765782
#
# Self signed certs generated by the ssl-cert package
# Don't use them in a production server!
#
# include snippets/snakeoil.conf;

root /var/www/html;

# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
ssl_certificate /etc/nginx/conf.d/www.mooreyxia.org.pem; --> 添加证书
ssl_certificate_key /etc/nginx/conf.d/www.mooreyxia.org.key;

[root@web01 conf.d]#nginx -s reload
[root@web01 conf.d]#ss -nlt
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 511 0.0.0.0:80 0.0.0.0:*
LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:*
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 127.0.0.1:6010 0.0.0.0:*
LISTEN 0 511 0.0.0.0:443 0.0.0.0:*
LISTEN 0 128 127.0.0.1:6011 0.0.0.0:*
LISTEN 0 511 [::]:80 [::]:*
LISTEN 0 128 [::]:22 [::]:*
LISTEN 0 128 [::1]:6010 [::]:*
LISTEN 0 128 [::1]:6011 [::]:*

#web02上的nginx也开启443
[root@web02 conf.d]#cat /etc/nginx/nginx.conf
...
server {
listen 443 ssl http2 default_server;
# listen [::]:443 ssl http2 default_server;
server_name _;
root /usr/share/nginx/html;
#
ssl_certificate "/etc/nginx/conf.d/www.mooreyxia.org.pem";
ssl_certificate_key "/etc/nginx/conf.d/www.mooreyxia.org.key";
# ssl_session_cache shared:SSL:1m;
# ssl_session_timeout 10m;
# ssl_ciphers PROFILE=SYSTEM;
# ssl_prefer_server_ciphers on;
#
# # Load configuration files for the default server block.
# include /etc/nginx/default.d/*.conf;
#
location / {
}
#
error_page 404 /404.html;
location = /40x.html {
}
#
# error_page 500 502 503 504 /50x.html;
# location = /50x.html {
# }
}

}
[root@web02 conf.d]#nginx -s reload
[root@web02 conf.d]#ss -nlt
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:80 0.0.0.0:*
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 100 127.0.0.1:25 0.0.0.0:*
LISTEN 0 128 0.0.0.0:443 0.0.0.0:*
LISTEN 0 128 [::]:80 [::]:*
LISTEN 0 128 [::]:22 [::]:*
LISTEN 0 100 [::1]:25 [::]:*

#测试
[root@rocky8 ~]#curl -k https://10.0.0.204
web01 www.mooreyxia.org 10.0.0.204
[root@rocky8 ~]#curl -k https://10.0.0.28
web02 www.mooreyxia.org 10.0.0.28

合并http和https两个集群

#两个节点都执行以下操作 - 将80和443作为相同的一组通讯 通过iptables贴标签的方式
[root@ka1 conf.d]#iptables -t mangle -A PREROUTING -d 10.0.0.200 -p tcp -m multiport --dports 80,443 -j MARK --set-mark 6
[root@ka2 conf.d]#iptables -t mangle -A PREROUTING -d 10.0.0.200 -p tcp -m multiport --dports 80,443 -j MARK --set-mark 6

#在keepalived中添加一组防火墙标签的服务
#ka1
[root@ka1 conf.d]#vim mooreyxia.org.conf
[root@ka1 conf.d]#cat mooreyxia.org.conf
vrrp_instance VI_1 {
state MASTER
interface eth1
virtual_router_id 88
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.200/24 dev eth0 lable eth0:1
}
unicast_src_ip 192.168.10.202
unicast_peer{
192.168.10.203
}
}

virtual_server fwmark 6 { #指定FWM为6
delay_loop 3
lb_algo wrr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80 #注意端口必须指定
real_server 10.0.0.204 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 10.0.0.28 80 {
weight 2
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
[root@ka1 conf.d]#systemctl restart keepalived
[root@ka1 conf.d]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 6 wrr
-> 10.0.0.28:0 Route 2 0 0
-> 10.0.0.204:0 Route 1 0 0
#ka2
[root@ka2 conf.d]#vim mooreyxia.org.conf
[root@ka2 conf.d]#cat mooreyxia.org.conf
vrrp_instance VI_1 {
state BACKUP
interface eth1
virtual_router_id 88
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.200/24 dev eth0 lable eth0:1
}
unicast_src_ip 192.168.10.203
unicast_peer{
192.168.10.202
}
}

virtual_server fwmark 6 {
delay_loop 3
lb_algo wrr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80
real_server 10.0.0.204 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 10.0.0.28 80 {
weight 2
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
[root@ka2 conf.d]#systemctl restart keepalived
[root@ka2 conf.d]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 6 wrr
-> 10.0.0.28:0 Route 2 0 0
-> 10.0.0.204:0 Route 1 0 0

测试 - http与https都会调度到同一集群,只要一个服务器调度到http,另一个服务器必定调度到https,实现负载均衡策略

[root@rocky8 ~]#curl http://www.mooreyxia.org  -k https://www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
web01 www.mooreyxia.org 10.0.0.204
[root@rocky8 ~]#curl http://www.mooreyxia.org -k https://www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28 --> 这里设置了权重导致
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl http://www.mooreyxia.org -k https://www.mooreyxia.org
web01 www.mooreyxia.org 10.0.0.204
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl http://www.mooreyxia.org -k https://www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28
web01 www.mooreyxia.org 10.0.0.204
[root@rocky8 ~]#curl http://www.mooreyxia.org -k https://www.mooreyxia.org
web02 www.mooreyxia.org 10.0.0.28 --> 这里设置了权重导致
web02 www.mooreyxia.org 10.0.0.28
[root@rocky8 ~]#curl http://www.mooreyxia.org -k https://www.mooreyxia.org
web01 www.mooreyxia.org 10.0.0.204
web02 www.mooreyxia.org 10.0.0.28

VRRP Script

  • keepalived利用 VRRP Script 技术,可以调用外部的辅助脚本进行资源监控,并根据监控的结果实现优先动态调整,从而实现其它应用的高可用性功能

案例:实现单主模式的 Nginx 反向代理的高可用

#在两个节点都配置nginx反向代理
[root@ka1-centos8 ~]#vim /etc/nginx/nginx.conf
http {
upstream websrvs {
server 10.0.0.7:80 weight=1;
server 10.0.0.17:80 weight=1;
}
server {
listen 80;
location /{
proxy_pass http://websrvs/;
}
}
}
#在两个节点都配置实现nginx反向代理高可用
[root@ka1-centos8 ~]#cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from kaadmin@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka1.mooreyxia.org #在另一个节点为ka2.mooreyxia.org
vrrp_mcast_group4 224.0.100.100
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
#script "/usr/bin/killall -0 nginx" 此写法支持
#script "/usr/bin/killall -0 nginx &>/dev/null" 不支持&>此写法
interval 1
weight -30
fall 3
rise 5
timeout 2
}
vrrp_instance VI_1 {
state MASTER #在另一个节点为BACKUP
interface eth0
virtual_router_id 66
priority 100 #在另一个节点为80
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
10.0.0.10/24 dev eth0 label eth0:1
}
track_interface {
eth0
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
track_script {
check_nginx
}
}
[root@ka1-centos8 ~]# yum install psmisc -y
[root@ka1-centos8 ~]# cat /etc/keepalived/check_nginx.sh
#!/bin/bash
/usr/bin/killall -0 nginx || systemctl restart nginx --> killall -0 service 可以检查僵尸进程
[root@ka1-centos8 ~]# chmod a+x /etc/keepalived/check_nginx.sh

我是moore,大家一起加油!!!