上一章节博主为大家介绍了目前大型互联网项目的keepalived+nginx(主备)高可用系统架构体系,相信大家应该看了博主的文章对keepalived/nginx技术已经有一定的了解,在本节博主将为大家分享keepalived+nginx(多主多活)高可用架构体系的相关技术以及配置过程。
由于前面的文章已经介绍了keepalived、nginx的安装过程,本节就不再重复累赘,直接开始讲多活的配置以及自动化脚本监控内容。
配置步骤(本次以三台nginx服务器的三主配置为例):
一、先按照前一章节安装好keepalived、nginx等软件
二、keepalived修改配置文件
(2.1)第一台服务器
keepalived.config配置文件(目录/etc/keepalived/keepalived.conf)
! Configuration File for keepalived
global_defs {
}
vrrp_script chk_nginx {
#script "[[ `ps -ef | grep nginx | grep -v grep | wc -l` -ge 2 ]] && exit 0 || exit 1"
script "/usr/local/keepalived/sbin/check_ng_pid.sh"
interval 1 #每隔1秒执行上述的脚本,去检查用户的程序ngnix
weight -10
}
vrrp_instance VI_1 {
state MASTER #指定A节点为主节点 备用节点上设置为BACKUP即可
interface eth0 #绑定虚拟IP的网络接口
virtual_router_id 52 #VRRP组名,在同一个instance中一致,在整个vrrp中唯一,以指明各个节点属于同一VRRP组
priority 200 #主节点的优先级(1-254之间),备用节点必须比主节点优先级低
advert_int 1 #组播信息发送间隔,两个节点设置必须一样
authentication { #设置验证信息,两个节点必须一致
auth_type PASS
auth_pass 1111
}
track_script {
chk_nginx
}
virtual_ipaddress { #指定虚拟IP,同一个instance中一致,整个vrrp中唯一
192.168.29.191/24 #如果两个nginx的ip分别是192.168.33.61,,...62,则此处的虚拟ip跟它俩同一个网段即可
}
notify_master "/usr/local/keepalived/sbin/notify.sh master"
notify_backup "/usr/local/keepalived/sbin/notify.sh backup"
notify_fault "/usr/local/keepalived/sbin/notify.sh fault"
}
vrrp_instance VI_2 {
state BACKUP #指定A节点为主节点 备用节点上设置为BACKUP即可
interface eth0 #绑定虚拟IP的网络接口
virtual_router_id 53 #VRRP组名,在同一个instance中一致,在整个vrrp中唯一,以指明各个节点属于同一VRRP组
priority 150 #主节点的优先级(1-254之间),备用节点必须比主节点优先级低
advert_int 1 #组播信息发送间隔,两个节点设置必须一样
authentication { #设置验证信息,两个节点必须一致
auth_type PASS
auth_pass 1111
}
track_script {
chk_nginx
}
virtual_ipaddress { #指定虚拟IP,同一个instance中一致,整个vrrp中唯一
192.168.29.192/24 #如果两个nginx的ip分别是192.168.33.61,,...62,则此处的虚拟ip跟它俩同一个网段即可
}
notify_master "/usr/local/keepalived/sbin/notify.sh master"
notify_backup "/usr/local/keepalived/sbin/notify.sh backup"
notify_fault "/usr/local/keepalived/sbin/notify.sh fault"
}
vrrp_instance VI_3 {
state BACKUP #指定A节点为主节点 备用节点上设置为BACKUP即可
interface eth0 #绑定虚拟IP的网络接口
virtual_router_id 54 #VRRP组名,在同一个instance中一致,在整个vrrp中唯一,以指明各个节点属于同一VRRP组
priority 100 #主节点的优先级(1-254之间),备用节点必须比主节点优先级低
advert_int 1 #组播信息发送间隔,两个节点设置必须一样
authentication { #设置验证信息,两个节点必须一致
auth_type PASS
auth_pass 1111
}
track_script {
chk_nginx
}
virtual_ipaddress { #指定虚拟IP,同一个instance中一致,整个vrrp中唯一
192.168.29.193/24 #如果两个nginx的ip分别是192.168.33.61,,...62,则此处的虚拟ip跟它俩同一个网段即可
}
notify_master "/usr/local/keepalived/sbin/notify.sh master"
notify_backup "/usr/local/keepalived/sbin/notify.sh backup"
notify_fault "/usr/local/keepalived/sbin/notify.sh fault"
}
(2.2)第二台服务器
keepalived.config配置文件(目录/etc/keepalived/keepalived.conf)
! Configuration File for keepalived
global_defs {
}
vrrp_script chk_nginx {
#script "[[ `ps -ef | grep nginx | grep -v grep | wc -l` -ge 2 ]] && exit 0 || exit 1"
script "/usr/local/keepalived/sbin/check_ng_pid.sh"
interval 1 #每隔1秒执行上述的脚本,去检查用户的程序ngnix
weight -10
}
vrrp_instance VI_1 {
state BACKUP #指定A节点为主节点 备用节点上设置为BACKUP即可
interface eth0 #绑定虚拟IP的网络接口
virtual_router_id 52 #VRRP组名,在同一个instance中一致,在整个vrrp中唯一,以指明各个节点属于同一VRRP组
priority 100 #主节点的优先级(1-254之间),备用节点必须比主节点优先级低
advert_int 1 #组播信息发送间隔,两个节点设置必须一样
authentication { #设置验证信息,两个节点必须一致
auth_type PASS
auth_pass 1111
}
track_script {
chk_nginx
}
virtual_ipaddress { #指定虚拟IP,同一个instance中一致,整个vrrp中唯一
192.168.29.191/24 #如果两个nginx的ip分别是192.168.33.61,,...62,则此处的虚拟ip跟它俩同一个网段即可
}
notify_master "/usr/local/keepalived/sbin/notify.sh master"
notify_backup "/usr/local/keepalived/sbin/notify.sh backup"
notify_fault "/usr/local/keepalived/sbin/notify.sh fault"
}
vrrp_instance VI_2 {
state MASTER #指定A节点为主节点 备用节点上设置为BACKUP即可
interface eth0 #绑定虚拟IP的网络接口
virtual_router_id 53 #VRRP组名,在同一个instance中一致,在整个vrrp中唯一,以指明各个节点属于同一VRRP组
priority 200 #主节点的优先级(1-254之间),备用节点必须比主节点优先级低
advert_int 1 #组播信息发送间隔,两个节点设置必须一样
authentication { #设置验证信息,两个节点必须一致
auth_type PASS
auth_pass 1111
}
track_script {
chk_nginx
}
virtual_ipaddress { #指定虚拟IP,同一个instance中一致,整个vrrp中唯一
192.168.29.192/24 #如果两个nginx的ip分别是192.168.33.61,,...62,则此处的虚拟ip跟它俩同一个网段即可
}
notify_master "/usr/local/keepalived/sbin/notify.sh master"
notify_backup "/usr/local/keepalived/sbin/notify.sh backup"
notify_fault "/usr/local/keepalived/sbin/notify.sh fault"
}
vrrp_instance VI_3 {
state BACKUP #指定A节点为主节点 备用节点上设置为BACKUP即可
interface eth0 #绑定虚拟IP的网络接口
virtual_router_id 54 #VRRP组名,在同一个instance中一致,在整个vrrp中唯一,以指明各个节点属于同一VRRP组
priority 150 #主节点的优先级(1-254之间),备用节点必须比主节点优先级低
advert_int 1 #组播信息发送间隔,两个节点设置必须一样
authentication { #设置验证信息,两个节点必须一致
auth_type PASS
auth_pass 1111
}
track_script {
chk_nginx
}
virtual_ipaddress { #指定虚拟IP,同一个instance中一致,整个vrrp中唯一
192.168.29.193/24 #如果两个nginx的ip分别是192.168.33.61,,...62,则此处的虚拟ip跟它俩同一个网段即可
}
notify_master "/usr/local/keepalived/sbin/notify.sh master"
notify_backup "/usr/local/keepalived/sbin/notify.sh backup"
notify_fault "/usr/local/keepalived/sbin/notify.sh fault"
}
(2.3)第三台服务器
keepalived.config配置文件(目录/etc/keepalived/keepalived.conf)
! Configuration File for keepalived
global_defs {
}
vrrp_script chk_nginx {
#script "[[ `ps -ef | grep nginx | grep -v grep | wc -l` -ge 2 ]] && exit 0 || exit 1"
script "/usr/local/keepalived/sbin/check_ng_pid.sh"
interval 1 #每隔1秒执行上述的脚本,去检查用户的程序ngnix
weight -10
}
vrrp_instance VI_1 {
state BACKUP #指定A节点为主节点 备用节点上设置为BACKUP即可
interface eth0 #绑定虚拟IP的网络接口
virtual_router_id 52 #VRRP组名,在同一个instance中一致,在整个vrrp中唯一,以指明各个节点属于同一VRRP组
priority 150 #主节点的优先级(1-254之间),备用节点必须比主节点优先级低
advert_int 1 #组播信息发送间隔,两个节点设置必须一样
authentication { #设置验证信息,两个节点必须一致
auth_type PASS
auth_pass 1111
}
track_script {
chk_nginx
}
virtual_ipaddress { #指定虚拟IP,同一个instance中一致,整个vrrp中唯一
192.168.29.191/24 #如果两个nginx的ip分别是192.168.33.61,,...62,则此处的虚拟ip跟它俩同一个网段即可
}
notify_master "/usr/local/keepalived/sbin/notify.sh master"
notify_backup "/usr/local/keepalived/sbin/notify.sh backup"
notify_fault "/usr/local/keepalived/sbin/notify.sh fault"
}
vrrp_instance VI_2 {
state BACKUP #指定A节点为主节点 备用节点上设置为BACKUP即可
interface eth0 #绑定虚拟IP的网络接口
virtual_router_id 53 #VRRP组名,在同一个instance中一致,在整个vrrp中唯一,以指明各个节点属于同一VRRP组
priority 100 #主节点的优先级(1-254之间),备用节点必须比主节点优先级低
advert_int 1 #组播信息发送间隔,两个节点设置必须一样
authentication { #设置验证信息,两个节点必须一致
auth_type PASS
auth_pass 1111
}
track_script {
chk_nginx
}
virtual_ipaddress { #指定虚拟IP,同一个instance中一致,整个vrrp中唯一
192.168.29.192/24 #如果两个nginx的ip分别是192.168.33.61,,...62,则此处的虚拟ip跟它俩同一个网段即可
}
notify_master "/usr/local/keepalived/sbin/notify.sh master"
notify_backup "/usr/local/keepalived/sbin/notify.sh backup"
notify_fault "/usr/local/keepalived/sbin/notify.sh fault"
}
vrrp_instance VI_3 {
state MASTER #指定A节点为主节点 备用节点上设置为BACKUP即可
interface eth0 #绑定虚拟IP的网络接口
virtual_router_id 54 #VRRP组名,在同一个instance中一致,在整个vrrp中唯一,以指明各个节点属于同一VRRP组
priority 200 #主节点的优先级(1-254之间),备用节点必须比主节点优先级低
advert_int 1 #组播信息发送间隔,两个节点设置必须一样
authentication { #设置验证信息,两个节点必须一致
auth_type PASS
auth_pass 1111
}
track_script {
chk_nginx
}
virtual_ipaddress { #指定虚拟IP,同一个instance中一致,整个vrrp中唯一
192.168.29.193/24 #如果两个nginx的ip分别是192.168.33.61,,...62,则此处的虚拟ip跟它俩同一个网段即可
}
notify_master "/usr/local/keepalived/sbin/notify.sh master"
notify_backup "/usr/local/keepalived/sbin/notify.sh backup"
notify_fault "/usr/local/keepalived/sbin/notify.sh fault"
}
(2.4)每台服务器上都需要的脚本,所有脚本都需要有执行权限:chmod 777 /usr/local/keepalived/sbin/notify.sh
通知脚本配置/usr/local/keepalived/sbin/notify.sh
#!/bin/bash
case "$1" in
master)
/usr/local/nginx/sbin/nginx
exit 0
;;
backup)
/usr/local/nginx/sbin/nginx -s stop
/usr/local/nginx/sbin/nginx
exit 0
;;
fault)
/usr/local/nginx/sbin/nginx -s stop
exit 0
;;
*)
echo 'Usage: notify.sh {master|backup|fault}'
exit 1
;;
esac
nginx服务检查脚本/usr/local/keepalived/sbin/check_ng_pid.sh
#监控nginx进程,若nginx主进程不存在则启动nginx
# 若5s后nginx进程还是不存在的话kill掉keepalived进程,防止nginx没运行该主机的keepalived还接管虚拟IP
#!/bin/bash
SERVER=127.0.0.1
PASSWORD=hadoop
#获取ssh公钥
get_rsa(){
expect -c "set timeout -1;
spawn ssh-keygen -t rsa;
expect {
*Enter* {send -- \r;exp_continue;}
{Overwrite (y/n)*} {send -- n\r;exp_continue}
eof {exit 0;}
}";
}
auto_ssh_copy_id() {
expect -c "set timeout -1;
spawn ssh-copy-id $1;
expect {
*(yes/no)* {send -- yes\r;exp_continue;}
*assword:* {send -- $2\r;exp_continue;}
# *ERROR: No identities found* {get_rsa;ssh_copy_id_to_all ;exp_continue; }
eof {exit 0;}
}";
}
ssh_copy_id_to_all() {
auto_ssh_copy_id $SERVER $PASSWORD
}
#在远程主机上执行delay_stop.sh
execute_sh(){
expect -c "set timeout -1;
spawn ssh root@$SERVER nohup /usr/local/keepalived/sbin/delay_stop.sh > /dev/null 2>&1 &
expect {
*(yes/no)* {send -- yes\r;exp_continue;}
*password:* {send -- $1\r;exp_continue;}
eof {exit 0;}
}";
}
get_rsa
ssh_copy_id_to_all
c1=`netstat -antp |grep -v grep |grep nginx |wc -l`
if [ $c1 -eq 0 ]; then
#/usr/local/nginx/sbin/nginx
#sleep 2
c2=`netstat -antp |grep -v grep |grep nginx |wc -l`
if [ $c2 -eq 0 ]; then
execute_sh $PASSWORD
# ssh root@$SERVER > /dev/null 2>&1 <<eeooff
#nohup service keepalived stop &
# nohup /usr/local/keepalived/sbin/delay_stop.sh > /dev/null 2>&1 &
# exit
#eeooff
exit 0
# ssh root@$SERVER "/etc/init.d/keepalived stop" > /usr/local/keepalived/sbin/a.txt
# service keepalived stop
# killall keepalived
# /etc/init.d/keepalived stop
#ps -ef | grep keepalived | grep -v grep | awk '{print $8}' | xargs kill
else
exit 0
fi
else
exit 0
fi
关闭keepalived服务,实现vip飘移的脚本/usr/local/keepalived/sbin/delay_stop.sh
#bin/bash
sleep 3
ssh root@127.0.0.1 service keepalived stop
#service keepalived stop
(2.5)配置好,检查openssh-server、openssh-clients、expect(自动化脚本中使用命令)是否已经安装
#安装ssh
rpm -qa|grep openssh 查看openssh组件是否已经安装
yum list|grep openssh 列出yum库中可用于安装的openssh软件包
yum install -y openssh-server 使用yum安装
yum install -y openssh-clients 使用yum安装
#安装expect
rpm -qa|grep expect
yum list|grep expect
yum install -y expect
(2.6)启动ssh服务,命令service sshd start,检查root用户是否可用连接
ssh root@127.0.0.1 如果报权限限制,需要改vi /etc/ssh/sshd_config,将PermitRootLogin yes 这行的注释”#“去掉;
注意:此处博主偷懒就直接使用root用户了,在生产环境一般会使用专门的用户来做自动化脚本的执行。
(2.7) 测试keepalived+nginx的高可用多主多活集群是否可用
a.停掉三台服务器上的防火墙 :service iptables stop
b.启动三台服务器上的keepalived命令(keepalived会自动启动nginx):service keepalived start
c.查看三台服务器网卡地址vip是否绑定正常
d.关闭其中一台服务器上的nginx服务执行命令:killall nginx,查看网卡vip绑定变化是否改变,
vip已经不在此台服务器上,
e.访问vip飘逸的那台机器的虚拟vip:http://192.168.29.191,发现是可以访问到nginx的,说明vip已经漂移到其他机器(可通过ip addr命令查看)
f.检查关闭nginx的这台服务器的keepalived进程,keepalived已经自动关闭完成漂移
g.重启关闭的这台keepalived,并检查vip绑定是否回到本机
h.检查该vip虚拟机,nginx是否可访问:http://192.168.29.191
i.vip漂移,重新绑定都顺利完成,教程结束
最后总结:由于本章节博主在录制教程的时候,花费了太多时间,此处就不配置防火墙了。博主在编写脚本时遇到很多问题,都通过shell脚本调试工具解决:sh -vx check_ng_pid.sh;如果大家配置中,遇到问题,通过shell调试命令执行一次即可完成。以上是nginx多主多活架构搭建的全过程,如果大家觉得博主的文章还不错,请点赞;如果您对博主其它服务器技术或者博主本人感兴趣,请关注博主博客,并且欢迎随时跟博主沟通交流。