FreeSWITCH的高可用部署方式有两种:主备切换和负载均衡,官方文档介绍的主备切换部署是采用Corosync & Pacemaker,负载均衡采用前置opensips。但对使用keepalived进行主备切换的高可用方式没有介绍,同时网上对该种部署方式也没有介绍。
本人对Corosync & Pacemaker不熟悉,目前在职的公司web应用大部分采用keepalived+haproxy,所以对keepalived稍微熟悉一点,因此尝试使用keepalived进行freeswitch进行主备切换的高可用部署。
使用keepalived进行freeswitch进行主备切换的高可用部署比较简单,本文介绍的部署方案有两个亮点(自以为):1、主节点不抢占VIP,否则可能导致主节点恢复后VIP切换导致正在交互的SIP信令处理失败;2、FS可用性检测脚本和主备切换后话务接管脚本。
一、前提
1、freeswicth和keepalived均能通过服务启动;
2、两个节点freeswitch连接同一个mysql数据库;
二、环境
OS:Red Hat Enterprise Linux Server release 7.9
A节点IP:100.207.104.77
B节点IP:100.207.104.78
VIP:100.207.104.89
freeswitch域名:callcenter.xxxxx.com ---域名也可以直接使用VIP ,---域名暂无启用。
三、配置
3.1、允许应用绑定非本机IP
在两个节点均执行以下命令:
echo 'net.ipv4.ip_nonlocal_bind=1' >> /etc/sysctl.conf
cat > /etc/sysctl.conf << EOF
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
kernel.sysrq = 1
kernel.pid_max = 524288
fs.file-max = 6553600
kernel.sem = 500 256000 250 8192
net.ipv4.ip_local_port_range = 10000 65000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 1800
net.ipv4.tcp_retries2 = 5
net.core.rmem_default = 1048576
net.core.rmem_max = 1048576
net.core.wmem_default = 262144
net.core.wmem_max = 262144
net.core.somaxconn = 1024
vm.swappiness = 0
net.ipv4.ip_nonlocal_bind=1
net.ipv4.neigh.default.gc_thresh1 = 10240
net.ipv4.neigh.default.gc_thresh2 = 20480
net.ipv4.neigh.default.gc_thresh3 = 40960
EOF
sysctl -p
3.2 配置freeswitch
修改“/usr/local/freeswitch/conf/vars.xml ” ---具体路径视实际情况
将“local_ip_v4”的值修改为VIP:“100.207.104.89”
将“domain”的值修改为:“callcenter.xxxxx.com” ---域名暂无启用。
freeswitch 使用systemctl 启用服务方法
cat >/lib/systemd/system/freeswitch.service<<EOF
[Unit]
Description=FreeSWITCH
After=syslog.target network.target
After=postgresql.service postgresql-9.3.service postgresql-9.4.service mysqld.service httpd.service
[Service]
User=appdeploy
EnvironmentFile=-/usr/local/freeswitch/etc/sysconfig/freeswitch
WorkingDirectory=/usr/local/freeswitch
ExecStart=/usr/local/freeswitch/bin/freeswitch -nc -nf $FREESWITCH_PARAMS
ExecReload=/usr/bin/kill -HUP $MAINPID
#ExecStop=/usr/local/freeswitch/bin/freeswitch -stop $FREESWITCH_PARAMS
ExecStop=/bin/kill -9 $MAINPID
[Install]
WantedBy=multi-user.target
EOF
freeswitch启动、检查、停止、重启命令
systemctl start freeswitch
systemctl status freeswitch
systemctl stop freeswitch
systemctl restart freeswitch
开机自启动
systemctl enable freeswitch
关闭开机自启动
systemctl disable freeswitch
加载配置
systemctl daemon-reload
过滤查看启动项
systemctl list-unit-files --type=service |grep enabled
3.3 安装、配置keepalived
下载 keepalived-2.2.7.tar.gz
wget https://www.keepalived.org/software/keepalived-2.2.7.tar.gz --no-check-certificate
安装keepalived-2.2.7.tar.gz
yum install curl gcc openssl-devel libnl3-devel net-snmp-devel
解压:tar zxvf keepalived-2.2.7.tar.gz
tar zxvf keepalived-2.2.7.tar.gz
cd keepalived-2.2.7
./configure --prefix=/app/keepalived
make
make install
systemctl 启动 keepalived服务
cat > /lib/systemd/system/keepalived.service <<EOF
[Unit]
Description=keepalive
After=syslog.target network.target
[Service]
Type=forking
ExecStart=/app/keepalived/sbin/keepalived -f /app/keepalived/etc/keepalived/keepalived.conf -p /app/keepalived/run/keep.pid -r /app/keepalived/run/vrrp.pid
ExecStop=/bin/kill -HUP $MAINPID
Restart=always
User=appdeploy
Group=mwopr
[Install]
WantedBy=multi-user.target
EOF
keepalived 要想使用普通用户运行,必须使用setcap授权
setcap cap_net_admin,cap_net_raw=eip /app/keepalived/sbin/keepalived
setcap 'CAP_SETUID+eip CAP_SETGID+eip CAP_NET_RAW+eip CAP_NET_ADMIN+eip cap_net_bind_service+eip' /app/keepalived/sbin/keepalived
CAP_NET_ADMIN:允许执行网络管理任务
CAP_NET_RAW:允许使用原始套接字
CAP_SETGID: 允许改变进程的组ID
CAP_SETUID: 允许改变进程的用户ID
CAP_NET_BIND_SERVICE: 允许绑定到小于1024的端口
A、B节点 创建日志和进程ID目录:
mkdir -p /app/keepalived/log
mkdir -p /app/keepalived/run
keepalived启动、检查、停止、重启命令
systemctl start keepalived
systemctl status keepalived
systemctl stop keepalived
systemctl restart keepalived
开机自启动
systemctl enable keepalived
关闭开机自启动
systemctl disable keepalived
加载配置
systemctl daemon-reload
过滤查看启动项
systemctl list-unit-files --type=service |grep enabled
配置keepalived
keepalived默认配置文件路径“/app/keepalived/etc/keepalived/keepalived.conf ”
A节点keepalived配置:100.207.104.77
cat > /app/keepalived/etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
#每个keepalived取个不同名称
router_id 40
}
vrrp_script check_fs {
script "/app/keepalived/script/check_fs.sh"
interval 1
weight 2
}
vrrp_instance VI_1 {
# MASTER为主实例,BACKUP 为从实例
state MASTER
# 网卡名称
interface eth0
# 主备这里要配置为同样的
virtual_router_id 68
# 优先级,主要高于备. 一般主配置为100 备配置为80
priority 100
advert_int 1
# 主动抢占,主备都开启的话,服务器抢占过去后,要等这个服务器keepalived停掉才会漂移到另一台
nopreempt
authentication {
# 主备必须配置成同样的
auth_type PASS
# 主备必须配置成同样的
auth_pass 1111
}
track_script {
check_fs
}
virtual_ipaddress {
# vip,主备必须配置一样
10.207.104.89
}
notify_master "/app/keepalived/script/fs_recover.sh"
}
EOF
添加普通用户启动权限
# 全局定义
global_defs {
# 路由id
router_id LVS_DEVEL
# 设置运行脚本默认用户和组。如果没有指定,则默认用户为keepalived_script(需要该用户存在),否则为root用户。默认groupname同username
script_user appdeploy
# 如果脚本路径的任一部分对于非root用户来说,都具有可写权限,则不会以root身份运行脚本
enable_script_security
}
# 全局定义 global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.0.123 smtp_connect_timeout 30 # 路由id router_id LVS_DEVEL # 设置运行脚本默认用户和组。如果没有指定,则默认用户为keepalived_script(需要该用户存在),否则为root用户。默认groupname同username script_user appdeploy # 如果脚本路径的任一部分对于非root用户来说,都具有可写权限,则不会以root身份运行脚本 enable_script_security } # 检测Nginx是否还存活的脚本 vrrp_script check_nginx { # 脚本文件的位置 script "/app/app/keepalived/etc/keepalived/check_nginx.sh" interval 2 # 检测脚本执行的间隔 weight 2 # 设置当服务器的权重 } # 虚拟IP的配置 vrrp_instance VI_1 { state BACKUP # 表示当前服务器是MASTER还是BACKUP interface eth0 # 绑定的网卡名称 virtual_router_id 51 # 主、备机的virtual_router_id必须相同 priority 80 # 主、备机取不同的优先级,主机值较大,备份机值较小 advert_int 1 # 每隔多久发送一次心跳,默认是1秒 # 权限校验 authentication { auth_type PASS auth_pass 1111 } # 虚拟IP地址 virtual_ipaddress { 192.168.3.20 } # 指定监控脚本 track_script { # 这里的check_nginx对应上面的vrrp_script的名字 check_nginx } }
B节点keepalived配置:100.207.104.78
cat > /app/keepalived/etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
#每个keepalived取个不同名称
router_id 40
}
vrrp_script check_fs {
script "/app/keepalived/script/check_fs.sh"
interval 1
weight 2
}
vrrp_instance VI_1 {
# MASTER为主实例,BACKUP 为从实例
state BACKUP
# 网卡名称
interface eth0
# 主备这里要配置为同样的
virtual_router_id 68
# 优先级,主要高于备. 一般主配置为100 备配置为80
priority 80
advert_int 1
# 主动抢占,主备都开启的话,服务器抢占过去后,要等这个服务器keepalived停掉才会漂移到另一台
nopreempt
authentication {
# 主备必须配置成同样的
auth_type PASS
# 主备必须配置成同样的
auth_pass 1111
}
track_script {
check_fs
}
virtual_ipaddress {
# vip,主备必须配置一样
10.207.104.89
}
notify_master "/app/keepalived/script/fs_recover.sh"
}
EOF
3.4 检测脚本
A、B节点 检测脚本路径:
mkdir -p /app/keepalived/script/
检测可用性脚本
cat > /app/keepalived/script/check_fs.sh << EOF
#!/bin/sh
FS_CLI_PROG='/usr/local/bin/fs_cli'
FS_CLI_HOST='127.0.0.1'
FS_CLI_PORT='8021'
FS_CLI_PASS='ClueCon'
PROFILES='10.207.104.89'
VIP='10.207.104.89'
fs_cli() {
$FS_CLI_PROG -H $FS_CLI_HOST -P $FS_CLI_PORT -p $FS_CLI_PASS -x "$1"
}
sofia_profile_started() {
fs_cli "sofia xmlstatus" | grep "<name>$1</name>" | wc -l
}
save_log(){
count=1
str_tmp="`date +%Y-%m-%d_%H:%M:%S` "
while [ $# -ge 1 ];do
str_tmp="$str_tmp $1"
count=count+1
shift
done
echo $str_tmp >>/app/keepalived/log/check_fs_`date +%Y-%m-%d`.log
}
check_vrrp(){
ip a|grep $VIP|wc -l
}
check_fs_service(){
ps -ef |grep freeswitch.service|grep -v 'grep'|wc -l
}
# fs_cli "sofia recover"
for p in $PROFILES; do
if [ `sofia_profile_started "$p"` -eq 0 ]; then
# echo "$p DOWN"
log_str="$p DOWN"
save_log $log_str
if [ `check_vrrp` -eq 1 ];then
save_log "本机已经绑定VRRP,即将重启keepalived和FreeSWITCH。"
#service keepalived restart
systemctl restart keepalived
save_log "vrrp切换完成!"
if [ `check_fs_service` -eq 1 ];then
save_log "freeswitch服务正在操作中。"
else
#service freeswitch restart
systemctl restart keepalived
save_log "freeswitch重启成功!"
fi
else
if [ `check_fs_service` -eq 1 ];then
save_log "freeswitch服务正在操作中。 "
else
save_log "本机没有绑定VRRP,重启FreeSWITCH。"
#service freeswitch restart
systemctl restart keepalived
save_log "freeswitch重启成功!"
fi
fi
exit 1
fi
done
save_log "freeswitch状态检测:OK!"
#echo "OK"
exit 0
EOF
切换为主节点后恢复通话脚本
cat > /app/keepalived/script/fs_recover.sh << EOF
#!/bin/sh
FS_CLI_PROG='/usr/local/bin/fs_cli'
FS_CLI_HOST='127.0.0.1'
FS_CLI_PORT='8021'
FS_CLI_PASS='ClueCon'
PROFILES='10.207.104.89'
VIP='10.207.104.89'
fs_cli() {
$FS_CLI_PROG -H $FS_CLI_HOST -P $FS_CLI_PORT -p $FS_CLI_PASS -x "$1"
}
save_log(){
counti=1
str_tmp="`date +%Y-%m-%d_%H:%M:%S` "
while [ $# -ge 1 ];do
str_tmp="$str_tmp $1"
count=count+1
shift
done
echo $str_tmp >>/app/keepalived/log/check_fs_`date +%Y-%m-%d`.log
}
save_log "本节点切换为主用状态,开始接管切换前的通话。"
fs_cli "sofia recover"
fs_cli "raloadxml"
exit 0
EOF