利用ldirectord实现RS的高可用性 --- 实验:搭建Ldirectord,实现Real Server的高可用性

LVS高可用性 --- RS的高可用

  • 1 Director不可用,整个系统将不可用;SPoF Single Point of Failure 单点失败

       解决方案:高可用
           keepalived(实现相对容易的,轻量级的解决方案)
           heartbeat/corosync(重量级的实现方法,在生产中用的越来越少)
  • 2 某RS不可用时,Director依然会调度请求至此RS

       解决方案: 由Director对各RS健康状态进行检查,失败时禁用,成功时启用
       keepalived
       heartbeat/corosync

    ldirectord(更好的解决方案,可以放便的实现健康状态检查功能,此外,还带有IPVS策略的定义,此前定义IPVS策略是手动键入命令,但是ldirectord自动就把IPVS策略配置好了,不用手动配置了)

  • 检测方式
    • (a) 网络层检测,icmp
    • (b) 传输层检测,端口探测
    • (c) 应用层检测,请求某关键资源
    • RS全不可用时:backup server, sorry server

LVS在具体实现的时候存在单点失败的问题,例如LVS本身出故障了。

Ldirectord策略可以替代ipvsadm策略,所以,安装Ldirectord就可以不要ipvsadm策略了
[root@localhost ~]# ipvsadm -C

Ldirectord只能够配置LVS服务器,配置不了Real Server

ldirectord官网:http://horms.net/projects/ldirectord/

ldirectord ldirectord is a daemon to monitor and administer real servers in a LVS cluster of load balanced virtual servers. ldirectord typically used as a resource for Linux-HA , but can also be run from the command line.

ldirectord ldirectord是监控和管理实际服务器守护进程在LVS集群负载均衡的虚拟服务器。linux - ha ldirectord通常作为一个资源,但也可以从命令行运行。

ldirectord属于高可用集成套件中的一个包而已

安装ldirectord

[root@LVS ~]# yum -y install ldirectord-3.9.6-0rc1.1.2.x86_64.rpm
ldirectord依赖很多perl包

安装后生成的文件

[root@LVS ~]# rpm -ql ldirectord
/etc/ha.d
/etc/ha.d/resource.d
/etc/ha.d/resource.d/ldirectord
/etc/logrotate.d/ldirectord
/usr/lib/ocf/resource.d/heartbeat/ldirectord
/usr/lib/systemd/system/ldirectord.service
/usr/sbin/ldirectord

搭建Ldirectord,实现Real Server的高可用性

在RS机器上需要运行的脚本

[root@rs1 ~]# cat lvs_dr_rs.sh 
#!/bin/bash
vip=10.0.0.100
mask='255.0.0.0'
dev=lo:1
rpm -q httpd &> /dev/null || yum -y install httpd &>/dev/null
service httpd start &> /dev/null && echo "The httpd Server is Ready!"
echo "`hostname`" > /var/www/html/index.html

case $1 in
start)
    echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
    ifconfig $dev $vip netmask $mask
    echo "The RS Server is Ready!"
    ;;
stop)
    ifconfig $dev down
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "The RS Server is Canceled!"
    ;;
*) 
    echo "Usage: $(basename $0) start|stop"
    exit 1
    ;;
esac
[root@rs1 ~]#

在LVS机器上需要运行的脚本

[root@LVS ~]# cat lvs_dr_vs.sh 
#!/bin/bash
vip='10.0.0.100'
iface='ens34:1'
mask='255.255.255.255'
port='80'
rs1='192.168.159.102'
rs2='192.168.159.103'
scheduler='wrr'
type='-g'
rpm -q ipvsadm &> /dev/null || yum -y install ipvsadm &> /dev/null

case $1 in
start)
    ifconfig $iface $vip netmask $mask #broadcast $vip up
    iptables -F

    ipvsadm -A -t ${vip}:${port} -s $scheduler
    ipvsadm -a -t ${vip}:${port} -r ${rs1} $type -w 1
    ipvsadm -a -t ${vip}:${port} -r ${rs2} $type -w 1
    echo "The VS Server is Ready!"
    ;;
stop)
    ipvsadm -C
    ifconfig $iface down
    echo "The VS Server is Canceled!"
    ;;
*)
    echo "Usage: $(basename $0) start|stop"
    exit 1
    ;;
esac
[root@LVS ~]#

配置ldirectord,把配置文件模板,复制到/etc/ha.d下,当做配置文件

[root@localhost ~]# cp /usr/share/doc/ldirectord-3.9.6/ldirectord.cf /etc/ha.d/

配置文件中大部分都是注释,注释大部分都是范例,可以根据自己的生产环境,参考例子改改就可以了

Ldirectord重要的作用就是健康性检查功能,检查后端的Real Server是不是可用

配置文件内容

[root@LVS ~]# vim /etc/ha.d/ldirectord.cf
checktimeout=3              检查超时时间,探测一次3秒不回应,就认为死了
checkinterval=1             探测间隔(一秒探测一次,探测时间太久用户就可能会发觉服务不可用)
#fallback=127.0.0.1:80      备用服务器的地址(如果服务器全挂了会看到拒绝访问,入如果配置了这个选项,用户就会看到这个服务器提供的页面,一般配置成LVS服务器所在的地址,所以要确保LVS可以提供网页服务。去掉注释,启用)

在LVS服务器上搭建备用服务器

yum -y install httpd && systemctl start httpd
echo Sorror,Server Down! > /var/www/html/index.html

[root@LVS ~]# curl 10.0.0.100
Sorror,Server Down!

#fallback6=[::1]:80     IPV6地址
autoreload=yes          IPV6地址,不用管它。这个配置文件将来需不需要改完以后通过systemctl restart的方式生效。       不需要配置,修改完以后自动生效(第一次还需要把ldirectord服务手动的起来,起来以后,在修改这个文件,就会自动生效了,而不用重启服务)

#logfile="/var/log/ldirectord.log"      日志
#logfile="local0"                       日志级别

# Sample for an http virtual service        VIP
virtual=192.168.6.240:80
        real=192.168.6.2:80 gate 1      gate(DR模型)  1(权重)
        real=192.168.6.3:80 gate
        real=192.168.6.6:80 gate
service=http                服务
        scheduler=rr            调度算法
        #persistent=600         超时时间
        #netmask=255.255.255.255
        protocol=tcp            协议
        checktype=negotiate     测试的类型(健康性检查的方式)
        checkport=80            健康性就检查80端口,看测试页是否可以被访问可以访问就没问题
        request="index.html"    要探测的页面(准备一个测试页比较好)
        receive="Test Page"     探测页面中的关键字符串
        virtualhost=www.x.y.z   不用加,注释掉

修改配置文件

[root@LVS ~]# vim /etc/ha.d/ldirectord.cf
# Global Directives
checktimeout=3
checkinterval=1
fallback=127.0.0.1:80
#fallback6=[::1]:80
autoreload=yes
logfile="/var/log/ldirectord.log"
logfile="local0"

# Sample for an http virtual service
virtual=10.0.0.100:80
        real=192.168.111.102 gate 1
        real=192.168.111.103 gate 3
#       fallback=127.0.0.1:80 gate
        service=http
        scheduler=wrr
        #persistent=600
        #netmask=255.255.255.255
        protocol=tcp
        checktype=negotiate
        checkport=80
        request="test.html"
        receive="test"
#       virtualhost=www.x.y.z

准备测试页
[root@RS1 ~]# echo test > /var/www/html/test.html
[root@RS2 ~]# echo test > /var/www/html/test.html

启动服务
[root@localhost ~]# systemctl start ldirectord

注意:没有手工加ipvsadm策略,启动服务的时候会自动根据配置文件,生成的ipvsadm策略

[root@LVS ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.100:80 wrr
  -> 192.168.111.102:80           Route   1      0          0         
  -> 192.168.111.103:80           Route   3      0          0

配置LVS网络

ifconfig ens34:1 10.0.0.100 netmask 255.255.255.255 broadcast 10.0.0.100 up

从配置文件中读取出来的,不用使用ipvsadm添加策略了,除此,还能够做健康性检查

[root@client ~]# for i in {1..10}; do curl 10.0.0.100; done
server1.ding.com
server2.ding.com
server2.ding.com
server1.ding.com
server2.ding.com
……

访问测试
[root@client ~]# for i in {1..100};do curl 10.0.0.100; sleep 0.2; done
RS1
RS2
RS2
RS2

Real Server停机测试,把检测的文件内容改变就可以了,ldirectord检测不到要探测的文件的内容就认为RS宕机了

Real Server停机模拟
[root@RS1 ~]# > /var/www/html/test.html

RS宕机,会自动的把出错的机器从调度列表中踢出了

[root@LVS ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.100:80 wrr
  -> 192.168.111.103:80           Route   3      0          49        
[root@localhost ~]#

如果RS全部停止服务
[root@RS2 ~]# systemctl stop httpd

访问测试
[root@client ~]# curl 10.0.0.100
Sorror,Server Down!

如果服务器修复,会自动上线服务器,并提供调度

查看日志

[root@LVS ~]# tail -f /var/log/ldirectord.log 
[Mon Mar  5 20:33:05 2018|ldirectord|2559] Deleted fallback server: 127.0.0.1:80 (192.168.6.240:80)
[Mon Mar  5 20:33:05 2018|ldirectord|2559] Deleted real server: 192.168.6.2:80 (192.168.6.240:80)

RS的日志文件中会记录探测,1s就会有一个测试的请求

[root@RS1 ~]# tail -f /var/log/httpd/access_log
192.168.111.100 - - [05/Mar/2018:20:55:12 +0800] "GET /test.html HTTP/1.1" 200 5 "-" "libwww-perl/6.05"
192.168.111.100 - - [05/Mar/2018:20:55:13 +0800] "GET /test.html HTTP/1.1" 200 5 "-" "libwww-perl/6.05"
192.168.111.100 - - [05/Mar/2018:20:55:14 +0800] "GET /test.html HTTP/1.1" 200 5 "-" "libwww-perl/6.05"

实现综合调度 - 实现http和https的综合调度

实现综合调度

在LVS机器上打标签
iptables -t mangle -A PREROUTING -d 10.0.0.100 -p tcp -m multiport --dports 80,443 -j MARK --set-mark 10

查看配置文件中关于打标签的配置

[root@LVS /etc/ha.d]# vim ldirectord.cf
# Sample configuration for a fwmark based service For an explanation of
# fwmark see the ipvsadm(8) man page
#virtual=1                                  标签
#       real=192.168.6.2 gate
#       real=192.168.6.3 gate
#       real=192.168.6.6 gate
#       fallback=127.0.0.1:80 gate
#       service=http
#       scheduler=rr
#       #persistent=600
#       #netmask=255.255.255.255
#       protocol=fwm
#       checktype=negotiate
#       checkport=80
#       request="index.html"
#       receive="Test Page"
#       virtualhost=x.y.z

修改配置文件

# Sample for an http virtual service
virtual=10
        real=192.168.111.102 gate 1         不用写端口号
        real=192.168.111.103 gate 3         不用写端口号
#       fallback=127.0.0.1:80 gate
        service=http
        scheduler=wrr
        #persistent=600             持久连接,启用后就会一直往一个服务器上调度了
        #netmask=255.255.255.255
        protocol=fwm                #这个加不加都可以
        checktype=negotiate
        checkport=80
        request="test.html"
        receive="test"
#       virtualhost=www.x.y.z

修改完配置文件以后不需要重启服务,因为有自动加载功能

查看(不用重启服务)

[root@LVS ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  10 wrr
  -> 192.168.111.102:0            Route   1      0          0         
  -> 192.168.111.103:0            Route   3      0          0         
[root@LVS ~]#

访问测试

[root@client ~]# for i in {1..100} ; do curl -k https://10.0.0.100; curl 10.0.0.100; done
[root@client ~]# ssh 10.0.0.100
root@10.0.0.100's password: 
Last login: Thu Apr 26 11:51:32 2018 from 192.168.3.204
[root@LVS ~]#

其他端口不会调度,因为防火墙策略里面已经定义只针对80和443端口的访问才打标签,别的就不知道标签是啥

添加策略控制外部主机访问内网
在路由器上配置防火墙,实现安全防护

iptables -A FORWARD -p tcp -m multiport --dports 80,443 -j ACCEPT
iptables -I FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -j REJECT

测试

[root@client ~]# for i in {1..100} ;do curl -k https://10.0.0.100;curl 10.0.0.100;done
RS2
RS1
[root@client ~]# ssh 10.0.0.100
ssh: connect to host 10.0.0.100 port 22: Connection refused

启用持久连接
修改配置文件

[root@LVS ~]# vim /etc/ha.d/ldirectord.cf
persistent=600      去掉注释文件

[root@LVS ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  10 wrr persistent 600
  -> 192.168.111.102:0            Route   1      0          0         
  -> 192.168.111.103:0            Route   3      0          0         
[root@LVS ~]#

测试

[root@client ~]# for i in {1..100} ;do curl -k https://10.0.0.100;curl 10.0.0.100;done
RS2
RS2
RS2
RS2