Lvs+keepalived

前期为了测试云存储搭建了一个lvstomcat负载均衡集群,近期要求的云存储以及前端集群做破坏性试验,这样为了满足系统的高可用性对lvsdirector节点做高可用,以及对hadoop的管理节点也要做高可用,因为公司测试环境及线上环境linux系统版本较老,heartbeat2heartbeat3等版本可能安装稍微麻烦一点,这里对lvsdirector节点尝试用keepalived做高可用,对hadoopheartbeat做高可用(以及共享存储,本文暂时不介绍dahoop的高可用及共享存储,以及安装配置测试遇到的“报错”和“问题”,过一段时间补上),下面介绍lvs负载均衡集群前端分发负载director节点的keepalived高可用实现过程

目前架构图如下:

 


 

1、  软件下载安装:(director主节点与备用节点都安装)

1)  lvs软件ipvsadm的下载安装

#wget http://www.linuxvirtualserver.org/software/kernel-2.6/ipvsadm-1.24.tar.gz

#tar zxvf ipvsadm-1.24.tar.gz

#ln -s /usr/src/kernels/2.6.9-89.EL-smp-x86_64 /usr/src/linux //注释:这一步针对自己的系统做,否则一会安装ipvsadm会报错

# cd ipvsadm-1.24

#make

#make install

2)  keepalived的下载安装

#wget http://www.keepalived.org/software/keepalived-1.1.15.tar.gz

# tar zxvf keepalived-1.1.15.tar.gz

#cd keepalived-1.1.15

//注释:如果系统rhel版本低于5.0以下安装报错,这里要修改系统配置

//vi /usr/src/linux/include/linux/types.h  将下面2行用/*  */注释掉,如下图

/*

typedef __u16 __bitwise __sum16;

typedef __u32 __bitwise __wsum;

*/

                   #./configure

                   #make

                   #make install

2、  keepalived做成服务启动

#cp /usr/local/etc/rc.d/init.d/keepalived /etc/init.d/       注释:拷贝服务启动文件

#cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/      

#mkdir /etc/keepalived/                       注释:创建配置文件目录

#cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/    注释:拷贝配置文件

#cp /usr/local/sbin/keepalived /usr/sbin/             注释:拷贝可执行文件

#service keepalived start                     注释:测试服务启动与关闭

#service keepalived stop

3、配置keepalived

在配置keepalived时发现这个软件用着比heartbeat要简单很多

主节点的配置

#vi /etc/keepalived/keepalived.conf

 

! Configuration File for keepalived

 

global_defs {

   notification_email {

        reaaa@163.com

   }

   notification_email_from Alexandre.Cassen@firewall.loc

   smtp_server 127.0.0.1

   router_id LVS_DEVEL

}

 

vrrp_instance VI_1 {

    state MASTER

    interface eth0

    virtual_router_id 51

    priority 100

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

virtual_ipaddress {

       192.168.2.120

    }

}

 

virtual_server 192.168.2.120 8080 {

    delay_loop 6

    lb_algo wlc

    lb_kind DR

    persistence_timeout 1

    protocol TCP

 

    real_server 192.168.5.54 8080 {

        weight 1

        TCP_CHECK {

        connect_timeout 1

        nb_get_retry 3

        delay_before_retry 3

        connect_port 8080

        }

    }

    real_server 192.168.5.57 8080 {

        weight 1

                            TCP_CHECK {

        connect_timeout 1

        nb_get_retry 3

        delay_before_retry 3

        connect_port 8080

        }

    }

}

         BACKUP节点配置:

         #vi /etc/keepalived/keepalived.conf

         ! Configuration File for keepalived

 

global_defs {

   notification_email {

        renxiaotian@ninetowns.com

   }

   notification_email_from Alexandre.Cassen@firewall.loc

   smtp_server 127.0.0.1

   router_id LVS_DEVEL

}

 

vrrp_instance VI_1 {

    state BACKUP

    interface eth0

    virtual_router_id 51

    priority 99

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

virtual_ipaddress {

        192.168.2.120

    }

}

 

virtual_server 192.168.2.120 8080 {

    delay_loop 6

    lb_algo wlc

    lb_kind DR

    persistence_timeout 1

    protocol TCP

 

    real_server 192.168.5.54 8080 {

        weight 1

        TCP_CHECK {

        connect_timeout 1

        nb_get_retry 3

        delay_before_retry 3

        connect_port 8080

        }

    }

    real_server 192.168.5.57 8080 {

        weight 1

                   TCP_CHECK {

        connect_timeout 1

        nb_get_retry 3

        delay_before_retry 3

        connect_port 8080

        }

    }

}

 

注释:因为我用的tomcat后端8080端口没有修改为80,这个请读者自己按照自己的情况配置,包括各种IP与持久连接,realserver的数量,如果keepalived启动后查看后端的realserver没有达到自己配置的数量,那么就是端口没有指对。

3、  配置realserver

#vi realserver.sh在脚本中添加如下内容

#!/bin/bash

VIP=192.168.2.120

. /etc/rc.d/init.d/functions

case "$1" in

start)

        echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore

        echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce

        echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore

        echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce

        ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up

        /sbin/route add -host $VIP dev lo:0

        sysctl -p > /dev/null 2>&1

        echo "realserver start OK"

        ;;

stop)

        echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore

        echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce

        echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore

        echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce

        ifconfig lo:0 down

        /sbin/route del $VIP > /dev/null 2>&1

        echo "realserver stoped"

#chmod +x realserver.sh

并执行

#./realserver.sh

注释:2web服务器做相同操作

如果想开机启动配置rc.local

#echo ‘/root/realserver.sh start >> /etc/rc.local

Keepalived的日志在/var/log/messages故障切换测试可以查看

watch -n 1 'ipvsadm -L -n'连续查看lvs状况命令