lvs简介: LVS是一个开源的软件,由毕业于国防科技大学的章文嵩博士于1998年5月创立,可以实现LINUX平台下的简单负载均衡。LVS是Linux Virtual Server的缩写,意思是Linux虚拟服务器.

Lvs 的集群工作模式有3种:VS/NAT,vs/tun,vs/dr.

Lvs的调度算法:

LVS的算法分为两大类:
静态算法:只是根据算法进行调度并不考虑后端REALSERVER的实际连接情况
rr-论调算法,假如有两台服务器A,B,第一个请求给A,第二个给B,第三个给A依次往复
wrr-加权论调,假如有两台服务器A,B,A的性能是B的两倍,则在论调的同时给A上面分配的请求也大致会是B上面的两倍
dh-假如调度器的后面是两台缓存服务器A,B而不是真正的REALSERVER,则会尽可能的把相同请求或者把同一用户的请求转发到同一个缓存服务器上面以提高缓存命中率
sh-假如公司有两台防火墙让员工上网,则会把某个员工往外的访问及向内返回的请求结果定向到同一台防火墙上面,方便防火墙做established的状态检测

动态算法:前端的调度器会根据后端REALSERVER的实际连接情况来分配请求
活动链接:当前有数据包传输
非活动链接:当前连接出于建立状态但是没有数据传输
lc-同时检查后端REALSERVER上面活动状态和非活动状态的连接数使用(活动连接数*256+非活动连接数)数字小的将接收下次访问请求
wlc-加权的lc,使用(活动连接数*256+非活动连接数)/权重,数字小的将接收下次访问请求,是最常用的算法
sed-不考虑非活动状态,使用(活动状态+1)*256,数字小的将接收下次访问请求,+1主要是为了提高权重大的服务器的响应能力
nq-假设有两台服务器A,B,权重比为10:1,按照sed算法,只有当A服务器已经响应了10个请求之时两者的计算数值才相同,为了避免权重小的服务器过于空闲,nq沿用sed算法但是确保让每个服务器都不空闲,只有在不考虑非活动连接的情况下nq才能取代wlc算法
lblc-在dh的基础上面考虑后台服务器的负载能力
lblcr-在lblc的基础上,假设有A,B两台缓存服务器,某个用户第一次访问被重定向到A,第二次访问时A负载很大,B过于空闲这时也会打破原来的规则把客户的第二次访问重定向给B

Lvs的dr模型

一:Web1的搭建 192.168.0.101

[root@zzu ~]# ifconfig lo:0 192.168.0.100 netmask 255.255.255.255

配置一个vip地址

[root@zzu ~]# ifconfig

eth0 Link encap:Ethernet HWaddr 00:0C:29:4B:61:16

inet addr:192.168.0.101 Bcast:192.168.0.255 Mask:255.255.255.0

inet6 addr: fe80::20c:29ff:fe4b:6116/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:3707 errors:0 dropped:0 overruns:0 frame:0

TX packets:915 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:354625 (346.3 KiB) TX bytes:127781 (124.7 KiB)

Interrupt:67 Base address:0x2000

lo:0 Link encap:Local Loopback

inet addr:192.168.0.100 Mask:255.255.255.255

UP LOOPBACK RUNNING MTU:16436 Metric:1

设置real服务器的arp选项保证在进行arp解析的时候只有director的vip进行响应

[root@zzu ~]# echo "net.ipv4.conf.all.arp_announce = 2" >>/etc/sysctl.conf

[root@zzu ~]# echo "net.ipv4.conf.lo.arp_announce = 2" >>/etc/sysctl.conf

[root@zzu ~]# echo "net.ipv4.conf.all.arp_ignore = 1" >>/etc/sysctl.conf

[root@zzu ~]# echo "net.ipv4.conf.lo.arp_ignore = 1" >>/etc/sysctl.conf

设置一条特殊的路由,保证在回复客户端时使用的是vip的地址

[root@zzu ~]# route add -host 192.168.0.100 dev lo:0

[root@zzu ~]# route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

192.168.0.100 0.0.0.0 255.255.255.255 UH 0 0 0 lo

192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0

[root@zzu Server]# rpm -ivh httpd-2.2.3-31.el5.i386.rpm

[root@zzu Server]# cd /var/www/html/

[root@zzu html]# vim index.html

web1

[root@zzu html]# links http://192.168.0.101

二:web2的搭建 192.168.0.102

[root@zzu ~]# ifconfig lo:0 192.168.0.100 netmask 255.255.255.255

配置一个vip地址

[root@zzu ~]# ifconfig

eth0 Link encap:Ethernet HWaddr 00:0C:29:4B:61:16

inet addr:192.168.0.102 Bcast:192.168.0.255 Mask:255.255.255.0

inet6 addr: fe80::20c:29ff:fe4b:6116/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:3707 errors:0 dropped:0 overruns:0 frame:0

TX packets:915 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:354625 (346.3 KiB) TX bytes:127781 (124.7 KiB)

Interrupt:67 Base address:0x2000

lo:0 Link encap:Local Loopback

inet addr:192.168.0.100 Mask:255.255.255.255

UP LOOPBACK RUNNING MTU:16436 Metric:1

设置real服务器的arp选项保证在进行arp解析的时候只有director的vip进行响应

[root@zzu ~]# echo "net.ipv4.conf.all.arp_announce = 2" >>/etc/sysctl.conf

[root@zzu ~]# echo "net.ipv4.conf.lo.arp_announce = 2" >>/etc/sysctl.conf

[root@zzu ~]# echo "net.ipv4.conf.all.arp_ignore = 1" >>/etc/sysctl.conf

[root@zzu ~]# echo "net.ipv4.conf.lo.arp_ignore = 1" >>/etc/sysctl.conf

[root@zzu ~]# route add -host 192.168.0.100 dev lo:0

设置一条特殊的路由,保证在恢复客户端使使用的是vip的地址

[root@zzu ~]# route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

192.168.0.100 0.0.0.0 255.255.255.255 UH 0 0 0 lo

192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0

[root@zzu Server]# rpm -ivh httpd-2.2.3-31.el5.i386.rpm

[root@zzu Server]# cd /var/www/html/

[root@zzu html]# vim index.html

Web2

[root@zzu html]# links http://192.168.0.102

三:director服务器的搭建

[root@zzu ~]# yum install ipvsadm*

四:lvs-dr模型下rr的测试

[root@zzu ~]# ipvsadm -A 192.168.0.100:80 -s rr

unexpected argument 192.168.0.100:80

[root@zzu ~]# ipvsadm -A -t 192.168.0.100:80 -s rr

[root@zzu ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.101 -g

[root@zzu ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.102 -g

[root@zzu ~]# ipvsdm –ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.0.100:80 rr

-> 192.168.0.102:80 Route 1 0 0

-> 192.168.0.101:80 Route 1 0 0

clip_image002

clip_image004

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.0.100:80 rr

-> 192.168.0.102:80 Route 1 0 6

-> 192.168.0.101:80 Route 1 0 6

五:lvs-dr模型下rr的测试 (ppc)

[root@zzu ~]# ipvsadm -A -t 192.168.0.100:80 -s rr -p 300

[root@zzu ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.102 -g

[root@zzu ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.101 –g

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.0.100:80 rr persistent 300

-> 192.168.0.101:80 Route 1 0 0

-> 192.168.0.102:80 Route 1 0 0

clip_image006

clip_image008

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.0.100:80 rr persistent 300

-> 192.168.0.101:80 Route 1 0 5

-> 192.168.0.102:80 Route 1 0 0

[root@zzu ~]# ipvsadm -A -t 192.168.0.100:22 -s rr -p 300

[root@zzu ~]# ipvsadm -a -t 192.168.0.100:22 -r 192.168.0.101 -g

[root@zzu ~]# ipvsadm -a -t 192.168.0.100:22 -r 192.168.0.102 -g

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.0.100:80 rr persistent 300

-> 192.168.0.101:80 Route 1 0 5

-> 192.168.0.102:80 Route 1 0 0

TCP 192.168.0.100:22 rr persistent 300

-> 192.168.0.102:22 Route 1 0 0

-> 192.168.0.101:22 Route 1 0 0

[root@zzu ~]#

clip_image010

clip_image012

clip_image014

clip_image016

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.0.100:80 rr persistent 300

-> 192.168.0.101:80 Route 1 0 5

-> 192.168.0.102:80 Route 1 0 0

TCP 192.168.0.100:22 rr persistent 300

-> 192.168.0.102:22 Route 1 0 10

-> 192.168.0.101:22 Route 1 0 0

六:lvs-dr模型下rr的测试 (pcc)

[root@zzu ~]# ipvsadm -A -t 192.168.0.100:0 -s rr -p 300

[root@zzu ~]# ipvsadm -a -t 192.168.0.100:0 -r 192.168.0.102 -g

[root@zzu ~]# ipvsadm -a -t 192.168.0.100:0 -r 192.168.0.101 -g

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.0.100:0 rr persistent 300

-> 192.168.0.101:0 Route 1 0 0

-> 192.168.0.102:0 Route 1 0 0

clip_image018

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.0.100:0 rr persistent 300

-> 192.168.0.101:0 Route 1 0 9

-> 192.168.0.102:0 Route 1 0 0

clip_image020

clip_image022

七:lvs-dr模型下带防火墙标记的持续连接(80和443)

将http和https打上标签

1:搭建https服务器web1

[root@zzu ~]# yum install openssl*

[root@zzu ~]# cd /etc/pki/

[root@zzu pki]# ll

drwx------ 3 root root 4096 2012-02-08 CA

drwxr-xr-x 2 root root 4096 2012-02-08 nssdb

drwxr-xr-x 2 root root 4096 2012-02-08 rpm-gpg

drwxr-xr-x 5 root root 4096 2012-02-08 tls

[root@zzu pki]# vim tls/openssl.cnf

45 dir = /etc/pki/CA

88 countryName = optional

89 stateOrProvinceName = optional

90 organizationName = optional

136 countryName_default = CN

141 stateOrProvinceName_default = beijing

144 localityName_default = Beijing

2:创建3个目录和两个文件

[root@zzu pki]# cd CA

[root@zzu CA]# mkdir certs newcerts crl

[root@zzu CA]# touch index.txt serial

[root@zzu CA]# echo "01" >>serial

[root@zzu CA]# openssl genrsa 1024 >private/cakey.pem

Generating RSA private key, 1024 bit long modulus

..............++++++

.....................................................................++++++

e is 65537 (0x10001)

[root@zzu CA]# openssl req -new -key private/cakey.pem -days 3650 -x509 -out cacert.pem

Country Name (2 letter code) [CN]:

State or Province Name (full name) [beijing]:

Locality Name (eg, city) [beijing]:

Organization Name (eg, company) [My Company Ltd]:qinghua

Organizational Unit Name (eg, section) []:qinghua

Common Name (eg, your name or your server's hostname) []:www.qinghua.com

2:为http办法证书

[root@zzu ~]# mkdir -pv /etc/httpd/certs

[root@zzu ~]# cd /etc/httpd/certs/

[root@zzu certs]# openssl genrsa 1024 > httpd.key

[root@zzu certs]# openssl req -new -key httpd.key -out httpd.csr

Country Name (2 letter code) [CN]:

State or Province Name (full name) [beijing]:

Locality Name (eg, city) [beijing]:

Organization Name (eg, company) [My Company Ltd]:bjdx

Organizational Unit Name (eg, section) []:sec

Common Name (eg, your name or your server's hostname) []:www.bj.com

[root@zzu certs]# openssl ca -in httpd.csr -out httpd.cert

Using configuration from /etc/pki/tls/openssl.cnf

Check that the request matches the signature

Signature ok

Certificate Details:

Serial Number: 1 (0x1)

Validity

Not Before: Feb 7 13:28:38 2012 GMT

Not After : Feb 6 13:28:38 2013 GMT

Subject:

countryName = CN

stateOrProvinceName = beijing

organizationName = bjdx

organizationalUnitName = sec

commonName = www.bj.com

3:绑紧证书文件

[root@zzu Server]# rpm -ivh distcache-1.4.5-14.1.i386.rpm

Preparing... ########################################### [100%]

1:distcache ########################################### [100%]

[root@zzu Server]# rpm -ivh mod_ssl-2.2.3-31.el5.i386.rpm

Preparing... ########################################### [100%]

1:mod_ssl ########################################### [100%]

[root@zzu ~]# cd /etc/httpd/certs/

[root@zzu certs]# cp /etc/pki/CA/cacert.pem ./

[root@zzu certs]# ll

-rw-r--r-- 1 root root 1168 02-07 21:34 cacert.pem

-rw-r--r-- 1 root root 0 02-07 21:28 httpd.cert

-rw-r--r-- 1 root root 643 02-07 21:27 httpd.csr

-rw-r--r-- 1 root root 887 02-07 21:26 httpd.key

[root@zzu ~]# vim /etc/httpd/conf.d/ssl.conf

112 SSLCertificateFile /etc/http/certs/httpd.cert

119 SSLCertificateKeyFile /etc/http/certs/httpd.key

128 SSLCertificateChainFile /etc/http/certs/cacert.pem

[root@zzu certs]# service httpd restart 重新启动www服务

Stopping httpd: [FAILED]

Starting httpd: [ OK ]

4.搭建https 服务器Web2

[root@server2 ~]# mkdir -pv /etc/httpd/certs

mkdir: created directory `/etc/httpd/certs'

[root@server2 ~]# cd /etc/httpd/certs

[root@server2 certs]# ll

total 0

[root@server2 certs]# scp 192.168.0.101:/etc/httpd/certs/* ./

The authenticity of host '192.168.0.101 (192.168.0.101)' can't be established.

RSA key fingerprint is 91:71:d8:d9:f2:63:a6:78:2f:0c:1e:e8:32:aa:55:3c.

Are you sure you want to continue connecting (yes/no)? y

Please type 'yes' or 'no': yes

Warning: Permanently added '192.168.0.101' (RSA) to the list of known hosts.

root@192.168.0.101's password:

cacert.pem 100% 1168 1.1KB/s 00:00

httpd.cert 100% 3082 3.0KB/s 00:00

httpd.csr 100% 643 0.6KB/s 00:00

httpd.key 100% 887 0.9KB/s 00:00

[root@server2 certs]# ll

-rw-r--r-- 1 root root 1168 Apr 30 17:33 cacert.pem

-rw-r--r-- 1 root root 3082 Apr 30 17:33 httpd.cert

-rw-r--r-- 1 root root 643 Apr 30 17:33 httpd.csr

-rw-r--r-- 1 root root 887 Apr 30 17:33 httpd.key

[root@server2 Server]# rpm -ivh distcache-1.4.5-14.1.i386.rpm

[root@server2 Server]# rpm -ivh mod_ssl-2.2.3-31.el5.i386.rpm

[root@server2~]#scp192.168.0.101:/etc/httpd/conf.d/ssl.conf /etc/httpd/conf.d/ssl.conf

root@192.168.0.101's password:

ssl.conf 100% 9655 9.4KB/s 00:00

[root@server2 ~]# service httpd restart

Stopping httpd: [ OK ]

Starting httpd: httpd: apr_sockaddr_info_get() failed for server2

httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName

[ OK ]

5:设置director服务器

[root@zzu ~]# iptables -t mangle -A PREROUTING -d 192.168.0.100 -p tcp --dport 80 -j MARK --set-mark 1

[root@zzu ~]# iptables -t mangle -A PREROUTING -d 192.168.0.100 -p tcp --dport 443 -j MARK --set-mark 1

[root@zzu ~]# iptables -t mangle -L

Chain PREROUTING (policy ACCEPT)

target prot opt source destination

MARK tcp -- anywhere 192.168.0.100 tcp dpt:http MARK set 0x1

MARK tcp -- anywhere 192.168.0.100 tcp dpt:https MARK set 0x1

[root@zzu ~]# ipvsadm -A -f 1 -s rr -p 1800

[root@zzu ~]# ipvsadm -a -f 1 -r 192.168.0.101 -g

[root@zzu ~]# ipvsadm -a -f 1 -r 192.168.0.102 -g

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

FWM 1 rr persistent 1800

-> 192.168.0.102:0 Route 1 0 0

-> 192.168.0.101:0 Route 1 0 0

访问测试:

clip_image024

clip_image026

clip_image028

clip_image030

八:lvs-dr模型下带防火墙标记的持续连接(ftp的被动方式)

ftp1 服务器

[root@zzu Server]# rpm -ivh vsftpd-2.0.5-16.el5.i386.rpm

Preparing... ########################################### [100%]

1:vsftpd ########################################### [100%]

[root@zzu ~]# cd /var/ftp/

[root@zzu ftp]# mkdir ftp1

[root@zzu ftp]# ll

total 8

drwxr-xr-x 2 root root 4096 Feb 7 22:27 ftp1

drwxr-xr-x 3 root root 4096 Feb 7 22:26 pub

[root@zzu ~]# vim /etc/vsftpd/vsftpd.conf

12 pasv_min_port=10000

13 pasv_max_port=20000

14 pasv_enable=YES

[root@zzu ftp]# service vsftpd restart

Shutting down vsftpd: [ OK ]

Starting vsftpd for vsftpd: [ OK ]

ftp2服务器同一

director服务器的设置

[root@zzu ~]# iptables -t mangle -A PREROUTING -p tcp -d 192.168.0.100/32 --dport 10000:20000 -j MARK --set-mark 21

[root@zzu ~]# iptables -t mangle -A PREROUTING -p tcp -d 192.168.0.100/32 --dport 21 -j MARK --set-mark 21

[root@zzu ~]# iptables -t mangle -L

Chain PREROUTING (policy ACCEPT)

target prot opt source destination

MARK tcp -- anywhere 192.168.0.100 tcp dpts:ndmp:dnp MARK set 0x15

MARK tcp -- anywhere 192.168.0.100 tcp dpt:ftp MARK set 0x15

[root@zzu ~]# ipvsadm -A -f 21 -s rr -p 1800

[root@zzu ~]# ipvsadm -a -f 21 -r 192.168.0.101 -g

[root@zzu ~]# ipvsadm -a -f 21 -r 192.168.0.102 -g

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

FWM 21 rr persistent 1800

-> 192.168.0.102:0 Route 1 0 0

-> 192.168.0.101:0 Route 1 0 0

clip_image032

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

FWM 21 rr persistent 1800

-> 192.168.0.102:0 Route 1 0 8

-> 192.168.0.101:0 Route 1 0 0

[root@zzu ~]# ipvsadm –lcn 我们在director上查看链接的状态

IPVS connection entries

pro expire state source virtual destination

TCP 00:19 FIN_WAIT 192.168.0.5:1309 192.168.0.100:21 192.168.0.102:21

TCP 00:19 FIN_WAIT 192.168.0.5:1310 192.168.0.100:10499 192.168.0.102:10499

TCP 00:14 FIN_WAIT 192.168.0.5:1306 192.168.0.100:14859 192.168.0.102:14859

TCP 00:14 FIN_WAIT 192.168.0.5:1305 192.168.0.100:21 192.168.0.102:21

IP 28:19 ERR! 192.168.0.5:0 0.0.0.21:0 192.168.0.102:0

欢迎加入郑州阳仔的网络工程师自由交流群--132444800(请注明自己的身份,就说是51cto的博友)