7.0下通过CARP+PF+relayd+PFSYNC实现健康检查等功能群集

Submitted by sysv on 2008, August 19, 3:32 PM. Cluster

在<<FreeBSD 7.0下通过CARP PF PFSYNC实现服务器群集笔记>>基础上，通过relayd来实现以下目标:

1. 对服务器池的健康检查；
2. 实现3层和7层的load balance;


一、安装relayd
1. 安装前需要确认内核配置文档中加入以下几个参数:

  1. options      INET6            # IPv6 communications protocols (relayd默认对ipv6进行了支持)  
  2. device       tun      # Packet tunnel.(relayd在实现上需要利用到tunnel, 必须加上)  
  3.   
  4. ### pf support ###  
  5. device carp  
  6. device pf  
  7. device pflog  
  8. device pfsync  
  9.   
  10. options          ALTQ  
  11. options          ALTQ_CBQ  
  12. options          ALTQ_RED  
  13. options          ALTQ_RIO  
  14. options          ALTQ_HFSC  
  15. options          ALTQ_PRIQ  
  16. options          ALTQ_NOPCC  

如重新配置了以上内核参数请重新编译内核，并重新升效

2. 开始安装relayd. 除自己DIY源代码, Makefile基本不必要改了。
# cd /usr/ports/net/relayd
# make install clean

二、配置rc.conf(加入的相关项)
# vim /etc/rc.conf

代码
  1. gateway_enable="YES"  
  2. defaultrouter="192.168.1.1"  
  3. hostname="master.cluster.org"  
  4. cloned_interfaces="carp0 carp1"  
  5.   
  6. # External Public Interface (for the secondary firewall use a different public ip.)  
  7. ifconfig_em0="inet 192.168.1.52 netmask 255.255.255.0"  
  8. # External Public Carp Interface  
  9. #ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24"  
  10. ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24 advskew 10"  
  11.   
  12. # Internal Interface (for the secondary firewall change the ip address to 192.168.10.11)  
  13. ifconfig_em1="inet 192.168.10.10 netmask 255.255.255.0"  
  14. # Internal Carp Interface
    # vhid应为唯一值，不可与carp0相同。否则会出问题
  15. ifconfig_carp1="vhid 10 pass 22222 192.168.10.100/24 advskew 10"  
  16.   
  17. # Heartbeat Interface (for the secondary firewall, change the ip address to 10.10.10.251)  
  18. ifconfig_vr0="10.10.10.250 netmask 255.255.255.0"  
  19.   
  20. # PFSync Interface  
  21. #ifconfig_pfsync0="up syncdev vr0"  
  22.   
  23. pf_enable="YES"  
  24. pf_rules="/etc/pf.conf"  
  25. pf_flags=""  
  26. pflog_enable="YES"  
  27. pflog_logfile="/var/log/pflog"  
  28. pflog_flags=""  
  29.   
  30. pfsync_enable="YES"      # Expose pf state to other hosts for syncing  
  31. pfsync_syncdev="vr0"     # Interface for pfsync to work through  
  32. pfsync_syncpeer=""       # IP address of pfsync peer host  
  33.   
  34. relayd_enable="YES"  
  35. relayd_flags=""  



三、配置sysctl.conf(加入的相关项)
# vim /etc/sysctl.conf

代码
  1. net.inet.carp.preempt=1  
  2.   
  3. net.inet.tcp.blackhole=2  
  4. net.inet.udp.blackhole=1  
  5. net.inet.tcp.sendspace=65536  
  6. net.inet.tcp.recvspace=65536  



四、配置pf.conf

# vim /etc/pf.conf

代码
  1. ################################################################################  
  2. # Macro and lists  
  3. ################################################################################  
  4. lop_if = "lo0"  
  5. ext_if = "em0"  
  6. int_if = "em1"  
  7. sync_if= "vr0"  
  8. ext_carp = "carp0"  
  9.   
  10. icmp_types = "echoreq"  
  11.   
  12. web_ports = "{ 80, 443 }"  
  13.   
  14. table <mng_subnet> persist { 222.222.222.0/24, 221.222.212.0/24 }  
  15.   
  16. table <srv_pool> persist \  
  17. { \  
  18. 192.168.10.20, \  
  19. 192.168.10.21, \  
  20. 192.168.10.22, \  
  21. 192.168.10.23, \  
  22. 192.168.10.24, \  
  23. 192.168.10.30, \  
  24. 192.168.10.31, \  
  25. 192.168.10.32, \  
  26. 192.168.10.33, \  
  27. 192.168.10.34 \  
  28. }  
  29.   
  30. ################################################################################  
  31. # Options, scrub  
  32. ################################################################################  
  33. set block-policy drop  
  34. set limit { states 10000, src-nodes 100000, frags 10000 }  
  35. set skip on $lop_if  
  36.   
  37. scrub in on $ext_if  
  38.   
  39. ################################################################################  
  40. # NAT  
  41. ################################################################################  
  42. nat on $ext_if from $int_if:network to any -> $ext_if  
  43.   
  44. ################################################################################  
  45. # Redirection  
  46. ################################################################################  
  47. #rdr on $ext_if proto tcp from any to $ext_carp port $web_ports -> <srv_pool> round-robin sticky-address  
  48. rdr-anchor "relayd/*"  
  49.   
  50.   
  51. ################################################################################  
  52. # Filtering Rules  
  53. ################################################################################  
  54. block in all  
  55. block out all  
  56.   
  57. pass quick on $lop_if all  
  58. #pass quick on $int_if all  
  59.   
  60. pass quick on { $sync_if } proto pfsync keep state (no-sync)  
  61. pass on { $ext_if, $int_if } proto carp keep state  
  62.   
  63. pass in quick inet proto icmp all icmp-type $icmp_types keep state  
  64.   
  65. # 这里做了对服务器池的访问策略，如果需要全网开放就将<mng_subnet>改成any即可  
  66. pass in quick on { $ext_if } proto tcp from <mng_subnet> to $ext_carp port $web_ports flags S/SA modulate state  
  67. pass out quick on { $int_if } proto tcp from <mng_subnet> to <srv_pool> port $web_ports flags S/SA modulate state  
  68. #还有这里  
  69. pass in quick on $ext_if inet from <mng_subnet> to any keep state  
  70.   
  71. pass in quick on $int_if inet from any to $int_if:network keep state  
  72. pass out quick on $int_if inet from $int_if:network to any keep state  
  73.   
  74. pass out quick on $ext_if proto tcp all flags S/SA modulate state  
  75. pass out quick on $ext_if proto { udp, icmp } all keep state  


五、配置relayd.conf

# vim /usr/etc/relayd.conf

PHP代码
  1. ##################  
  2. # MACROS          #  
  3. ##################  
  4. #www1="192.168.10.20, 192.168.10.21"  
  5. #www2="192.168.10.30, 192.168.10.31"  
  6.   
  7. ##### Global Options #####  
  8. interval 10  
  9. prefork 10  
  10.   
  11. ##################  
  12. # TABLES          #  
  13. ##################  
  14. #这里就是你的服务器池的IP了。  
  15. table <www_pool> { \  
  16. 192.168.10.20, \  
  17. 192.168.10.21, \  
  18. 192.168.10.22, \  
  19. 192.168.10.23, \  
  20. 192.168.10.24, \  
  21. 192.168.10.30, \  
  22. 192.168.10.31, \  
  23. 192.168.10.32, \  
  24. 192.168.10.33, \  
  25. 192.168.10.34 \  
  26. }  
  27.   
  28. table <fallback> { 127.0.0.1 }  
  29.   
  30. ##### REDIRECTIONS #####  
  31. redirect "www" {  
  32.      #注意这里的网口是ext_if。  
  33.      listen on 192.168.1.51 port 80 interface em0  
  34.      #forward to <srv_pool> mode roundrobin check http "/" code 200  
  35.      #forward to <srv_pool> port 80 mode roundrobin sticky-address check tcp  
  36.      #因为服务器池的应用是动态的，在重定向时我用了sticky-address来保证同个源请求到原目标服务器。并利用tcp对服务器池进行健康检查。  
  37.      tag REDIRECTED  
  38.      sticky-address  
  39.      forward to <www_pool> port 80 mode roundrobin timeout 300 check tcp  
  40.      forward to <fallback> check icmp  
  41. }  

六、启动服务
# /usr/local/etc/rc.d/relayd start
# pfctl -Fa -f /etc/pf.conf

测试, 通过relayctl可以看到服务器池的健康情况，不健康的就不会进行调度了。relayctl命令还有一些比较实用的用法，请自行man 了。

# relayctl show summary

PHP代码
  1. Id       Type             Name                             Avlblty Status  
  2. 0        redirect         www                                      active  
  3. 2        table            www_pool:80                              active (5 hosts up)  
  4. 20       host             192.168.10.20                    99.95%   up  
  5. 19       host             192.168.10.21                    99.95%   up  
  6. 18       host             192.168.10.22                    99.94%   up  
  7. 17       host             192.168.10.23                    99.95%   up  
  8. 16       host             192.168.10.24                    99.94%   up  
  9. 15       host             192.168.10.30                    74.62%   down  
  10. 14       host             192.168.10.31                    74.62%   down  
  11. 13       host             192.168.10.32                    74.62%   down  
  12. 12       host             192.168.10.33                    74.61%   down  
  13. 11       host             192.168.10.34                    74.61%   down  
  14. 3        table            fallback:80                              active (1 hosts up)  
  15. 21       host             127.0.0.1                        100.00% up  

七、服务器池的配置在<<FreeBSD 7.0下通过CARP PF PFSYNC实现服务器群集笔记>>上已讲过，配置是一样的。

八、LoadBalance 的master和slave的配置以<<FreeBSD 7.0下通过CARP PF PFSYNC实现服务器群集笔记>>上的方法来配置。

Tags: relayd, pf, carp, pfsync, loadbalance

出处 http://blog.sysv.cn/?action=showid=14