Start:

nginx can be used as a reverse proxy to load balance HTTP requests among back-end servers.

(Pardon!!!You do not know english!!!You are kidding!!!)

OK,Go on.

1.Network topology

网站架构1

                     photograph 1.1 Network architecture 1

 

网站架构2

                   photograph 1.2 Network architecture 2

两种架构最主要区别在数据库方面,架构2把DB server 独立出来,具有很好的可扩展性,但是代价较高;架构1的实现较为简单,代价较小,且已经可以满足相当高的需求了,此处为了以后做HA、双机热备、mysql主从复制等,采用架构1来实现,对于架构2后续也会做实现。

2.Environment

    2.1 Key points

          External network switch 外网交换机

          Internal network switch  内网交换机

          Proxy server  代理服务器    

          Web server 1 Web服务器1

          Web server 2 Web服务器2

    2.2 Explanations

          网络拓扑图中虚的外网网线表示可有可无,建议部署环境时保留,待稳定运行后撤下。另因为使用了***,故此处外网地址为192.168.80.X 内网地址为192.168.1.X。

          Proxy server :

          Name:ubuntu

          OS: ubuntu10.10 x86

          NIC1: eth0 with public IP 192.168.80.8

          NIC2: eth1 with internal IP 192.168.1.8

          Runs nginx listening on 192.168.80.8:80

          Web server 1:

          Name:ubuntu2

          OS: ubuntu10.10 x86

          NIC1: eth0 with public IP 192.168.80.9(will be disabled after this deployment)

          NIC2: eth1 with internal IP 192.168.1.9

          Runs any http server ( Eg: apache2 ) listening on 192.168.1.9:80

          Runs mysql server listening on 192.168.1.9:3306

          Web server 2:

          Name:ubuntu3

          OS: ubuntu10.10 x86

          NIC1: eth0 with public IP 192.168.80.10 ( will be disabled after this deployment )

          NIC2: eth1 with internal IP 192.168.1.10

          Runs any http server ( Eg: apache2 ) listening on 192.168.1.10:80

          Runs mysql server listening on 192.168.1.10:3306

3.Configuration

    3.1 Install nginx on the Proxy server :

          You can use any way to install nginx. Here we will use the most simple method to install nginx.( Can view http://nginx.org/en/download.html )

    Configure network address

         edit /etc/network/interfaces like

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
        address 192.168.80.8
        netmask 255.255.255.0
        network 192.168.80.0
        broadcast 192.168.80.255
        gateway 192.168.80.1
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 210.22.70.3

auto eth1
iface eth1 inet static
        address 192.168.1.8
        netmask 255.255.255.0

Or run the following commands:

     root@ubuntu:~# ifconfig eth0 192.168.80.8 netmask 255.255.255.0 up

     root@ubuntu:~# ifconfig eth1 192.168.1.8 netmask 255.255.255.0 up

     root@ubuntu:~# route add default gw 192.168.80.1

     root@ubuntu:~# sed -i '1 i\nameserver 210.22.70.3' /etc/resolv.conf

     root@ubuntu:~# sed -i '1 i\nameserver 210.22.84.3' /etc/resolv.conf

   Install nginx. For ubuntu 10.10:

root@ubuntu:~# wget http://nginx.org/keys/nginx_signing.key

root@ubuntu:~# apt-key add nginx_signing.key

root@ubuntu:~# vi /etc/apt/sources.list

     Append the following to the end of the /etc/apt/sources.list file

    deb http://nginx.org/packages/ubuntu/ lucid nginx

    deb-src http://nginx.org/packages/ubuntu/ lucid nginx

     Then run the following commands:

    root@ubuntu:~# apt-get update

    root@ubuntu:~# apt-get install nginx

     Installation directory

root@ubuntu:~# find / -name nginx

/usr/sbin/nginx

/usr/share/doc/nginx

/usr/share/nginx

/var/log/nginx

/var/cache/nginx

/var/lib/update-rc.d/nginx

/etc/nginx

/etc/logrotate.d/nginx

/etc/init.d/nginx

/etc/default/nginx

root@ubuntu:~#

    Install System-V style init script links(add system startup item)

root@ubuntu:~# update-rc.d nginx defaults

3.2 Install apache2 on the Web server 1 and 2:(You must run the same commands on ubuntu1 and ubuntu2.)

            You can use any way to install LAMP. Here we will use the most simple method to install LAMP.( hahaha...)

      Configure network address

            Web 1 eth0:192.168.80.9 eth1:192.168.1.9

            Web 2 eth0:192.168.80.10 eth1:192.168.1.10

            DNS:210.22.70.3/210.22.84.3

      Install LAMP.For ubuntu 10.10 run the following commands.(You must run the same commands on the Web server 1 and 2.)

     root@ubuntu:~# apt-get update

root@ubuntu:~# apt-get install apache2 mysql-server mysql-client php5 php5-gd php5-mysql libapache2-mod-php5 phpmyadmin

     Install System-V style init script links(add system startup item) (so…) 

         root@ubuntu:~# update-rc.d apache2 defaults

         root@ubuntu:~# update-rc.d mysql defaults

3.3 Configure ubuntu1 and ubuntu2(192.168.1.9 and 192.168.1.10)

      Change the apache2 home directory permissions

        root@ubuntu:~# chmod 777 /var/www

      Enable mod_rewrite module

        root@ubuntu:~# a2enmod rewrite

      Configure the Apache support .htm .html .Php

        root@ubuntu:~# vim /etc/apache2/apache2.conf

      Append the following to the end of the /etc/apache2/apache2.conf file

            AddType application/x-httpd-php .php .htm .html

   Set password for mysql

            Method 1: Use SET PASSWORD command
     root@ubuntu:~# mysql -u root
     mysql> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('newpass');

            Method 2:Use mysqladmin command 
      mysqladmin -u root password "newpass" 
         If the root has been set password, using the following method

      mysqladmin -u root password oldpass "newpass"

            Method 3: Use UPDATE command edit user table

      root@ubuntu:~# mysql -u root

          mysql> use mysql;
      mysql> UPDATE user SET Password = PASSWORD('newpass') WHERE user = 'root';
      mysql> FLUSH PRIVILEGES;

      Test mysql

root@ubuntu:~# touch /var/www/mysql_test.php

root@ubuntu:~# vim /var/www/mysql_test.php

Enter the following

<?php

$link = mysql_connect("localhost","root","longwind");

if (!$link)

{

die('Could not connect: ' . mysql_error());

}

else echo "Mysql success!";

mysql_close($link);

?>

     On the client, open a brower, input http://192.168.80.9/mysql_test.php , if you see “Mysql success!” Congratulations! Mysql is OK!

       Web server 1:

root@ubuntu:~# vim /etc/apache2/ports.conf

Modify NameVirtualHost *:80  => NameVirtualHost 192.168.1.9:80

root@ubuntu:~# vim /var/www/index.html

        Edit this file like

<html><body><h1>It works!</h1>
<p>This is the default web page for this server(192.168.1.9).</p>
<p>The web server software is running but no content has been added, yet.</p>
</body></html>

Then
root@ubuntu:~# kill –HUP `cat /var/run/apache2.pid`

Web server 2:

root@ubuntu:~# vim /etc/apache2/ports.conf

Modify NameVirtualHost *:80 => NameVirtualHost 192.168.1.10:80

root@ubuntu:~# vim /var/www/index.html

Edit this file like

<html><body><h1>It works!</h1>
<p>This is the default web page for this server(192.168.1.10).</p>
<p>The web server software is running but no content has been added, yet.</p>
</body></html>

Then
root@ubuntu:~# kill –HUP `cat /var/run/apache2.pid`

3.4 Configure ubuntu(192.168.1.8)

       Edit /etc/nginx/conf.d/default.conf

root@ubuntu:~# cp /etc/nginx/conf.d/default.conf /etc/nginx/conf.d/default.conf.bak

root@ubuntu:~# vim /etc/nginx/conf.d/default.conf

      Type :1,$d to empty the contents of the file. Then enter the following.

upstream   loadbalance {
      server 192.168.1.9 weight=10 max_fails=3 fail_timeout=30s; # Reverse proxy to  Web server 1 
      server 192.168.1.10 weight=10 max_fails=3 fail_timeout=30s; # Reverse proxy to  Web server 2 
      server 192.168.1.11 backup;
      server 192.168.1.12 down;
      #ip_hash;
}
server
{
         listen 192.168.80.8:80;     # Listen on the external interface

         server_name 192.168.80.8;   # The server name , also can www.xxx.com

         location / {
                        proxy_pass       
http://loadbalance; # Load balance the URL location "/" to the upstream loadbalance

                        proxy_set_header   Host             $host;

                        proxy_set_header   X-Real-IP        $remote_addr;

                        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for; # Get the real client ip address
}

         log_format 192.168.80.8  '$remote_addr - $remote_user [$time_local] $request '
                         '"$status" $body_bytes_sent "$http_referer" '
                         '"$http_user_agent" "$http_x_forwarded_for"';
         access_log  /var/log/nginx/access.log  192.168.80.8;                    # access log format

         location /nginx_status {            # nginx status monitor
                        stub_status on;
                        access_log off;          

                        allow 10.10.10.3;
                        deny all;}           # only allow 10.10.10.3 monitor nginx
}

Then
root@ubuntu:~# kill –HUP `cat /var/run/apache2.pid`

3.5 Additional content

         HttpUpstreamModule      

                  nginx 的 upstream目前支持 4 种方式的分配

1)、轮询(默认)
每个请求按时间顺序逐一分配到不同的后端服务器,如果后端服务器down掉,能自动剔除。
2)、weight
指定轮询几率,weight和访问比率成正比,用于后端服务器性能不均的情况。
2)、ip_hash
每个请求按访问ip的hash结果分配,这样每个访客固定访问一个后端服务器,可以解决session的问题。
3)、fair(第三方)
按后端服务器的响应时间来分配请求,响应时间短的优先分配。
4)、url_hash(第三方)

                 简要说明upstream模块中的几个状态:

weight :默认为1.weight越大,负载的权重就越大

max_fails :允许请求失败的次数默认为1.当超过最大次数时,返回proxy_next_upstream 模块定义的错误

fail_timeout:max_fails 次失败后,暂停的时间

backup: 其它所有的非backup机器down或者忙的时候,请求backup机器。所以这台机器压力会最轻

down :表示当前的server暂时不参与负载 
(下面中文内容参考自
http://www.cnblogs.com/xiaogangqq123/archive/2011/03/04/1971002.html

配置负载均衡比较简单,但是最关键的一个问题是怎么实现多台服务器之间session的共享

下面有几种方法(以下内容来源于网络,第四种方法没有实践.)

1) 不使用session,换作cookie

能把session改成cookie,就能避开session的一些弊端,在从前看的一本J2EE的书上,也指明在集群系统中不能用session,否则惹出祸端来就不好办。如果系统不复杂,就优先考虑能否将session去掉,改动起来非常麻烦的话,再用下面的办法。

2) 应用服务器自行实现共享

asp.net可以用数据库或memcached来保存session,从而在asp.net本身建立了一个session集群,用这样的方式可以令 session保证稳定,即使某个节点有故障,session也不会丢失,适用于较为严格但请求量不高的场合。但是它的效率是不会很高的,不适用于对效率 要求高的场合。

以上两个办法都跟nginx没什么关系,下面来说说用nginx该如何处理:

3) ip_hash

nginx中的ip_hash技术能够将某个ip的请求定向到同一台后端,这样一来这个ip下的某个客户端和某个后端就能建立起稳固的session,ip_hash是在upstream配置中定义的:

upstream backend {
server 127.0.0.1:8080 ;
server 127.0.0.1:9090 ;
ip_hash;
}

ip_hash是容易理解的,但是因为仅仅能用ip这个因子来分配后端,因此ip_hash是有缺陷的,不能在一些情况下使用:

1/ nginx不是最前端的服务器。ip_hash要求nginx一定是最前端的服务器,否则nginx得不到正确ip,就不能根据ip作hash。譬如使用的是squid为最前端,那么nginx取ip时只能得到squid的服务器ip地址,用这个地址来作分流是肯定错乱的。

2/ nginx的后端还有其它方式的负载均衡。假如nginx后端又有其它负载均衡,将请求又通过另外的方式分流了,那么某个客户端的请求肯定不能定位到同一台session应用服务器上。这么算起来,nginx后端只能直接指向应用服务器,或者再搭一个squid,然后指向应用服务器。最好的办法是用location作一次分流,将需要session的部分请求通过ip_hash分流,剩下的走其它后端去。

4) upstream_hash

为了解决ip_hash的一些问题,可以使用upstream_hash这个第三方模块,这个模块多数情况下是用作url_hash的,但是并不妨碍将它用来做session共享:

假如前端是squid,他会将ip加入x_forwarded_for这个http_header里,用upstream_hash可以用这个头做因子,将请求定向到指定的后端:

可见这篇文档:http://www.sudone.com/nginx/nginx_url_hash.html

在文档中是使用$request_uri做因子,稍微改一下:

hash $http_x_forwarded_for;

这样就改成了利用x_forwarded_for这个头作因子,在nginx新版本中可支持读取cookie值,所以也可以改成:

hash $cookie_jsessionid;

假如在php中配置的session为无cookie方式,配合nginx自己的一个userid_module模块就可以用nginx自发一个cookie,可参见userid模块的英文文档:
http://wiki.nginx.org/NginxHttpUserIdModule
另可用姚伟斌编写的模块upstream_jvm_route:
http://code.google.com/p/nginx-upstream-jvm-route/

4.The end

        When all configured(ubuntu,nginx, ubuntu1, ubuntu2, apache2, mysql).Now we can test it.Go!

        Disable external NIC on ubuntu1 and ubuntu2.

        Restart Web server on ubuntu1 and ubuntu2.

        On the client (*** address:10.10.10.3).Open a brower.Type http://192.168.80.8  now ,you will see the following picture.

QQ截图20120411103424

        Constantly refresh your browser,you can also see the following picture

QQ截图20120411103433

       Type http://192.168.80.8/nginx_status  you will see the following picture.

QQ截图20120422172446

       If you input http://192.168.80.8/nginx_status on the other machines, you will see

QQ截图20120411094812

 

5.Conclusion

      到此处,已经基本实现了用nginx负载平衡后端服务器的HTTP请求。但是很明显还有很多的问题,比如session、数据库同步、访问日志、时间同步等等等等,后续还会做相关优化设置,比如squid结合,双线处理,备份容灾,安全防护,web cache等方面。

     关于这方面的内容请关注本博客,会持续连载更新。