记录一下方便以后自己查看
1.环境准备
lb-01:192.168.33.135 nginx-lb centos7 rs-01:192.168.33.131 apache-web centos6.x rs-02:192.168.33.132 nginx-web centos6.x
2.环境安装
lb-01 安装nginx,配置nginx源
# cat /etc/yum.repos.d/nginx.repo [nginx] name=nginx repo baseurl=http://nginx.org/packages/centos/7/$basearch/ gpgcheck=0 enabled=1
yum安装
[root@lb-01 ~]# yum install nginx -y
启动
[root@lb-01 ~]# systemctl start nginx.service
测试
[root@lb-01 ~]# curl -I localhost HTTP/1.1 200 OK Server: nginx/1.10.1 Date: Tue, 28 Jun 2016 21:17:48 GMT Content-Type: text/html Content-Length: 612 Last-Modified: Tue, 31 May 2016 14:09:55 GMT Connection: keep-alive ETag: "574d9b33-264" Accept-Ranges: bytes
3.rs节点安装web服务
其他rs机器rs-01安装apache使用默认yum安装即可,rs-02安装nginx
rs-01安装
# yum install httpd -y
rs-02安装
# yum install nginx -y
4.后端web节点访问测试结果
rs-01 web访问结果
[root@rs-01 ~]# cat /var/www/html/index.html <h1>this is rs-01 ip 192.168.33.131 </h1> [root@rs-01 ~]# curl localhost <h1>this is rs-01 ip 192.168.33.131 </h1>
rs-02 web访问结果
[root@rs-02 ~]# cat /usr/share/nginx/html/index.html <h1>this is rs-02 ip 192.168.33.132 </h1> [root@rs-02 ~]# curl localhost <h1>this is rs-02 ip 192.168.33.132 </h1>
4.lb负载均衡配置
[root@lb-01 ~]# cat /etc/nginx/conf.d/upstream.conf upstream blog { server 192.168.33.131:80 weight=3; server 192.168.33.132:80 weight=3; server 192.168.33.133:81 weight=3; } server { listen 80; server_name blog.zxl.com; location / { proxy_pass http://blog; } }
upsteam池中端口不写也没关系默认是80,如果是别的端口可以填写相关端口即可。upsteam必须在http{}
标签内,weight
权重。upstream.conf配置文件放在include /etc/nginx/conf.d/*.conf
目录下,官方地址http://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream
upstream模块常用的指令有:
ip_hash:基于客户端IP地址完成请求的分发,它可以保证来自于同一个客户端的请求始终被转发至同一个upstream服务器; keepalive:每个worker进程为发送到upstream服务器的连接所缓存的个数; least_conn:最少连接调度算法; server:定义一个upstream服务器的地址,还可包括一系列可选参数,如: weight:权重;默认1 max_fails:最大失败连接次数,失败连接的超时时长由fail_timeout指定;默认1,0则为禁止失败尝试 fail_timeout:等待请求的目标服务器发送响应的时长;默认10s backup:用于fallback的目的,所有服务均故障时才启动此服务器; down:手动标记其不再处理任何请求;
4.1绑定hosts文件
[root@lb-01 ~]# tail -n 1 /etc/hosts 192.168.33.135 blog.zxl.com
4.2检查lb-nginx
[root@lb-01 ~]# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@lb-01 ~]# nginx -s reload
4.3测试负载均衡
[root@lb-01 ~]# curl http://blog.zxl.com <h1>this is rs-02 ip 192.168.33.132 </h1> [root@lb-01 ~]# curl http://blog.zxl.com <h1>this is rs-01 ip 192.168.33.131 </h1> [root@lb-01 ~]# curl http://blog.zxl.com <h1>this is rs-02 ip 192.168.33.132 </h1> [root@lb-01 ~]# curl http://blog.zxl.com <h1>this is rs-01 ip 192.168.33.131 </h1>
注:为什么没有访问33.131:81服务,因为我没有这台机器假如有的话,没有81端口。nginx本身自带健康检查,有问题的话会自带剔除
4.4可以看到上面访问rr权重,假如某个后端服务停止,也不影响使用,假如吧rs-02服务关闭,然后再次访问
[root@lb-01 ~]# curl http://blog.zxl.com <h1>this is rs-01 ip 192.168.33.131 </h1> [root@lb-01 ~]# curl http://blog.zxl.com <h1>this is rs-01 ip 192.168.33.131 </h1> [root@lb-01 ~]# curl http://blog.zxl.com <h1>this is rs-01 ip 192.168.33.131 </h1>
注:可以看到访问一直都是后端rs-01服务
注意:如果upstream
中使用ip_hash
算法,那么不能使用weight和backup
参数
nginx核心模块 http://nginx.org/en/docs/http/ngx_http_core_module.html
4.5location匹配优先级
location [ = | ~ | ~* | ^~ ] uri { ... } location @name { ... } 功能:允许根据用户请求的URI来匹配指定的各location以进行访问配置;匹配到时,将被location块中的配置所处理;比如:http://www.zxl.com/p_w_picpaths/logo.gif =:精确匹配; ~:正则表达式模式匹配,匹配时区分字符大小写 ~*:正则表达式模式匹配,匹配时忽略字符大小写 ^~: URI前半部分匹配,不检查正则表达式
4.6http核心模块的内置变量:
$uri: 当前请求的uri,不带参数; $request_uri: 请求的uri,带完整参数; $host: http请求报文中host首部;如果请求中没有host首部,则以处理此请求的虚拟主机的主机名代替; $hostname: nginx服务运行在的主机的主机名; $remote_addr: 客户端IP $remote_port: 客户端Port $remote_user: 使用用户认证时客户端用户输入的用户名; $request_filename: 用户请求中的URI经过本地root或alias转换后映射的本地的文件路径; $request_method: 请求方法 $server_addr: 服务器地址 $server_name: 服务器名称 $server_port: 服务器端口 $server_protocol: 服务器向客户端发送响应时的协议,如http/1.1, http/1.0 $scheme: 在请求中使用scheme, 如https://www.magedu.com/中的https; $http_HEADER: 匹配请求报文中指定的HEADER,$http_host匹配请求报文中的host首部 $sent_http_HEADER: 匹配响应报文中指定的HEADER,例如$http_content_type匹配响应报文中的content-type首部; $document_root:当前请求映射到的root配置;
5.后端节点配置多个虚拟机
分别配置三个虚拟主机域名为 www.zxl.com bbs.zxl.com blog.zxl.com
5.1rs-01后端apache服务虚拟主机配置
[root@rs-01 conf]# tail -n 18 httpd.conf <VirtualHost *:80> DocumentRoot /var/www/www ServerName www.zxl.com ErrorLog logs/www.zxl.com-error_log CustomLog logs/www.zxl.com-access_log common </VirtualHost> <VirtualHost *:80> DocumentRoot /var/www/bbs ServerName bbs.zxl.com ErrorLog logs/bbs.zxl.com-error_log CustomLog logs/bbs.zxl.com-access_log common </VirtualHost> <VirtualHost *:80> DocumentRoot /var/www/blog ServerName blog.zxl.com ErrorLog logs/blog.zxl.com-error_log CustomLog logs/blog.zxl.com-access_log common </VirtualHost>
注:apache配置虚拟主机时把#NameVirtualHost :80注释需要打开*
创建虚拟主机站点目录
[root@rs-01 httpd]# mkdir /var/www/{www,bbs,blog} -pv mkdir: created directory `/var/www/www' mkdir: created directory `/var/www/bbs' mkdir: created directory `/var/www/blog'
rs-01节点配置hosts文件
[root@rs-01 ~]# tail -n 1 /etc/hosts 192.168.33.131 www.zxl.com 192.168.33.131 blog.zxl.com 192.168.33.131 bbs.zxl.com
站点文件内容
[root@rs-01 ~]# cat /var/www/www/index.html apache www 131 [root@rs-01 ~]# cat /var/www/bbs/index.html apache bbs 131 [root@rs-01 ~]# cat /var/www/blog/index.html apache blog 131
使用curl访问测试
[root@rs-01 ~]# curl www.zxl.com apache www 131 [root@rs-01 ~]# curl blog.zxl.com apache blog 131 [root@rs-01 ~]# curl bbs.zxl.com apache bbs 131
rs-02节点wwww虚拟主机配置文件如下
[root@rs-02 conf.d]# cat www.conf server { listen 80; server_name www.zxl.com; location / { index index.html index.htm; root /usr/share/nginx/www; } }
rs-02节点bbs虚拟主机配置文件如下
[root@rs-02 conf.d]# cat bbs.conf server { listen 80; server_name bbs.zxl.com; location / { index index.html index.htm; root /usr/share/nginx/bbs; } }
rs-02节点blog虚拟主机配置文件如下
[root@rs-02 conf.d]# cat blog.conf server { listen 80; server_name blog.zxl.com; location / { index index.html index.htm; root /usr/share/nginx/blog; } }
检查rs-02节点配置文件语法以及重新加载
[root@rs-02 conf.d]# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@rs-02 conf.d]# nginx -s reload
创建站点配置文件
[root@rs-02 conf.d]# mkdir /usr/share/nginx/{www,bbs,blog} -pv mkdir: created directory `/usr/share/nginx/www' mkdir: created directory `/usr/share/nginx/bbs' mkdir: created directory `/usr/share/nginx/blog'
创建站点内容
[root@rs-02 ~]# cat /usr/share/nginx/www/index.html nginx www 132 [root@rs-02 ~]# cat /usr/share/nginx/bbs/index.html nginx bbs 132 [root@rs-02 ~]# cat /usr/share/nginx/blog/index.html nginx blog 132
rs-02节点配置hosts文件
[root@rs-02 conf.d]# tail -n 1 /etc/hosts 192.168.33.132 www.zxl.com 192.168.33.132 blog.zxl.com 192.168.33.132 bbs.zxl.com
rs-02节点进行测试访问
[root@rs-02 ~]# curl www.zxl.com nginx www 132 [root@rs-02 ~]# curl bbs.zxl.com nginx bbs 132 [root@rs-02 ~]# curl blog.zxl.com nginx blog 132
6.lb负载均衡器上操作
配置hosts文件
[root@lb-01 ~]# tail -n 3 /etc/hosts 192.168.33.135 www.zxl.com 192.168.33.135 blog.zxl.com 192.168.33.135 bbs.zxl.com
访问blog.zxl.com
[root@lb-01 conf.d]# curl blog.zxl.com apache www 131 [root@lb-01 conf.d]# curl blog.zxl.com nginx blog 132
注:为什么会出现和预期的不一样呢?因为lb根本不知道该去找个节点,因为后端都是80端口。lb需要进行参数设置
proxy_set_header Host $host ;获取后端head的信息,代理那个后端主机,完整示例如下
[root@lb-01 conf.d]# cat upstream.conf upstream blog { server 192.168.33.131:80 weight=3; server 192.168.33.132:80 weight=3; } server { listen 80; server_name blog.zxl.com; location / { proxy_pass http://blog; proxy_set_header Host $host; } }
其实就是在location中添加了proxy_set_header Host $host;
检查lb上nginx语法以及重新加载并测试
[root@lb-01 conf.d]# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@lb-01 conf.d]# nginx -s reload
结果就是可以rr权重了
[root@lb-01 conf.d]# curl blog.zxl.com apache blog 131 [root@lb-01 conf.d]# curl blog.zxl.com nginx blog 132
bbs和www和blog类似,完整示例如下
[root@lb-01 conf.d]# cat upstream.conf upstream blog { server 192.168.33.131:80 weight=3; server 192.168.33.132:80 weight=3; } server { listen 80; server_name blog.zxl.com; location / { proxy_pass http://blog; proxy_set_header Host $host; } } server { listen 80; server_name bbs.zxl.com; location / { proxy_pass http://blog; proxy_set_header Host $host; } } server { listen 80; server_name www.zxl.com; location / { proxy_pass http://blog; proxy_set_header Host $host; } }
注:bbs和www以及blog使用相同的upstream池,因为后端节点都是一样的
检查语法以及测试
[root@lb-01 conf.d]# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@lb-01 conf.d]# nginx -s reload
可以看到每个站点都是rr轮询访问的
[root@lb-01 conf.d]# curl www.zxl.com apache www 131 [root@lb-01 conf.d]# curl www.zxl.com nginx www 132 [root@lb-01 conf.d]# curl bbs.zxl.com apache bbs 131 [root@lb-01 conf.d]# curl bbs.zxl.com nginx bbs 132
后端节点如何查看来自客户端访问的真实ip地址?看的话只能从日志看了,所以设置好后端节点的日志,apache上面定义虚拟主机的时候已经配置好了,下面配置nginx节点的日志,示例如下
rs-02后端节点bbs虚拟主机log日志
[root@rs-02 conf.d]# cat bbs.conf server { listen 80; server_name bbs.zxl.com; location / { index index.html index.htm; root /usr/share/nginx/bbs; access_log logs/bbs.access.log main; } }
rs-02后端节点blog虚拟主机log日志
[root@rs-02 conf.d]# cat blog.conf server { listen 80; server_name blog.zxl.com; location / { index index.html index.htm; root /usr/share/nginx/blog; access_log logs/blog.access.log main; } }
rs-02后端节点www虚拟主机log日志
[root@rs-02 conf.d]# cat www.conf server { listen 80; server_name www.zxl.com; location / { index index.html index.htm; root /usr/share/nginx/www; access_log logs/www.access.log main; } }
注:access__log添加字段是记录log日志的
创建rs-02节点虚拟主机记录log日志的目录
[root@rs-02 conf.d]# mkdir /usr/share/nginx/logs
检查语法以及重新加载
[root@rs-02 conf.d]# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@rs-02 conf.d]# nginx -s reload
7.客户端测试
在自己电脑上设置hosts文件绑定域名进行访问,然后查看后端节点日志看看是来自哪个ip地址?C:\Windows\System32\drivers\etc\hosts
192.168.33.135 www.zxl.com 192.168.33.135 blog.zxl.com 192.168.33.135 bbs.zxl.com
7.1浏览器中输入相应的域名即可,后端节点查看日志
rs-02节点日志
[root@rs-02 logs]# tail bbs.access.log 192.168.33.135 - - [27/Jun/2016:09:53:07 +0800] "GET / HTTP/1.0" 200 14 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0" "-" 192.168.33.135 - - [27/Jun/2016:09:53:10 +0800] "GET / HTTP/1.0" 200 14 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0" "-" [root@rs-02 logs]# tail blog.access.log 192.168.33.135 - - [27/Jun/2016:10:02:39 +0800] "GET / HTTP/1.0" 200 15 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0" "-" 192.168.33.135 - - [27/Jun/2016:10:02:56 +0800] "GET / HTTP/1.0" 304 0 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0" "-" [root@rs-02 logs]# tail www.access.log 192.168.33.135 - - [27/Jun/2016:10:03:33 +0800] "GET / HTTP/1.0" 200 14 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0" "-"
注:可以看到上面ip地址来自lb负载均衡器
rs-01节点日志
[root@rs-01 httpd]# tail www.zxl.com-access_log 192.168.33.135 - - [17/Jun/2016:07:22:34 +0800] "GET / HTTP/1.0" 200 15 [root@rs-01 httpd]# tail bbs.zxl.com-access_log 192.168.33.135 - - [17/Jun/2016:07:23:00 +0800] "GET / HTTP/1.0" 304 - [root@rs-01 httpd]# tail blog.zxl.com-access_log 192.168.33.135 - - [17/Jun/2016:07:22:53 +0800] "GET / HTTP/1.0" 200 16
注:上面ip地址来自lb负载均衡器
后端节点如何才能记录来自真实的ip地址访问呢?
其实后端节点nginx已经准备接收参数默认已经开启了,http{}
标签中,log_format main ....$http_x_forwarded_for
,$http_x_forwarded_for
记录客户端的真实ip地址
那么apache呢?修改示例如下
[root@rs-01 conf]# sed -n '498p' httpd.conf LogFormat "\"%{X-Forwarded-For}i\" %l %u %t \"%r\" %>s %b" common
修改配置文件后重新启动服务
[root@rs-01 conf]# /etc/init.d/httpd restart Stopping httpd: [ OK ] Starting httpd: httpd: apr_sockaddr_info_get() failed for rs-01 httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
8.lb负载均衡器更改nginx配置文件
完成示例如下
[root@lb-01 conf.d]# cat upstream.conf upstream blog { server 192.168.33.131:80 weight=3; server 192.168.33.132:80 weight=3; } server { listen 80; server_name blog.zxl.com; location / { proxy_pass http://blog; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } } server { listen 80; server_name bbs.zxl.com; location / { proxy_pass http://blog; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $remote_addr; } } server { listen 80; server_name www.zxl.com; location / { proxy_pass http://blog; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } }
注:上面server
段标签中的proxy_set_header X-Forwarded-For $remote_addr;
与proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
功能都是一样获取来自真实的客户端访问的ip地址
检查lb的nginx语法以及重新加载服务
[root@lb-01 conf.d]# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@lb-01 conf.d]# nginx -s reload
9.客户端访问以及查看日志是否来自真实的客户端的ip地址
查看rs-01的bbs站点日志访问
[root@rs-01 ~]# tail -n 2 /var/log/httpd/bbs.zxl.com-access_log "192.168.33.1" - - [17/Jun/2016:20:07:35 +0800] "GET / HTTP/1.0" 200 15 "192.168.33.1" - - [17/Jun/2016:20:07:35 +0800] "GET / HTTP/1.0" 200 15
注:可以看到已经记录了来自客户端访问的真实ip地址了
查看rs-02的bbs站点日志访问
[root@rs-02 ~]# tail -n 2 /usr/share/nginx/logs/bbs.access.log 192.168.33.135 - - [27/Jun/2016:22:57:04 +0800] "GET / HTTP/1.0" 200 14 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0" "192.168.33.1" 192.168.33.135 - - [27/Jun/2016:22:57:40 +0800] "GET / HTTP/1.0" 200 14 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0" "192.168.33.1
注:可以看到已经记录了来自客户端访问的真实ip地址了
其实lb负载均衡器配置文件中的location中可以把一些n多参数使用配置文件引用即可,就拿blog站点示例如下
[root@lb-01 conf.d]# cat upstream.conf upstream blog { server 192.168.33.131:80 weight=3; server 192.168.33.132:80 weight=3; } server { listen 80; server_name blog.zxl.com; location / { proxy_pass http://blog; #proxy_set_header Host $host; #proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; include /etc/nginx/conf.d/a.conf; } }
a.conf文件内容如下
[root@lb-01 conf.d]# cat a.conf proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; client_max_body_size 10m; client_body_buffer_size 128k; proxy_connect_timeout 30; proxy_send_timeout 15; proxy_read_timeout 15;
10.lb-nginx动态分离
使用www站点来做说明
lb-nginx配置如下
[root@lb-01 conf.d]# cat upstream.conf upstream static_pools { server 192.168.33.131:80 weight=3; } upstream dynamic_pools { server 192.168.33.132:80 weight=3; } server { listen 80; server_name www.zxl.com; location /static/ { proxy_pass http://static_pools; include /etc/nginx/conf.d/a.conf; } location /dynamic/ { proxy_pass http://dynamic_pools; include /etc/nginx/conf.d/a.conf; } }
include引用的a.conf配置文件
检查lb-nginx配置文件语法以及重新加载
[root@lb-01 conf.d]# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@lb-01 conf.d]# nginx -s reload
rs-02节点配置如下
[root@rs-02 nginx]# pwd /usr/share/nginx [root@rs-02 nginx]# ll total 20 drwxr-xr-x. 2 root root 4096 Jun 27 09:26 bbs drwxr-xr-x. 2 root root 4096 Jun 27 09:26 blog drwxr-xr-x. 2 root root 4096 Jun 27 09:18 html drwxr-xr-x. 2 root root 4096 Jun 27 09:56 logs drwxr-xr-x. 2 root root 4096 Jun 27 09:26 www [root@rs-02 nginx]# mkdir www/dynamic -p [root@rs-02 nginx]# echo dynamic > www/dynamic/index.html
rs-02节点测试访问www.zxl.com
[root@rs-02 nginx]# curl www.zxl.com nginx www 132 [root@rs-02 nginx]# curl www.zxl.com/dynamic/index.html dynamic [root@rs-02 nginx]# curl www.zxl.com/static/index.html <html> <head><title>404 Not Found</title></head> <body bgcolor="white"> <center><h1>404 Not Found</h1></center> <hr><center>nginx/1.0.15</center> </body> </html>
同理得rs-01节点也是如此
rs-01节点配置如下
[root@rs-01 ~]# mkdir /var/www/www/static/ [root@rs-01 ~]# echo static >/var/www/www/static/index.html
rs-01节点测试访问结果如下
[root@rs-01 ~]# curl www.zxl.com/static/index.html static [root@rs-01 ~]# curl www.zxl.com/dynamic/index.html <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /dynamic/index.html was not found on this server.</p> <hr> <address>Apache/2.2.15 (CentOS) Server at www.zxl.com Port 80</address> </body></html>
以上访问就已经实现动态分离了根据lb-nginx的字符串匹配,使用浏览器访问结果也是一样的
11.案例,匹配后缀名动态分离,示例如下
lb-nginx配置文件内容如下
server { listen 80; server_name www.zxl.com; location ~ .*. (gif|jpg|jpeg|png|bmp|swf|css|js)$ { proxy_pass http://static_pools; include a.conf; } location ~ .*. (php|php5)$ { proxy_pass http://dynamic_pools; include a.conf; } }