Varnish---原理及应用
原理如下图:
用户请求到达Varnish服务器,经由网卡将请求接入进来到达tcp/ip协议栈解封装后由varnish将报文中请求资源的uri进行hash计算,而后根据计算的得到的键,到进程维持的hash表对比,若键相同,则根据相应的键去值指针所标识的,内存地址空间或是硬盘地址空间将结果取出,而后构建响应报文,将结果响应给客户端,若在hash表中没有响应的键,则进入过程2将请求的报文重新封装,发送给后端的backend server去处理,得到结果保存在缓存中,而后重新构建响应报文,响应给客户端.
Varnish的软件架构
Varnish的安装
官方网站下载相应发行版的软件包共享库和帮助文档repo.varnish-cache.org
然后执行yum安装
[root@www ~]# yum install \
varnish-3.0.5-1.el6.x86_64.rpm \
varnish-docs-3.0.0-1.el6.x86_64.\
varnish-libs-3.0.5-1.el6.x86_64.rpm
需要配置epel源
配置文件分为两部分
1,master的配置文件:/etc/sysyconfig/varnish
2,child 的配置文件:/etc/varnish/default.vcl------>>>由master调用vcl编译器转换为C语言形式而后c编译器编译后交由child使用这样避免语法错误child可直接使用
简单应用篇:
实验拓扑
配置varnish
[root@www ~]# vim /etc/sysconfig/varnish
NFILES=131072
MEMLOCK=82000
NPROCS="unlimited"
RELOAD_VCL=1
VARNISH_VCL_CONF=/etc/varnish/default.vcl
VARNISH_LISTEN_PORT=80
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082
VARNISH_SECRET_FILE=/etc/varnish/secret
VARNISH_MIN_THREADS=50
VARNISH_MAX_THREADS=1000
VARNISH_THREAD_TIMEOUT=120
VARNISH_STORAGE_SIZE=64M
VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"
VARNISH_TTL=120
DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
-f ${VARNISH_VCL_CONF} \
-T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
-t ${VARNISH_TTL} \
-p thread_pool_min=${VARNISH_MIN_THREADS} \
-p thread_pool_max=${VARNISH_MAX_THREADS} \
-p thread_pool_timeout=${VARNISH_THREAD_TIMEOUT} \
-u varnish -g varnish \
-S ${VARNISH_SECRET_FILE} \
-s ${VARNISH_STORAGE}"
[root@www ~]# vim /etc/varnish/default.vcl
vcl 4.0;
backend default {
.host = "192.168.1.2";
.port = "80";
}
配置http编辑网页文件
Vim /var/www/html/index.html
<h1 Bakend-server 192.168.1.2 </h1>
启动各节点服务
Service vanish start
Service httpd start
客户端测试
进阶应用
需要使用VCL[varnish configure language]
请求数据在varnish中的流向
每个vcl_*函数都要有终止语句return(X)来说明下一个步要交给哪个函数处理
VCL基本语法[详情及变量man vcl]
1,//,#,/*comment*/ 表示注释
2,sub name,定义一个函数
3,不支持循环,有众多内置变量
4,使用终止语句,没有返回值
5,域专用
6,操作符 =赋值 ==等值比较 ~模式匹配 !取反 &&逻辑与 || 逻辑或
内置函数:
regsub (str,regex,sub) 搜索str中能被指定模式匹配的第一个字符串替换为sub
resuball (str,regex,sub) 搜索str中能被指定模式匹配的全部字符串替换为sub
Purge:从缓从中挑选出某对象及相关变种一并删除
Reyurn():当vcl域运行结束时控制权交由指定的下一个域处理
Return():重新运行整个vcl每次重启都会增加req.restarts变量中的值 max_restarts限定最大次数
Vcl_recv示例:
sub vcl_recv {
if (req.http.User-Agent ~ "iPad" ||
req.http.User-Agent ~ "iPhone" ||
req.http.User-Agent ~ "Android") {
set req.http.X-Device = "mobile";
} else {
set req.http.X-Device = "desktop";
}
}
此示例的含义是
如果请求的个护短代理类型是ipad,iphone,或是android那么就在求情的首部中添加一个新的首部X-device为mobile否则就设置为desktop
下面实验说明;在后端service的httpd访问日志中记录下X-Device的值
以下实验紧接上一个实验
首先配置varnish
sub vcl_recv {
# Happens before we check if we have this in cache already.
#
# Typically you clean up the request here, removing cookies you don't need,
# rewriting the request, etc.
if (req.http.User-Agent ~ "ipad"||
req.http.User-Agent ~ "iphone"||
req.http.User-Agent ~ "iphone"){
set req.http.X-Device = "mobile";
}else {
set req.http.X-Device = "desktop";
}
}
配置http日志格式加入X-Device首部的显示
重启httpd varnish使用客户端访问站点
验证结果
查看访问日志
上图所示的客户端地址为前端的varnish地址若向显示真实的地址方法很简单只需要在上述配置中增加一条vcl语句
sub vcl_recv {
if (req.http.User-Agent ~ "ipad"||
req.http.User-Agent ~ "iphone"||
req.http.User-Agent ~ "iphone"){
set req.http.X-Device = "mobile";
}else {
set req.http.X-Device = "desktop";
}
set req.http.X-Forwarded-For = client.ip;
}
编辑http访问日志格式
查看日志,客户端的真实地址就显示出来了
判断缓存是否命中
sub vcl_deliver {
accounting or modifying the final object here.
if (obj.hits > 0 ) {
set resp.http.X-Cache = "hit cache" + " " + server.hostname;
}else {
set resp.http.X-Cache = "miss cache" + " " + server.hostname;
}
}
测试结果
如何绕过缓存
sub vcl_recv {
if (req.http.User-Agent ~ "ipad"||
req.http.User-Agent ~ "iphone"||
req.http.User-Agent ~ "iphone"){
set req.http.X-Device = "mobile";
}else {
set req.http.X-Device = "desktop";
}
set req.http.X-Forwarded-For = client.ip;
if (req.url ~ "^/") {
return(pass);
}
}
测试结果,无论刷新几次都不会命中缓存
如何删除指定的缓存
acl purgers {
"127.0.0.1";
"172.16.0.0"/16;
}
sub vcl_recv {
if (req.request == "PURGE") {
if (!client.ip ~ purgers) {
error 405 "Method not allowed";
}
return (lookup);
}
}
sub vcl_hit {
if (req.request == "PURGE") {
purge;
error 200 "Purged";
}
}
sub vcl_miss {
if (req.request == "PURGE") {
purge;
error 404 "Not in cache";
}
}
sub vcl_pass {
if (req.request == "PURGE") {
error 502 "PURGE on a passed object";
}
}
客户端测试
这里为了方便测试使用linux工具curl
[root@localhost html]# curl http://192.168.1.2
<h1 Bakend-server 192.168.1.2 </h1>
查看头部信息
[root@localhost html]# curl -I http://192.168.1.2
HTTP/1.1 200 OK
Server: nginx/1.0.15
Content-Type: text/html
Last-Modified: Mon, 29 Sep 2014 06:15:54 GMT
Content-Length: 37
Accept-Ranges: bytes
Date: Mon, 29 Sep 2014 06:23:24 GMT
X-Varnish: 1450346254 1450346253
Age: 48
Via: 1.1 varnish
Connection: keep-alive
X-Cache: HIT via localhost.localdomain
此时缓存时命中的下面指定请求的方法看是否能够将缓存清除
[root@localhost html]# curl -X PURGE http://172.16.34.1
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<title>200 Purged</title>
</head>
<body>
<h1>Error 200 Purged</h1>
<p>Purged</p>
<h3>Guru Meditation:</h3>
<p>XID: 1450346261</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
看到返回的信息缓存已被清除
下面再次请求主页面验证是否清除 如果缓存没有hit 证明清除成功
[root@localhost html]# curl -I http://172.16.34.1
HTTP/1.1 200 OK
Server: nginx/1.0.15
Content-Type: text/html
Last-Modified: Mon, 29 Sep 2014 06:15:54 GMT
Content-Length: 37
Accept-Ranges: bytes
Date: Mon, 29 Sep 2014 06:27:00 GMT
X-Varnish: 1450346262
Age: 0
Via: 1.1 varnish
Connection: keep-alive
X-Cache: MISS via localhost.localdomain
由此可见缓存MISS证明清除成功
Varnish实现负载均衡
编辑配置文件
backend web1 {
.host = "192.168.1.2";
.port = "8080";
.probe = {
.url="/.health.html";
.interval = 2s;
.window = 8;
.threshold = 3;
.initial = 2 ;
}
}
backend web2 {
.host = "192.168.1.3";
.port = "8080";
.probe = {
.url="/.health.html"; ------------->检测的页面
.interval = 2s; ------------->健康监测的时间间隔
.window = 8; ------------->检测的次数
.threshold = 3; -------------->3次失败才认为后端主机不可用
.initial = 2 ; ------------->2次成功就说明后端主机正常
}
}
director http_server round-robin { ----->算法为rr
{.backend=web1;}
{.backend=web2;}
}
acl purgers {
"127.0.0.1";
"172.16.0.0"/16;
}
sub vcl_recv {
if (req.url ~ "^/"){
set req.backend = http_server;
return (pass);
}
if (req.request == "PURGE") {
if (!client.ip ~ purgers) {
error 405 "Method not allowed";
}
return (lookup);
}
}
sub vcl_hit {
if (req.request == "PURGE") {
purge;
error 200 "Purged";
}
}
sub vcl_miss {
if (req.request == "PURGE") {
purge;
error 404 "Not in cache";
}
}
sub vcl_pass {
if (req.request == "PURGE") {
error 502 "PURGE on a passed object";
}
}
sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT via" + " " + server.hostname;
} else {
set resp.http.X-Cache = "MISS via" + " " + server.hostname;
}
}
配置后端服务器页面建立健康监测页面两个节点分别建立
Server1 :Vim /var/www/html/{index.html,.health.html}
Index.html内容 -----><h1> Bacekend-server1 192.16.1.2 </h1>
health.html内容 -----> OK
Server2 :Vim /var/www/html/{index.html,.health.html}
Index.html内容 -----><h1> Bacekend-server2 192.16.1.3 </h1>
health.html内容 -----> OK
客户端测试
[root@localhost ~]# curl http://172.16.101.200
<h1> Backend-server1 192.168.1.2 </h1>
[root@localhost ~]# curl http://172.16.101.200
<h1> Back-server2 192.168.1.3 </h1>
[root@localhost ~]# curl http://172.16.101.200
<h1> Backend-server1 192.168.1.2 </h1>
[root@localhost ~]# curl http://172.16.101.200
<h1> Back-server2 192.168.1.3 </h1>
查看后端状态
[root@localhost ~]# varnishadm
varnish> backend.list
200
Backend name Refs Admin Probe
web1(192.168.1.2,,8080) 1 probe Healthy 8/8
web2(192.168.1.3,,8080) 1 probe Healthy 8/8
停止其中一个backend-server
varnish> backend.list
200
Backend name Refs Admin Probe
web1(192.168.1.2,,8080) 1 probe Sick 0/8
web2(192.168.1.3,,8080) 1 probe Healthy 8/8
这是只能访问web2
