前言
本章将介绍Nginx监控安装
1. 安装Nginx / OpenResty
1.1 安装 OpenResty
VeryNginx 基于 OpenResty,所以你需要先安装它:
OpenResty安装前准备
- Centos:yum install -y readline-devel pcre-devel openssl-devel gcc GeoIP-devel
- Debian|Ubuntu:apt-get install libreadline-dev libncurses5-dev libpcre3-dev libssl-dev perl make build-essential libgeoip-dev
-
geoip-devel
为geoip模块所需要的基础库
●wget https://openresty.org/download/openresty-1.11.2.1.tar.gz
●tar -zxvf openresty-1.11.2.1.tar.gz
●cd openresty-1.11.2.1
●./configure --with-luajit --with-pcre --with-http_gzip_static_module --with-http_realip_module --with-http_geoip_module --with-http_ssl_module --with-http_stub_status_module --prefix=/app/openresty
●make && make install
VeryNginx 实际使用到了 OpenResty 中的这些模块
- lua-nginx-module
- http_stub_status_module
- lua-cjson library
如果你不想安装 OpenResty,或者你已经有了一个正在工作的 Nginx,你也可以自己手动为 Nginx 编译安装这些模块
1.2 安装部署 VeryNginx
克隆 VeryNginx 仓库到本地, 复制 nginx.conf 和 VeryNginx 文件夹到 Nginx 的工作目录.
cd ~
git clone https://github.com/alexazhou/VeryNginx.git
rm -f /app/openresty/nginx/conf/nginx.conf
cp ~/VeryNginx/nginx.conf /app/openresty/nginx/conf/nginx.conf
# 创建 /app/openresty/nginx/waf/verynginx 目录
mkdir -p /app/openresty/nginx/waf/verynginx
# 把 /VeryNginx/verynginx 拷贝到 /app/openresty/nginx/waf/verynginx
cp -r ~/VeryNginx/verynginx /app/openresty/nginx/waf/verynginx
1.3 编辑 Nginx 配置文件
VeryNginx 项目提供了一个配置模版 /app/openresty/nginx/conf
。你需要把自己站点的 Nginx
配置加到这个模版里面。 但是记得不要修改配置 VeryNginx 的那部分代码(除非你知道自己在干啥 ? )。
接下来,说明配置文件有三点比较重要:
include /app/openresty/nginx/waf/verynginx/verynginx/nginx_conf/in_external.conf;
include /app/openresty/nginx/waf/verynginx/verynginx/nginx_conf/in_http_block.conf;
include /app/openresty/nginx/waf/verynginx/verynginx/nginx_conf/in_server_block.conf;
in_http_block.conf z合格配置文件需要修改
upstream vn_upstream{
server 127.0.0.1;
balancer_by_lua_file /app/openresty/nginx/waf/verynginx/verynginx/lua_script/on_banlance.lua;
keepalive 1024; #Connection pool
}
lua_package_path '/app/openresty/nginx/waf/verynginx/verynginx/lua_script/?.lua;;/app/openresty/nginx/waf/verynginx/verynginx/lua_script/module/?.lua;;';
lua_package_cpath '/app/openresty/nginx/waf/verynginx/verynginx/lua_script/?.so;;';
lua_code_cache on;
lua_shared_dict status 1m;
lua_shared_dict frequency_limit 10m;
lua_shared_dict summary_long 10m;
lua_shared_dict summary_short 10m;
init_by_lua_file /app/openresty/nginx/waf/verynginx/verynginx/lua_script/on_init.lua;
rewrite_by_lua_file /app/openresty/nginx/waf/verynginx/verynginx/lua_script/on_rewrite.lua;
access_by_lua_file /app/openresty/nginx/waf/verynginx/verynginx/lua_script/on_access.lua;
log_by_lua_file /app/openresty/nginx/waf/verynginx/verynginx/lua_script/on_log.lua;
其他2个文件不要改变
配置 VeryNginx 的代码是下面这部分:
#-----------------VeryNginx config code------------------
#user nginx;
worker_processes auto;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
include /app/openresty/nginx/waf/verynginx/verynginx/nginx_conf/in_external.conf;
#include /app/openresty/nginx/waf/verynginx/nginx_conf/in_http_block.conf;
http {
charset utf-8;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
server_tokens off;
types_hash_max_size 2048;
server_names_hash_bucket_size 128;
large_client_header_buffers 4 32k;
fastcgi_intercept_errors on;
proxy_intercept_errors on;
underscores_in_headers on;
client_header_timeout 15s;
client_body_timeout 15s;
client_max_body_size 100m;
client_body_buffer_size 2m;
client_header_buffer_size 256k;
#keepalive_timeout 0;
keepalive_timeout 90s;
keepalive_requests 2000;
# MIME
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
#tcp_nopush on;
#keepalive_timeout 0;
# gzip
gzip on;
gzip_min_length 1k;
gzip_buffers 4 16k;
gzip_http_version 1.1;
gzip_comp_level 6;
gzip_types text/plain text/css application/javascript application/x-javascript application/xml application/json;
gzip_vary on;
proxy_connect_timeout 5s;
proxy_read_timeout 1800s;
proxy_send_timeout 1800s;
proxy_buffer_size 128k;
proxy_buffers 100 128k;
#proxy_busy_buffers_size 1m;
#proxy_temp_file_write_size 512k;
#this line shoud be include in every http block
include /app/openresty/nginx/waf/verynginx/verynginx/nginx_conf/in_http_block.conf;
log_format main '$remote_addr -'
' $remote_user'
' [$time_local]'
' "$request"'
' $status'
' $body_bytes_sent'
' "$http_referer"'
' "$http_user_agent"'
' "$http_x_forwarded_for"'
' $upstream_response_time'
' $upstream_addr';
access_log logs/access.log main;
upstream gateway_pool {
# 网关的地址
server 106.12.129.14:9200;
}
server {
listen 80;
#this line shoud be include in every server block
include /app/openresty/nginx/waf/verynginx/verynginx/nginx_conf/in_server_block.conf;
#include /app/openresty/nginx/waf/verynginx/nginx_conf/in_http_block.conf;
location / {
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_redirect off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
proxy_set_header Host $host;
proxy_pass http://gateway_pool;
}
}
}
#---------------VeryNginx config code end-----------------
如果不使用 VeryNginx 提供的配置模版,你也可以手动把这部分加入到自己的 Nginx 配置文件中. (如果安装路径不是/app/openresty ,需要对 lua_package_cpath 和 lua_package_path 的值进行修改)
1.4 启/停服务
启动服务: /app/openresty/nginx/sbin/nginx
停止服务:/app/openresty/nginx/sbin/nginx -s stop
1.5 配置VeryNginx
打开浏览器访问 http://127.0.0.1/VeryNginx/dashboard/index.html
。
默认用户名和密码:verynginx / verynginx 。
登录之后就可以查看状态,并对配置进行修改了。修改配置后,记得到 「配置 > 系统 > 全部配置」去保存.
提示
- 通过 VeryNginx 控制面板保存新配置之后,会立刻生效,并不需要 restart/reload Nginx。
- VeryNginx 把配置保存在 /app/openresty/nginx/waf/verynginx/configs/config.json 里面。
- 如果因为配错了什么选项,导致无法登录,可以手动删除 config.json 来清空配置。
已安装成功openresty
可以用网盘的覆盖/app/openresty/nginx/waf目录
链接:https://pan.baidu.com/s/1nt2SfrDtk83Q4gvwIlpBfQ 提取码:g5hs
2. 安装elasticsearch
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.4.tar.gz
tar -zxvf elasticsearch-6.5.4.tar.gz -C /usr/local/
useradd es
chown -R es:es /usr/local/elasticsearch-6.5.4/
cd /usr/local/elasticsearch-6.5.4
修改config/jvm.options为内存的一半大小
vi config/jvm.options
-Xms512m
-Xmx512m
修改 max file 和 max virtual memory 参数
用root 或 sudo 用户
vi /etc/sysctl.conf
添加下面配置:
vm.max_map_count=655360
并执行命令:
sysctl -p
配置端口 跨域
vi /usr/local/elasticsearch-6.5.4/config/elasticsearch.yml
cluster.name: elasticsearch
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
node.max_local_storage_nodes: 2
http.cors.enabled: true
http.cors.allow-origin: "*"
切换es用户
su - es
在Elasticsearch主目录下运行下列命令来安装这些插件:
bin/elasticsearch-plugin install ingest-geoip
bin/elasticsearch-plugin install ingest-user-agent
es用户启动
/usr/local/elasticsearch-6.5.4/bin/elasticsearch -d
root用户启动
su - es -c '/usr/local/elasticsearch-6.5.4/bin/elasticsearch -d'
2.1 windows 安装elasticseach-head
- 访问 https://github.com/mobz/elasticsearch-head 下载 head 插件(选择 zip 压缩包下载方式)。
- 修改 ~\elasticsearch-6.6.2\elasticsearch-head-master\Gruntfile.js,在对应的位置加上 hostname:’*’ 配置项。
- 在 ~\elasticsearch-6.6.2\elasticsearch-head-master 下执行 npm install 开始安装,完成后可执行 grunt server 或者 npm run start 运行 head 插件。
- 安装成功,访问 http://localhost:9100/。
2.2 注意事项
- 在 head 中连接 ES 失败。
对于 Access-Control-Allow-Origin 的问题,可以在 ElasticSearch 6.x 的 ~\config\elasticsearch.yml 文件的末尾加入以下代码:
http.cors.enabled: true
http.cors.allow-origin: "\*"
node.master: true node.data: true
配置更新后,重启 ES 即可连接成功。
3. 安装Filebeat
推荐博客:
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.5.4-linux-x86_64.tar.gz
tar -zxvf filebeat-6.5.4-linux-x86_64.tar.gz -C /usr/local/
cd /usr/local/
mv filebeat-6.5.4-linux-x86_64 filebeat-6.5.4
cd /usr/local/filebeat-6.5.4
vi filebeat.yml
filebeat.inputs:
- type: log
enabled: false
paths:
- /app/openresty/nginx/logs/access.log
#============================= Filebeat modules ===============================
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
setup.template.name: "nginx-log-"
setup.template.pattern: "nginx-log-*"
setup.template.overwrite: true
output.elasticsearch:
enabled: true
hosts: ["106.13.3.200:9200"]
index: "nginx-log-%{+yyyy-MM-dd}"
#================================ Processors =====================================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
#- add_host_metadata: ~
#- add_cloud_metadata: ~
- drop_fields:
fields: ["beat.name", "beat.version", "host.architecture","host.architecture","host.name","beat.hostname","log.file.path"]
3.1 启用模块nginx
cd /usr/local/filebeat-6.5.4/modules.d
vi nginx.yml
执行以下配置
- module: nginx
# Access logs
access:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/app/openresty/nginx/logs/access.log"]
# Error logs
error:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/app/openresty/nginx/logs/error.log"]
3.2 配置 default.json
cd /usr/local/filebeat-6.5.4/module/nginx/access/ingest
vi default.json
{
"description": "Pipeline for parsing Nginx access logs. Requires the geoip and user_agent plugins.",
"processors": [{
"grok": {
"field": "message",
"patterns":[
"\"?%{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{GREEDYDATA:nginx.access.info}\" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} \"%{DATA:nginx.access.referrer}\" \"%{DATA:nginx.access.agent}\" \"%{GREEDYDATA:nginx.access.xforwardedfor}\" %{GREEDYDATA:nginx.access.upstream_response_time} %{GREEDYDATA:nginx.access.upstream_addr}"
],
"pattern_definitions": {
"IP_LIST": "%{IP}(\"?,?\\s*%{IP})*"
},
"ignore_missing": true
}
}, {
"grok": {
"field": "nginx.access.info",
"patterns": [
"%{WORD:nginx.access.method} %{DATA:nginx.access.url} HTTP/%{NUMBER:nginx.access.http_version}",
""
],
"ignore_missing": true
}
}, {
"remove": {
"field": "nginx.access.info"
}
}, {
"split": {
"field": "nginx.access.remote_ip_list",
"separator": "\"?,?\\s+"
}
}, {
"script": {
"lang": "painless",
"inline": "boolean isPrivate(def ip) { try { StringTokenizer tok = new StringTokenizer(ip, '.'); int firstByte = Integer.parseInt(tok.nextToken()); int secondByte = Integer.parseInt(tok.nextToken()); if (firstByte == 10) { return true; } if (firstByte == 192 && secondByte == 168) { return true; } if (firstByte == 172 && secondByte >= 16 && secondByte <= 31) { return true; } if (firstByte == 127) { return true; } return false; } catch (Exception e) { return false; } } def found = false; for (def item : ctx.nginx.access.remote_ip_list) { if (!isPrivate(item)) { ctx.nginx.access.remote_ip = item; found = true; break; } } if (!found) { ctx.nginx.access.remote_ip = ctx.nginx.access.remote_ip_list[0]; }"
}
}, {
"remove":{
"field": "message"
}
}, {
"rename": {
"field": "@timestamp",
"target_field": "read_timestamp"
}
}, {
"date": {
"field": "nginx.access.time",
"target_field": "@timestamp",
"formats": ["dd/MMM/YYYY:H:m:s Z"]
}
},{
"remove": {
"field": "nginx.access.time"
}
}, {
"user_agent": {
"field": "nginx.access.agent",
"target_field": "nginx.access.user_agent"
}
}, {
"rename": {
"field": "nginx.access.agent",
"target_field": "nginx.access.user_agent.original"
}
}, {
"geoip": {
"field": "nginx.access.remote_ip",
"target_field": "nginx.access.geoip"
}
}, {
"script": {
"lang": "painless",
"inline": "String tmp=ctx.nginx.access.upstream_response_time; if (tmp=='-'){ctx.nginx.access.upstream_response_time=-1.0}else{ctx.nginx.access.upstream_response_time=Float.parseFloat(tmp)}"
}
}],
"on_failure" : [{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}]
}
cd /usr/local/filebeat-6.5.4
nohup ./filebeat -e -c filebeat.yml >&/dev/null &
4. 安装Grafana
4.1 安装步骤
wget https://dl.grafana.com/oss/release/grafana-6.1.6.linux-amd64.tar.gz
tar -zxvf grafana-6.1.6.linux-amd64.tar.gz
cd /app/grafana-6.1.6/bin
grafana-cli plugins install grafana-piechart-panel
grafana-cli plugins install grafana-piechart-panel
grafana-cli plugins install grafana-worldmap-panel
grafana-cli plugins install grafana-piechart-panel
grafana-cli plugins install grafana-piechart-panel
./grafana-server
4.1 Grafana配置
默认安装3000端口,这里地址为:106.13.3.200:3000
默认用户名/密码 admin/admin
4.1.1 配置数据源
4.1.2 json模板
由于Json模板过长,另存入百度网盘中。
4.1.3 导入nginx dashboard
划重点
以上来自开源项目OCP: https://gitee.com/owenwangwen/open-capacity-platform
项目演示地址
http://59.110.164.254:8066/login.html 用户名/密码:admin/admin
项目监控 http://106.13.3.200:3000 用户名/密码:admin/1q2w3e4r
项目代码地址 https://gitee.com/owenwangwen/open-capacity-platform