资源路径 JAVA http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html
ELK https://www.elastic.co/
安装目录: /usr/local/elasticsearch /usr/local/logstash /usr/local/kibana
服务端安装
一、JDK安装 1、上传JDK8 rz 2、解压JDK8 tar -zxvf jdk-8u181-linux-x64.tar.gz 3、将JDK移动到/usr/local/java cp -r jdk1.8.0_181 /usr/local/java 4、修改环境变量 vim /etc/profile
export JAVA_HOME=/usr/local/java
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH
5、使改变生效并检测版本 source /etc/profile java -version
二、安装ELK 1、上传并解压 tar -zxvf elasticsearch-6.4.1.tar.gz tar -zxvf logstash-6.4.1.tar.gz tar -zxvf kibana-6.4.1-linux-x86_64.tar.gz
mv elasticsearch-6.4.1 /usr/local/elasticsearch mv logstash-6.4.1 /usr/local/logstash mv kibana-6.4.1-linux-x86_64 /usr/local/kibana
2、修改权限(elasticsearch不能以root用户运行) useradd elk passwd elk chkown -R elk.elk /usr/local/elasticsearch chown -R elk.elk /usr/local/elasticsearch
3、修改配置: 修改文件描述符:ulimited vim /etc/security/limits.conf elk soft nofile 102400 elk hard nofile 102400 其他也为 102400
vim /etc/security/limits.d/90-nproc.conf * soft nproc 65535
修改/etc/sysctl.conf配置文件,添加 vm.max_map_count=262144 需要重启机器
elasticsearch.yml:
注意要在Network下面:
network.host: 0.0.0.0
http.port: 9200
在elasticsearch.yml中配置bootstrap.system_call_filter为false,注意要在Memory下面:
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
kibana/config/kibana.yml
server.host: "10.10.10.101"
elasticsearch.url: "http://10.10.10.100:9200"
4、运行 elasticsearch:(注意,保存nohup.out的目录,elk必须有写入权限) su elk nohup /usr/local/elasticsearch/bin/elasticsearch &
kibana:
nohup /usr/local/kibana/bin/kibana &
logstash:(如果不加入redis,可以不用这个)
nohup /usr/local/logstash/bin/logstash agent -f ../conf/logstash.conf &
5、安装Nginx并配置 在 location/ 下增加
auth_basic "secret";
auth_basic_user_file /data/nginx/db/passwd.db;
创建认证数据文件
htpasswd -c /usr/opt/nginx/passwd.db root
如果没有htpasswd,则要安装httpd
yum -y install httpd
客户端安装
安装目录: /usr/local/logstash
运行目录: /data/ELK/logstash
配置目录: /data/ELK/logstash/conf
一、JDK安装 1、上传JDK8 rz 2、解压JDK8 tar -zxvf jdk-8u181-linux-x64.tar.gz 3、将JDK移动到/usr/local/java cp -r jdk1.8.0_181 /usr/local/java 4、修改环境变量 vim /etc/profile
export JAVA_HOME=/usr/local/java
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH
5、使改变生效并检测版本 source /etc/profile java -version
二、安装ELK 1、上传并解压 tar -zxvf logstash-6.4.1.tar.gz mv logstash-6.4.1 /usr/local/logstash
2、配置logstash logstash_nginx.conf内容 input { file { add_field => {"server_name"=>"nginx_33"} path => [ "/data/nginx_log/*access.log" ] start_position => "beginning" ignore_older => 0 } }
filter { grok { patterns_dir => ["/data/ELK/logstash/conf/patterns"] match => { "message" => "%{NGINXACCESS}" }
}
geoip {
source => "clientip"
target => "geoip"
database => "/data/ELK/logstash/conf/GeoLite2-City.mmdb"
fields => ["country_name","region_code", "city_name", "ip"]
#add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
#add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
# convert => [ "[geoip][coordinates]", "float" ]
convert => [ "response","integer" ]
convert => [ "bytes","integer" ]
replace => { "type" => "nginx_access" }
remove_field => "message"
}
date {
match => [ "timestamp","dd/MMM/yyyy:HH:mm:ss Z"]
}
mutate {
remove_field => "timestamp"
}
} output { elasticsearch { hosts => ["10.10.10.100:9200"] index => "logstash-nginx-access-%{+YYYY.MM.dd}" } #stdout {codec => rubydebug} }
/data/ELK/logstash/conf/patterns中nginx 文件内容: WZ ([^ ]*) NGINXACCESS %{IP:clientip} - - [%{HTTPDATE:timestamp}] "%{WORD:method} %{WZ:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} %{NUMBER:bytes} %{QS:referer} %{QS:agent} %{QS:xforward}
GeoLite2
cd /data/ELK/logstash/conf
wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
tar zxvf GeoLite2-City.tar.gz
cp GeoLite2-City_20180911/GeoLite2-City.mmdb ./
3、运行 nohup /usr/local/logstash/bin/logstash -f conf/logstash_nginx.conf &