logfile -> filebeat > kafka（依赖zookeeper）-> logstash -> elasticsearch -> kibana
1.部署zookeeper集群1.1 主机IP规划
hostname | IP |
zk-kfk-1 | 192.168.2.26 |
zk-kfk-2 | 192.168.2.27 |
zk-kfk-3 | 192.168.2.28 |
3个服务器上都需要操作以下安装配置步骤
1.2 安装jdk11
apt install openjdk-11-jdk -y
1.3 下载安装包
wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz
1.4 部署配置
tar xf apache-zookeeper-3.7.0-bin.tar.gz -C /usr/local/
cd /usr/local
ln -s apache-zookeeper-3.7.0-bin/ zookeeper
cd conf/
cp zoo_sample.cfg zoo.cfg
mkdir -p /data/zookeeper-data
# 配置文件修改如下：
root@zk-kfk-1:/usr/local/zookeeper/conf# sed -e '/^#/d' -e '/^$/d' zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper-data
clientPort=2181
maxClientCnxns=150
autopurge.snapRetainCount=3
autopurge.purgeInterval=1
server.1=192.168.2.26:2888:3888
server.2=192.168.2.27:2888:3888
server.3=192.168.2.28:2888:3888
1.5 启动服务并查看运行状态
# /usr/local/zookeeper/bin/zkServer.sh start
root@zk-kfk-1:# /usr/local/zookeeper/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
root@zk-kfk-2:# /usr/local/zookeeper/bin/zkServer.sh status
...
Mode: follower
root@zk-kfk-3:# /usr/local/zookeeper/bin/zkServer.sh status
...
Mode: leader
复用zookeeper集群3台服务器部署，以下操作3台服务器上都需要执行
2.1 下载安装包并解压
# 下载
wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/3.0.0/kafka_2.13-3.0.0.tg
# 解压
tar xf kafka_2.13-3.0.0.tgz -C /usr/local/
cd /usr/local
ln -s kafka_2.13-3.0.0/ kafka
# 创建数据目录
mkdir -p /data/kafka-logs
2.2 修改配置文件
# vim /usr/local/kafka/config/server.properties
broker.id=26
listeners=PLAINTEXT://192.168.2.26:9092
log.dirs=/data/kafka-logs
zookeeper.connect=192.168.2.26:2181,192.168.2.27:2181,192.168.2.28:2181
2.3 启动服务
cd /usr/local/kafka && ./bin/kafka-server-start.sh -daemon config/server.properties
使用绝对路径启动会报错：
# /usr/local/kafka/bin/kafka-server-start.sh --daemon /usr/local/kafka/config/server.properties
[2021-10-28 08:51:14,097] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-10-28 08:51:14,455] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
...
复用zookeeper,kafka 3台服务器
3.1 下载安装包
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.15.1-amd64.deb
3.2 安装
dpkg -i elasticsearch-7.15.1-amd64.deb
3.3 修改配置
集群中的另外2个节点的node.name，network.host 2个配置需按实际情况修改
# es node-1的配置
root@zk-kfk-1:/data# cat /etc/elasticsearch/elasticsearch.yml
cluster.name: k8s-elk
node.name: node-1
path.data: /data/es/data
path.logs: /data/es/logs
network.host: 192.168.2.26
http.port: 9200
discovery.seed_hosts: ["192.168.2.26", "192.168.2.27","192.168.2.28"]
cluster.initial_master_nodes: ["192.168.2.26", "192.168.2.27","192.168.2.28"]
action.destructive_requires_name: true
# 创建数据和日志目录
mkdir -p /data/es/data
mkdir -p /data/es/logs
# 修改目录权限
chown -R elasticsearch:elasticsearch /data/es/
3.4 启动服务
systemctl start elasticsearch.service
3.5 使用elasticsearch-head 查看es集群状态
chrome浏览器加载elasticsearch-head 插件参考：
采用将filebeat和nginx放到一个容器里面方案
4.1 制作镜像
Dockerfile
FROM nginx:1.21.1
COPY nginx.conf /etc/nginx
COPY default.conf /etc/nginx/conf.d
COPY filebeat-7.15.1-amd64.deb /root
COPY start_filebeat.sh /docker-entrypoint.d
RUN rm -f /var/log/nginx/* \
&& dpkg -i /root/filebeat-7.15.1-amd64.deb \
&& rm -f /root/filebeat-7.15.1-amd64.deb
COPY filebeat.yml /etc/filebeat
filebeat.yaml配置文件
filebeat.inputs:
- type: log
paths:
- /var/log/nginx/access.log
tail_files: true
tags: ["nginx-access"]
fields:
log_topics: nginx-access
setup.kibana:
host: "192.168.2.29:5601"
output.kafka:
enabled: true
hosts: ["192.168.2.26:9092","192.168.2.27:9092","192.168.2.28:9092"]
topic: '%{[fields][log_topics]}'
filebeat运行脚本
cat start_filebeat.sh
#/bin/bash
nohup /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat 2>&1 &
镜像目录文件列表
filebeat-7.15.1-amd64.deb 可以在官网下载
总用量 35016
drwxr-xr-x 2 root root 4096 10月 29 14:42 ./
drwxr-xr-x 4 root root 4096 10月 28 18:59 ../
-rwxr-xr-x 1 root root 137 10月 29 14:39 build_command.sh*
-rw-r--r-- 1 root root 619 10月 28 18:59 default.conf
-rw-r--r-- 1 root root 317 10月 29 14:42 Dockerfile
-rw-r--r-- 1 root root 35821726 10月 28 18:59 filebeat-7.15.1-amd64.deb
-rw-r--r-- 1 root root 518 10月 29 14:27 filebeat.yml
-rw-r--r-- 1 root root 1270 10月 28 19:01 nginx.conf
-rwxr-xr-x 1 root root 207 10月 29 11:24 start_filebeat.sh*
build_command.sh 脚本
#!/bin/bash
TAG=$1
docker build -t 192.168.1.110/web/nginx-filebeat:${TAG} .
sleep 1
docker push 192.168.1.110/web/nginx-filebeat:${TAG}
生成镜像文件并上传到本地harbor
# ./build_command.sh 2021.10.29-144201
# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
192.168.1.110/web/nginx-filebeat 2021.10.29-144201 8f011ec312e5 4 hours ago 312MB
使用docker run运行测试
# docker run -d -p 81:80 192.168.1.110/web/nginx-filebeat:2021.10.29-144201
root@k8-deploy:~/k8s-yaml/web/nginx_2/dockerfile# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
40dff03b2e4a 192.168.1.110/web/nginx-filebeat:2021.10.29-144201 "/docker-entrypoint.…" 4 hours ago Up 4 hours 0.0.0.0:81->80/tcp exciting_perlman
进入容器并查看nginx log
root@40dff03b2e4a:/# tail -f /var/log/nginx/*.log
==> /var/log/nginx/access.log <==
29/Oct/2021:06:44:57 +0000|172.17.0.1|localhost|GET / HTTP/1.1|200|612|0.000|905|76|-|-|curl/7.68.0
29/Oct/2021:06:51:43 +0000|127.0.0.1|localhost|GET / HTTP/1.1|200|612|0.000|905|73|-|-|curl/7.64.0
使用kafka客户端工具（offsetexplorer_64bit.exe）查看kafka中是否已经有数据
5.1 下载logstash安装包
https://artifacts.elastic.co/downloads/logstash/logstash-7.15.1-amd64.deb
5.2 安装
dpkg -i logstash-7.15.1-amd64.deb
5.3 编写logstash处理nginx日志的配置文件
配置文件路径：/etc/logstash/conf.d/kafka-to-es.conf
input {
kafka {
bootstrap_servers => "192.168.2.26:9092,192.168.2.27:9092,192.168.2.28:9092"
auto_offset_reset => "latest"
topics=> ["nginx-access"]
consumer_threads => 2
decorate_events => "true"
}
}
#=============================================================================
filter {
json {
source => "message"
}
mutate{
split => ["message","|"]
add_field => { "time_local" => "%{[message][0]}" }
add_field => { "client_ip" => "%{[message][1]}" }
add_field => { "domain_name" => "%{[message][2]}" }
add_field => { "request" => "%{[message][3]}" }
add_field => { "status" => "%{[message][4]}" }
add_field => { "body_bytes_sent" => "%{[message][5]}" }
add_field => { "request_time" => "%{[message][6]}" }
add_field => { "bytes_sent" => "%{[message][7]}" }
add_field => { "request_length" => "%{[message][8]}" }
add_field => { "upstream_response_time" => "%{[message][9]}" }
add_field => { "http_referer" => "%{[message][10]}" }
add_field => { "user_agent" => "%{[message][11]}" }
remove_field => ["message", "log", "file", "offset", "ecs", "version", "path", "agent", "beat", "@version", "input", "prospector", "source", "offset"]
}
date {
match => [ "time_local","dd/MMM/YYYY:HH:mm:ss Z" ]
target => "@timestamp"
}
}
#=============================================================================
#调试用
output {
stdout { codec => rubydebug }
}
# 生产环境使用
#output {
# elasticsearch {
# hosts => [ "http://192.168.2.26:9200","http://192.168.2.27:9200","http://192.168.2.28:9200"]
# index => "%{tags}-%{+YYYY.MM.dd}"
# }
#}
测试配置文件
root@zk-kfk-1:/etc/logstash/conf.d# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/kafka-to-es.conf -t
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-10-29 19:05:14.181 [main] runner - Starting Logstash {"logstash.version"=>"7.15.1", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.12+7 on 11.0.12+7 +indy +jit [linux-x86_64]"}
[INFO ] 2021-10-29 19:05:14.293 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2021-10-29 19:05:14.330 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2021-10-29 19:05:15.140 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-10-29 19:05:18.207 [LogStash::Runner] Reflections - Reflections took 166 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2021-10-29 19:05:20.035 [LogStash::Runner] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
Configuration OK
5.5 使用调试模式将logstash处理后的日志直接打印到屏
root@zk-kfk-1:/etc/logstash/conf.d# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/kafka-to-es.conf
...
5.6 在nginx容器内重新写几条数据进行测试
root@40dff03b2e4a:/# curl 127.0.0.1
5.7 观察logstash是否有相关日志的输出
{
"request" => "GET / HTTP/1.1",
"client_ip" => "127.0.0.1",
"status" => "200",
"@timestamp" => 2021-10-29T11:07:16.000Z,
"request_length" => "73",
"http_referer" => "-",
"domain_name" => "localhost",
"user_agent" => "curl/7.64.0",
"bytes_sent" => "905",
"request_time" => "0.000",
"time_local" => "29/Oct/2021:11:07:16 +0000",
"fields" => {
"log_topics" => "nginx-access"
},
"host" => {
"name" => "40dff03b2e4a"
},
"upstream_response_time" => "-",
"tags" => [
[0] "nginx-access"
],
"body_bytes_sent" => "612"
}
5.7 测试没问题后，更改logstash的配置文件，将日志写入es集群，并重新以服务的方式启动logstash
# 修改配置文件 kafka-to-es.conf
#output {
# stdout { codec => rubydebug }
#}
output {
elasticsearch {
hosts => [ "http://192.168.2.26:9200","http://192.168.2.27:9200","http://192.168.2.28:9200"]
index => "%{tags}-%{+YYYY.MM.dd}"
}
}
# systemctl start logstash.service
6.1 下载安装包并安装
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.15.1-amd64.deb
dpkg -i kibana-7.15.1-amd64.deb
6.2 修改配置文件如下参数
# 配置文件 /etc/kibana/kibana.yml
server.port: 5601
server.host: "192.168.2.27"
elasticsearch.hosts: ["http://192.168.2.29:9200"]
i18n.locale: "zh-CN"
6.3 查看kibana监听端口
root@zk-kfk-2:~# netstat -ntlp |grep 5601
tcp 0 0 192.168.2.27:5601 0.0.0.0:* LISTEN 31969/node
6.4 浏览器访问kibana并测试
本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。
