本文默认k8s环境以及已经部署ingress controller

公司所用ingress监控是由prometheus+grafana进行,但是监控不够全面,故使用filebeat去采集ingress日志,并自主进行可视化展示

1、ingress nginx日志数据落盘

在ingress controller中将configmap改为


kind: ConfigMap apiVersion: v1 metadata:   name: ingress-nginx-controller data:   access-log-path: /var/log/nginx/access.log   compute-full-forwarded-for: 'true'   enable-vts-status: 'true'   error-log-path: /var/log/nginx/error.log   forwarded-for-header: X-Forwarded-For   log-format-upstream: >-     {"@timestamp":     "$time_iso8601","remote_addr":"$remote_addr","x-forward-for":"$http_x_forwarded_for","request_id":"$req_id","remote_user":"$remote_user","bytes_sent":$bytes_sent,"request_time":$request_time,"status":$status,"vhost":"$host","request_proto":"$server_protocol","path":"$uri","request_query":"$args","request_length":$request_length,"duration":$request_time,"method":"$request_method","http_referrer":"$http_referer","http_user_agent":"$http_user_agent","upstream-sever":"$proxy_upstream_name","proxy_alternative_upstream_name":"$proxy_alternative_upstream_name","upstream_addr":"$upstream_addr","upstream_response_length":$upstream_response_length,"upstream_response_time":$upstream_response_time,"upstream_status":$upstream_status}   use-forwarded-headers: 'true'


2、生成filebeat镜像

新建目录,目录如下

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_linux



dockerfile


FROM  million12/centos-supervisor:4.0.2 WORKDIR /usr/local ADD filebeat-7.5.0-linux-x86_64.tar.gz . RUN ln -s filebeat-7.5.0-linux-x86_64  filebeat \  && cd filebeat       \  && mkdir  config     \  && chmod +x filebeat \  && cp filebeat.yml config/ \   && yum -y install logrotate crontabs   COPY supervisord.conf /etc/supervisord.conf  RUN mkdir -p /var/log/supervisor EXPOSE 22 80 CMD ["/usr/bin/supervisord"]


因为需要使用logrotate进行日志轮转,需要安装

logrotate crontabs
supervisord.conf配置如下


[supervisord] nodaemon=true  [program:cron] command=/usr/sbin/crond -i  [program:filebeat] command=/usr/local/filebeat/filebeat -c /usr/local/filebeat/config/filebeat.yml



3、修改原有ingress controller depl,将filebeat与ingress controller放到同一pod中,使用emptydir卷共享ingress日志,使filebeat能够读取,另外一个是面对日志的持续正常如何处理,这里使用logrotate,将logrotate在filebeat中配置,尽量对ingress影响小点,首先增加filebeat configmap



kind: ConfigMap
apiVersion: v1
metadata:
name: filebeat-config
data:
filebeat.yml: |
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
json.add_error_key: true
json.ignore_decoding_error: true
tags: ["access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
json.keys_under_root: true
json.overwrite_keys: true
json.add_error_key: true
json.ignore_decoding_error: true
tags: ["error"]
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
output.elasticsearch:
hosts: ["es-local.nxgp.svc.cluster.local:9200"]
index: "nginx_log-%{+yyyy.MM.dd}"
indices:
- index: "nginx_access-%{[beat.version]}-%{+yyyy.MM.dd}"
when.contains:
tags: "access"
- index: "nginx_error-%{[beat.version]}-%{+yyyy.MM.dd}"
when.contains:
tags: "error"
setup.template.name: "nginx_log"
setup.template.pattern: "nginx_*"
setup.template.enabled: true
setup.ilm.enabled: false
setup.template.overwrite: false




kind: ConfigMap apiVersion: v1 metadata:   name: nginx-ingress-logrotate data:   nginx: |     /var/log/nginx/*.log {       su root root       size 50M       notifempty       copytruncate       rotate 3       missingok       compress       dateext       dateformat .%Y%m%d-%H   }


然后进行depl更新,只展示新增部分


volumes:         - name: ingress-log           emptyDir: {}         - name: filebeat-config           configMap:             name: filebeat-config             defaultMode: 420         - name: logrotateconf           configMap:             name: nginx-ingress-logrotate             items:               - key: nginx                 path: nginx             defaultMode: 420       containers:         - name: controller           volumeMounts:             - name: ingress-log               mountPath: /var/log/nginx/         - name: filebeat           image: 'xxx/filebeat:7.5.0'           resources:             limits:               cpu: '2'               memory: 2Gi             requests:               cpu: '1'               memory: 1Gi           volumeMounts:             - name: filebeat-config               mountPath: /usr/local/filebeat/config/             - name: ingress-log               mountPath: /var/log/nginx/             - name: logrotateconf               mountPath: /etc/logrotate.d/nginx               subPath: nginx           terminationMessagePath: /dev/termination-log           terminationMessagePolicy: File           imagePullPolicy: Always       restartPolicy: Always       terminationGracePeriodSeconds: 300       dnsPolicy: ClusterFirst       nodeSelector:         kubernetes.io/os: linux       serviceAccountName: ingress-nginx       serviceAccount: ingress-nginx       securityContext: {}       schedulerName: default-scheduler   strategy:     type: RollingUpdate     rollingUpdate:       maxUnavailable: 25%       maxSurge: 25%   revisionHistoryLimit: 10   progressDeadlineSeconds: 600




logrotate是按天更新,更新时间不定



轮转效果内存占用高的是还没有进行打包的







4、es可视化展示

(1)PV

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_json_02

(2)UV

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_json_03

(3)Top10(接口访问量)

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_状态码_04k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_nginx_05

(4)Top10(客户端IP访问占比)

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_nginx_06k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_linux_07

(5)Top10(最慢接口)

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_nginx_08k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_状态码_09

(6)后端upstream占比

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_json_10k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_nginx_11

(7)实时流量

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_json_12k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_linux_13

(8)客户端访问占比

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_json_14k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_linux_15

(9)平均并发数

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_linux_16k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_nginx_17

(10)异常状态码统计

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_nginx_18k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_客户端_19k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_状态码_20


(11)总流量

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_客户端_21

(12)接口异常响应码

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_客户端_22k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_linux_23k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_json_24

(13)接口访问耗时占比

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_json_25k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_linux_26k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_客户端_27

(14)每10秒接口访问平均耗时

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_linux_28k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_json_29

(15)每10秒接口访问最大耗时

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_nginx_30k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_客户端_31

(16)状态码统计

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_客户端_32k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_json_33

(17)访问量趋势图

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_状态码_34k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_linux_35

(18)超过30秒以上的接口

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_nginx_36k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_客户端_37

(19)超过30秒以上的接口出现次数

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_客户端_38k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_状态码_39k8s filebeat sidecar模式收集ingress nginx日志并可视化展示_客户端_40