log-pilot：k8s中日志收集神器

k8s中pod的日志收集有2种常见的解决方案；

• 方案一：使用fluentd作为daemonset收集stdout和/var/lib/containers目录下的所有日志（因为对fluentd不太熟悉，所以感觉麻烦）；

• 方案二：使用filebeat作为sidecar方式（这种方式过于繁琐，需要在每个pod中添加这个容器）

无意间发现阿里云开源的log-pilot收集k8s的日志真的超级方便，配置也简单；

官方介绍：

github地址：https://github.com/AliyunContainerService/log-pilot

log-pilot官方介绍：https://yq.aliyun.com/articles/674327

log-pilot官方搭建：https://yq.aliyun.com/articles/674361?spm=a2c4e.11153940.0.0.21ae21c3mTKwWS

log-pilot的daemonset文件：

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: log-pilot
  labels:
    app: log-pilot
  # 设置期望部署的namespace
  namespace: kube-system
spec:
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: log-pilot
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      # 是否允许部署到Master节点上
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: log-pilot
        # 版本请参考https://github.com/AliyunContainerService/log-pilot/releases
        image: registry.cn-hangzhou.aliyuncs.com/acs/log-pilot:0.9.7-filebeat
        resources:
          limits:
            memory: 500Mi
          requests:
            cpu: 200m
            memory: 200Mi
        env:
          - name: "NODE_NAME"
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: "LOGGING_OUTPUT"
            value: "elasticsearch"
          # 请确保集群到ES网络可达
          - name: "ELASTICSEARCH_HOSTS"
            value: "10.10.5.78:9200"
          # 配置ES访问权限
          #- name: "ELASTICSEARCH_USER"
          #  value: "{es_username}"
          #- name: "ELASTICSEARCH_PASSWORD"
          #  value: "{es_password}"
        volumeMounts:
        - name: sock
          mountPath: /var/run/docker.sock
        - name: root
          mountPath: /host
          readOnly: true
        - name: varlib
          mountPath: /var/lib/filebeat
        - name: varlog
          mountPath: /var/log/filebeat
        - name: localtime
          mountPath: /etc/localtime
          readOnly: true
        livenessProbe:
          failureThreshold: 3
          exec:
            command:
            - /pilot/healthz
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        securityContext:
          capabilities:
            add:
            - SYS_ADMIN
      terminationGracePeriodSeconds: 30
      volumes:
      - name: sock
        hostPath:
          path: /var/run/docker.sock
      - name: root
        hostPath:
          path: /
      - name: varlib
        hostPath:
          path: /var/lib/filebeat
          type: DirectoryOrCreate
      - name: varlog
        hostPath:
          path: /var/log/filebeat
          type: DirectoryOrCreate
      - name: localtime
        hostPath:
          path: /etc/localtime

创建nginx测试pod收集日志示例：

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: node-affinity
spec:
  selector:
    matchLabels:
      app: node-affinity
  replicas: 3
  template:
    metadata:
      labels:
        app: node-affinity
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: IfNotPresent
        env:
        - name: aliyun_logs_nginx
          value: "stdout"
---
apiVersion: v1
kind: Service
metadata:
  name: node-affinity
spec:
  selector:
    app: node-affinity
  ports:
  - port: 80
    targetPort: 80
  type: NodePort

创建tomcat测试pod收集日志示例：

apiVersion: v1
kind: Pod
metadata:
  name: tomcat
spec:
  containers:
  - name: tomcat
    image: "tomcat:8.0"
    env:
    # 1、stdout为约定关键字，表示采集标准输出日志
    # 2、配置标准输出日志采集到ES的catalina索引下
    - name: aliyun_logs_catalina
      value: "stdout"
    # 1、配置采集容器内文件日志，支持通配符
    # 2、配置该日志采集到ES的access索引下
    - name: aliyun_logs_access
      value: "/usr/local/tomcat/logs/catalina.*.log"
    # 容器内文件日志路径需要配置emptyDir
    volumeMounts:
      - name: tomcat-log
        mountPath: /usr/local/tomcat/logs
  volumes:
    - name: tomcat-log
      emptyDir: {}