k8s原生的集群监控方案(Heapster+InfluxDB+Grafana)

  1. Heapster+InfluxDB+Grafana简介
    heapster是一个监控计算、存储、网络等集群资源的工具,以k8s内置的cAdvisor作为数据源收集集群信息,并汇总出有价值的性能数据(Metrics):cpu、内存、network、filesystem等,然后将这些数据输出到外部存储(backend),如InfluxDB,最后再通过相应的UI界面进行可视化展示,如grafana。 另外heapster的数据源和外部存储都是可插拔的,所以可以很灵活的组建出很多监控方案,如:Heapster+ElasticSearch+Kibana等等。
  2. Heapster的整体架构图
    k8s原生的集群监控方案(Heapster+InfluxDB+Grafana)
  3. 创建InfluxDB资源对象
    #下载influxdb.yaml
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
        name: monitoring-influxdb
        namespace: kube-system
    spec:
        replicas: 1
        template:
            metadata:
                labels:
                    task: monitoring
                    k8s-app: influxdb
            spec:
                containers:
                - name: influxdb
                    image: k8s.gcr.io/heapster-influxdb-amd64:v1.3.3
                    volumeMounts:
                    - mountPath: /data
                        name: influxdb-storage
                volumes:
                - name: influxdb-storage
                    emptyDir: {}
    ---
    apiVersion: v1
    kind: Service
    metadata:
        labels:
            task: monitoring
            #For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
            #If you are NOT using this as an addon, you should comment out this line.
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: monitoring-influxdb
        name: monitoring-influxdb
        namespace: kube-system
    spec:
        type: NodePort
        ports:
        - nodePort: 31001
            port: 8086
            targetPort: 8086
        selector:
            k8s-app: influxdb

    所需的Heapster+InfluxDB+Grafana配置文件,请在Kubernetes Dashboard1.8.3部署中的yaml链接中下载使用。

    #influxdb.yaml文件需更改的地方:
    (1) image: k8s.gcr.io/heapster-influxdb-amd64:v1.3.3 (换成自己的images)
    ##说明:这里我在前文中提供的有images下载链接,直接下载使用不用更改!
    (2)这里我们使用NotePort暴露monitoring-influxdb服务在主机的31001端口上,那么InfluxDB服务端的地址:http://[host-ip]:31001 ,记下这个地址,以便创建heapster和为grafana配置数据源时,可以直接使用。
    spec:
        type: NodePort
        ports:
            - nodePort: 31001
                port: 8086
                targetPort: 8086
            selector:
                k8s-app: influxdb
  4. 创建Grafana资源对象
    #下载grafana.yaml
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
        name: monitoring-grafana
        namespace: kube-system
    spec:
        replicas: 1
        template:
            metadata:
                labels:
                    task: monitoring
                    k8s-app: grafana
            spec:
                containers:
                - name: grafana
                    image: k8s.gcr.io/heapster-grafana-amd64:v4.4.3
                    ports:
                    - containerPort: 3000
                        protocol: TCP
                    volumeMounts:
                    - mountPath: /etc/ssl/certs
                        name: ca-certificates
                        readOnly: true
                    - mountPath: /var
                        name: grafana-storage
                    env:
                    - name: INFLUXDB_HOST
                        value: monitoring-influxdb
                    - name: GF_SERVER_HTTP_PORT
                        value: "3000"
                        #The following env variables are required to make Grafana accessible via
                        #the kubernetes api-server proxy. On production clusters, we recommend
                        #removing these env variables, setup auth for grafana, and expose the grafana
                        #service using a LoadBalancer or a public IP.
                    - name: GF_AUTH_BASIC_ENABLED
                        value: "false"
                    - name: GF_AUTH_ANONYMOUS_ENABLED
                        value: "true"
                    - name: GF_AUTH_ANONYMOUS_ORG_ROLE
                        value: Admin
                    - name: GF_SERVER_ROOT_URL
                        #If you're only using the API Server proxy, set this value instead:
                        #value: /api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
                        value: /
                volumes:
                - name: ca-certificates
                    hostPath:
                        path: /etc/ssl/certs
                - name: grafana-storage
                    emptyDir: {}
    ---
    apiVersion: v1
    kind: Service
    metadata:
        labels:
            #For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
            #If you are NOT using this as an addon, you should comment out this line.
            kubernetes.io/cluster-service: 'true'
            kubernetes.io/name: monitoring-grafana
        name: monitoring-grafana
        namespace: kube-system
    spec:
        #In a production setup, we recommend accessing Grafana through an external Loadbalancer
        #or through a public IP.
        #type: LoadBalancer
        #You could also use NodePort to expose the service at a randomly-generated port
        #type: NodePort
        type: NodePort
        ports:
        - nodePort: 30108
            port: 80
            targetPort: 3000
        selector:
            k8s-app: grafana
    ##说明

    虽然Heapster已经预先配置好了Grafana的Datasource和Dashboard,但是为了方便访问,这里我们使用NotePort暴露monitoring-grafana服务在主机的30108上,那么Grafana服务端的地址:http://192.168.245.16:30108 ,通过浏览器访问,为Grafana修改数据源,如下:
    k8s原生的集群监控方案(Heapster+InfluxDB+Grafana)
    标红的地方,为上一步记录下的InfluxDB服务端的地址。

  5. 创建Heapster资源对象
    #下载heapster-rbac.yaml  
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
        name: heapster
    roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: system:heapster
    subjects:
    - kind: ServiceAccount
        name: heapster
        namespace: kube-system
    #下载heapster.yaml  
    apiVersion: v1
    kind: ServiceAccount
    metadata:
        name: heapster
        namespace: kube-system
    ---
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
        name: heapster
        namespace: kube-system
    spec:
        replicas: 1
        template:
            metadata:
                labels:
                    task: monitoring
                    k8s-app: heapster
            spec:
                serviceAccountName: heapster
                containers:
                - name: heapster
                    image: k8s.gcr.io/heapster-amd64:v1.5.3
                    imagePullPolicy: IfNotPresent
                    command:
                    - /heapster
                    - --source=kubernetes:https://kubernetes.default
                #- --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086
                    - --sink=influxdb:http://192.168.246.167:31001 #influxdb服务端地址
    ---
    apiVersion: v1
    kind: Service
    metadata:
        labels:
            task: monitoring
            #For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
            #If you are NOT using this as an addon, you should comment out this line.
            kubernetes.io/cluster-service: 'true'
            kubernetes.io/name: Heapster
        name: heapster
        namespace: kube-system
    spec:
        ports:
        - port: 80
            targetPort: 8082
        selector:
            k8s-app: heapster
    ##说明

    (1)
    --source 为heapster指定获取集群信息的数据源。参考:https://github.com/kubernetes/heapster/blob/master/docs/source-configuration.md
    --sink 为heaster指定后端存储,这里我们使用InfluxDB,其他的,请参考:https://github.com/kubernetes/heapster/blob/master/docs/sink-owners.md
    (2)heapster-rbac.yaml 文件作用
    如没有heapster-rbac.yaml 将导致权限的问题,heaster默认使用一个令牌(Token)与ApiServer进行认证,通过查看heapster.yml发现 serviceAccountName: heapster ,现在明白了吧,就是heaster没有权限,那么如何授权呢-----给heaster绑定一个有权限的角色就行了,即heapster-rbac.yaml配置的那样!

  6. 通过dashboard查看集群概况
    k8s原生的集群监控方案(Heapster+InfluxDB+Grafana)
    k8s原生的集群监控方案(Heapster+InfluxDB+Grafana)
  7. 通过Grafana查看集群详情(cpu、memory、filesystem、network)
    k8s原生的集群监控方案(Heapster+InfluxDB+Grafana)k8s原生的集群监控方案(Heapster+InfluxDB+Grafana)