1 为什么选择loki
https://github.com/grafana/loki
Loki 是 Grafana Labs 团队的开源项目,是一个水平可扩展,高可用性,多租户的日志聚合系统。它的设计非常经济高效且易于操作,因为它不会为日志内容设置索引,而是为每个日志流设置一组标签(就像 Prometheus 标签一样),官方的介绍就是:Like Prometheus, but for logs.,类似于 Prometheus 的日志系统
架构如下
- Promtail:Loki 默认客户端,负责采集并上报日志
- Distributor:日志写入入口,负责将数据转发到 Ingester
- Ingester:日志的写入服务,缓存并写入日志内容和索引到底层存储
- Querier:日志读取服务,执行搜索请求
Grafana Loki 主要由 3 部分组成:
- Loki:主服务,日志记录引擎,负责存储日志和处理查询
https://github.com/grafana/loki
https://grafana.com/docs/loki/latest/installation/
- Promtail:代理服务,负责收集日志并将其发送给 loki
https://grafana.com/docs/loki/latest/clients/promtail/installation/
- Grafana:UI 界面,通过 Web 界面来提供数据展示、查询、告警等功能。
2 安装部署
如果没有安装过 Helm3
请安装
# 下载二进制包
$ wget https://get.helm.sh/helm-v3.5.2-linux-amd64.tar.gz
# 解压二进制包
$ tar zxf helm-v3.5.2-linux-amd64.tar.gz
# 复制可执行文件
$ cp -pr linux-amd64/helm /usr/local/bin/
# 验证安装是否成功
$ helm --help
2.1 部署 Loki
https://grafana.com/docs/loki/latest/installation/simple-scalable-helm/
使用 Helm 将 Loki 部署到 K8s 集群
# 添加 Loki 的 Chart 仓库
helm repo add grafana https://grafana.github.io/helm-charts
# 更新仓库
helm repo update
# 搜索 loki(Helm2:helm search loki)
root@tjdata05:~# helm search repo loki
NAME CHART VERSION APP VERSION DESCRIPTION
grafana/loki 2.12.2 v2.5.0 Loki: like Prometheus, but for logs.
grafana/loki-canary 0.8.1 2.5.0 Helm chart for Grafana Loki Canary
grafana/loki-distributed 0.49.0 2.5.0 Helm chart for Grafana Loki in microservices mode
grafana/loki-simple-scalable 1.4.1 2.5.0 Helm chart for Grafana Loki in simple, scalable...
grafana/loki-stack 2.6.5 v2.4.2 Loki: like Prometheus, but for logs.
grafana/fluent-bit 2.3.1 v2.1.0 Uses fluent-bit Loki go plugin for gathering lo...
grafana/promtail 6.0.0 2.5.0 Promtail is an agent which ships the contents o...
root@tjdata05:~#
grafana/loki
只有单独的 loki 组件grafana/loki-canary
loki 的另外一个组件,主要给 loki 做性能分析grafana/loki-distributed
分布式(微服务)模式grafana/loki-stack
loki 的工具栈grafana/fluent-bit
日志收集客户端grafana/promtail
loki 自带的日志收集客户端
获取 loki-stack 的 Chart 包并解压:
# Helm3
helm pull grafana/loki-stack --untar --version 2.6.5 # 2.6.5 是 CHART VERSION
# 如果是 Helm2
helm fetch grafana/loki-stack --untar --version 2.6.5
loki-stack 这个 Chart 包里面包含所有的 Loki 相关工具依赖,在安装的时候可以根据需要开启或关闭。默认情况下 loki、promtail 是自动开启的,也可以根据需要选择使用 filebeat 或者 logstash,同样在 Chart 包根目录下自定义安装的 Values 文件:
root@tjdata05:~/loki-stack# cat values-prod.yaml
loki:
enabled: true
replicas: 1
persistence:
enabled: true
storageClassName: ceph-rbd
accessModes:
- ReadWriteOnce
size: 80Gi
promtail:
enabled: true
grafana:
enabled: true
service:
type: ClusterIP
port: 80
targetPort: 3000
persistence:
enabled: true
storageClassName: ceph-rbd
accessModes:
- ReadWriteOnce
size: 10Gi
ingress:
enabled: true
path: /
# pathType is only for k8s >= 1.1=
pathType: Prefix
hosts:
- loki.test.cn
root@tjdata05:~/loki-stack#
然后直接使用上面的 values-prod.yaml
文件进行安装即可
root@tjdata05:~/loki-stack# kubectl create ns logging
namespace/logging created
root@tjdata05:~/loki-stack# helm upgrade --install loki -n logging -f values-prod.yaml .
Release "loki" does not exist. Installing it now.
NAME: loki
LAST DEPLOYED: Fri Jun 24 17:47:04 2022
NAMESPACE: logging
STATUS: deployed
REVISION: 1
NOTES:
The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.
See http://docs.grafana.org/features/datasources/loki/ for more detail.
root@tjdata05:~/loki-stack#
安装完成后可以查看 Pod 的状态
root@tjdata05:~# kubectl -n logging get pods
NAME READY STATUS RESTARTS AGE
loki-0 1/1 Running 0 4m32s
loki-grafana-7595cff4f5-4szxs 2/2 Running 0 4m31s
loki-promtail-2n28d 1/1 Running 0 4m32s
loki-promtail-46q2l 1/1 Running 0 4m32s
loki-promtail-6fjfm 1/1 Running 0 4m32s
loki-promtail-8fzzp 1/1 Running 0 4m32s
loki-promtail-8ndgj 1/1 Running 0 4m32s
loki-promtail-h7nfx 1/1 Running 0 4m32s
loki-promtail-nn25b 1/1 Running 0 4m32s
loki-promtail-r7nmv 1/1 Running 0 4m32s
loki-promtail-xbxpg 1/1 Running 0 4m32s
root@tjdata05:~#
查看 Service 信息
root@tjdata05:~# kubectl -n logging get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
loki ClusterIP 10.68.124.157 <none> 3100/TCP 4m9s
loki-grafana ClusterIP 10.68.25.148 <none> 80/TCP 4m9s
loki-headless ClusterIP None <none> 3100/TCP 4m9s
loki-memberlist ClusterIP None <none> 7946/TCP 4m9s
root@tjdata05:~#
查看 ingress 信息
root@tjdata05:~# kubectl -n logging get ing
NAME HOSTS ADDRESS PORTS AGE
loki-grafana loki.test.cn 80 5m6s
root@tjdata05:~#
查看 pvc 信息
root@tjdata05:~# kubectl -n logging get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
loki-grafana Bound pvc-4186c1a5-584b-45c1-bc3c-c432c545907e 10Gi RWO ceph-rbd 7m40s
storage-loki-0 Bound pvc-1b6f111c-5292-4e54-b87c-45657d406500 80Gi RWO ceph-rbd 2d23h
root@tjdata05:~#
获取 Grafana 的登录密码
kubectl get secret --namespace logging loki-grafana -o jsonpath="{.data.admin-password}" | \
base64 --decode ; echo
# 示例
root@tjdata05:~# kubectl get secret --namespace logging loki-grafana -o jsonpath="{.data.admin-password}" | \
> base64 --decode ; echo
zWWmbfmHkhRBsJONZHiambvgn
root@tjdata05:~#
使用用户名 admin 和上面的获取的密码即可登录 Grafana,由于 Helm Chart 已经为 Grafana 配置好了 Loki 的数据源,所以我们可以直接获取到日志数据了。点击左侧 Explore 菜单,然后就可以筛选 Loki 的日志数据了:
https://loki.test.cn
user:admin
pass:zWWmbfmHkhRBsJONZHiambvgn
注意:
- Loki 默认显示 1000 行
- 为了方便查看,也可以导入 Dashboard
https://grafana.com/grafana/dashboards/15141
我们使用 Helm 安装的 Promtail 默认已经帮我们做好了配置,已经针对 K8s 做了优化,并且将 Promtail 配置进行了加密:
kubectl -n logging get secret loki-promtail -o yaml
# 解密
echo "" | base64 -d
内容如下
server:
log_level: info
http_listen_port: 3101
client:
url: http://loki:3100/loki/api/v1/push
positions:
filename: /run/promtail/positions.yaml
scrape_configs:
# See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
- job_name: kubernetes-pods
pipeline_stages:
- cri: {}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_controller_name
regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
action: replace
target_label: __tmp_controller_name
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- __meta_kubernetes_pod_label_app
- __tmp_controller_name
- __meta_kubernetes_pod_name
regex: ^;*([^;]+)(;.*)?$
action: replace
target_label: app
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
- __meta_kubernetes_pod_label_component
regex: ^;*([^;]+)(;.*)?$
action: replace
target_label: component
- action: replace
source_labels:
- __meta_kubernetes_pod_node_name
target_label: node_name
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
replacement: $1
separator: /
source_labels:
- namespace
- app
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
- action: replace
replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
- action: replace
regex: true/(.*)
replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
- __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
- __meta_kubernetes_pod_container_name
target_label: __path__
通过 api 查询 loki 中日志
root@tjdata05:~# kubectl -n logging get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
loki ClusterIP 10.68.124.157 <none> 3100/TCP 88m
loki-grafana ClusterIP 10.68.25.148 <none> 80/TCP 88m
loki-headless ClusterIP None <none> 3100/TCP 88m
loki-memberlist ClusterIP None <none> 7946/TCP 88m
root@tjdata05:~#
# 打印当前 loki 中所有的标签
root@tjdata05:~# curl 10.68.124.157:3100/loki/api/v1/labels
{"status":"success","data":["__name__","app","component","container","filename","job","namespace","node_name","pod"]}
# 打印标签(node_name)的值
root@tjdata05:~# curl 10.68.124.157:3100/loki/api/v1/label/node_name/values
{"status":"success","data": # 省略
# 打印标签(pod)的值
root@tjdata05:~# curl 10.68.124.157:3100/loki/api/v1/label/pod/values