grafana agent 集成tempo 试用

很简单就是几个工具椽到一起,方便学习了解grafana agent 对于tempo的支持

预备说明

grafana agent 对于tempo的支持,我们需要使用opentelemetry-collector,关于opentelemetry-collector 的使用
以及功能可以参考相关资料

环境准备

  • 系统参考架构

grafana agent 集成tempo 试用_分享

 

 

  • docker-compose
 
version: "3"
services:
  tempo:
    image: grafana/tempo:latest
    command: -config.file=/etc/tempo.yaml
    volumes:
    - ./etc/tempo.yaml:/etc/tempo.yaml
    ports:
      - "14268:14268"  # jaeger ingest
      - "3300:3300"
    logging:
      driver: loki
      options:
        loki-url: 'http://localhost:3100/api/prom/push'
  tempo-query:
    image: grafana/tempo-query:latest
    command: ["--grpc-storage-plugin.configuration-file=/etc/tempo-query.yaml"]
    volumes:
      - ./etc/tempo-query.yaml:/etc/tempo-query.yaml
    ports:
      - "16686:16686"  # jaeger-ui
    logging:
      driver: loki
      options:
        loki-url: 'http://localhost:3100/api/prom/push'
  otel-collector:
    image: otel/opentelemetry-collector-dev:latest
    volumes:
    - ./etc/otel-collector-config.yaml:/etc/otel-collector/config.yaml
    command: --config=/etc/otel-collector/config.yaml
    logging:
      driver: loki
      options:
        loki-url: 'http://localhost:3100/api/prom/push'
  grafana-agent:
    image: grafana/agent:v0.9.1
    volumes:
      - ./etc/agent2.yaml:/etc/agent-config/agent.yaml
    command:  -config.file=/etc/agent-config/agent.yaml   --prometheus.wal-directory=/tmp/agent/wal
    ports:
      -  "12345:12345"
    logging:
      driver: loki
      options:
        loki-url: 'http://localhost:3100/api/prom/push'
  victoriametrics:
    image: victoriametrics/victoria-metrics
    volumes:
      - ./example-data/victoriametrics:/victoriametrics
    ports:
      - "8428:8428"
    command: 
      - '-storageDataPath=/victoriametrics'
      - '-retentionPeriod=1'
    logging:
      driver: loki
      options:
        loki-url: 'http://localhost:3100/api/prom/push'
  synthetic-load-generator:
    image: omnition/synthetic-load-generator:1.0.25
    volumes:
      - ./etc/load-generator.json:/etc/load-generator.json
    environment:
      - TOPOLOGY_FILE=/etc/load-generator.json
      - JAEGER_COLLECTOR_URL=http://grafana-agent:14268
    logging:
      driver: loki
      options:
        loki-url: 'http://localhost:3100/api/prom/push'
  grafana:
    image: grafana/grafana:7.3.6
    volumes:
      - ./example-data/datasources:/etc/grafana/provisioning/datasources
      - ./example-data/dashboards-provisioning:/etc/grafana/provisioning/dashboard
    ports:
      - "3000:3000"
    logging:
      driver: loki
      options:
        loki-url: 'http://localhost:3100/api/prom/push'
  loki:
    image: grafana/loki:2.1.0
    command: -config.file=/etc/loki/local-config.yaml
    ports:
      - "3100:3100"                                
    environment:
      - JAEGER_AGENT_HOST=tempo
      - JAEGER_ENDPOINT=http://tempo:14268/api/traces # send traces to Tempo
      - JAEGER_SAMPLER_TYPE=const
      - JAEGER_SAMPLER_PARAM=1
    logging:
      driver: loki
      options:
        loki-url: 'http://localhost:3100/api/prom/push'
  • 依赖组件说明
    loki log 处理,grafana dashborad 展示,synthetic-load-generator 生成模拟数据,victoriametrics prometheus
    remote storage,opentelemetry-collector agent 集成tempo 的一个依赖,grafane agent 我们学习的主角,tempo
    agent集成tempo的依赖
  • 配置说明
    主要介绍关于核心配置的,具体可以参考demo
    tempo 配置,主要是关于tempo 几个核心组件的配置,我们暴露了一些常用trace 的服务
 
auth_enabled: false
server:
  http_listen_port: 3300
distributor:
  receivers:                           # this configuration will listen on all ports and protocols that tempo is capable of.
    jaeger:                            # the receives all come from the OpenTelemetry collector.  more configuration information can
      protocols:                       # be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/master/receiver
        thrift_http:                   #
        grpc:                          # for a production deployment you should only enable the receivers you need!
        thrift_binary:
        thrift_compact:
    zipkin:
    otlp:
      protocols:
        http:
        grpc:
    opencensus:
ingester:
  trace_idle_period: 10s               # the length of time after a trace has not received spans to consider it complete and flush it
  traces_per_block: 100                # cut the head block when it his this number of traces or ...
  max_block_duration: 5m               #   this much time passes
compactor:
  compaction:
    compaction_window: 1h              # blocks in this time window will be compacted together
    max_compaction_objects: 1000000    # maximum size of compacted blocks
    block_retention: 1h
    compacted_block_retention: 10m
storage:
  trace:
    backend: local                     # backend configuration to use
    wal:
      path: /tmp/tempo/wal             # where to store the the wal locally
      bloom_filter_false_positive: .05 # bloom filter false positive rate.  lower values create larger filters but fewer false positives
      index_downsample: 10             # number of traces per index record
    local:
      path: /tmp/tempo/blocks
    pool:
      max_workers: 100                 # the worker pool mainly drives querying, but is also used for polling the blocklist
      queue_depth: 10000

agent 配置,agent 是我们的核心部分,集成了metrics,log,trace

server:
  log_level: debug
  http_listen_port: 12345
# metrics 部分
prometheus:
  global:
    scrape_interval: 5s
  configs:
    - name: default
      scrape_configs:
        - job_name: agent
          static_configs:
            - targets: ['127.0.0.1:12345']
        - job_name: victoriametrics
          static_configs:
            - targets: ['victoriametrics:8428']
        - job_name: tempo
          static_configs:
            - targets: ['tempo:3300']
              labels:
                cluster: 'tempo'
                container: 'tempo'
      remote_write:
        - url: http://victoriametrics:8428/api/v1/write
#  log
loki:
  positions:
    filename: /tmp/positions.yaml
  scrape_configs:
    - job_name: varlogs
      static_configs:
        - targets: [localhost]
          labels:
            job: varlogs
            __path__: /var/log/*log
  clients:
    - url: http://loki:3100/loki/api/v1/push
# 集成temp 的trace
tempo:
  receivers:
    jaeger:
      protocols:
        thrift_http: 
  attributes:
    actions:
    - action: upsert
      key: env
      value: prod
## 需要依赖opentelemetry-collector
  push_config:
    endpoint: otel-collector:55680 # 
    insecure: true
    batch:
      timeout: 5s
      send_batch_size: 100

opentelemetry-collector
主要是opentelemetry-collector 标配组件的配置,receivers,exporters,pipeline (services)

 
receivers:
  otlp:
    protocols:
      grpc:
exporters:
  jaeger:
    endpoint: tempo:14250
    insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [jaeger]

运行说明

  • 首先安装docker loki driver
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
  • 启动
docker-compose up -d
  • 效果

grafana agent 集成tempo 试用_分享_02

 

 

说明

grafana 的agent 是一个不错的选择,集成的功能很多

参考资料

https://github.com/rongfengliang/grafana-tempo-learning
https://github.com/grafana/agent/blob/master/docs/configuration-reference.md
https://github.com/grafana
https://github.com/grafana/tempo
https://github.com/rongfengliang/grafana-tempo-learning
https://github.com/open-telemetry/opentelemetry-collector
https://grafana.com/docs/tempo/latest/getting-started/
https://grafana.com/docs/grafana/latest/datasources/loki/
https://github.com/open-telemetry/opentelemetry-specification/blob/master/experimental/trace/zpages.md
https://grafana.com/docs/loki/latest/