编写docker-compose文件配置zk 和kafka集群

#vim kafka.yml


version: '2'
services:
    zoo1:
        image: zookeeper
        restart: always
        container_name: zoo1
        ports:
            - "2181:2181"
        volumes:
            - "/data/zookeeper/zookeeper1/data:/data"
            - "/data/zookeeper/zookeeper1/datalog:/datalog"
        environment:
            ZOO_MY_ID: 1
            ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888

    zoo2:
        image: zookeeper
        restart: always
        container_name: zoo2
        ports:
            - "2182:2181"
        volumes:
            - "/data/zookeeper/zookeeper2/data:/data"
            - "/data/zookeeper/zookeeper2/datalog:/datalog"
        environment:
            ZOO_MY_ID: 2
            ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888

    zoo3:
        image: zookeeper
        restart: always
        container_name: zoo3
        ports:
            - "2183:2181"
        volumes:
            - "/data/zookeeper/zookeeper3/data:/data"
            - "/data/zookeeper/zookeeper3/datalog:/datalog"
        environment:
            ZOO_MY_ID: 3
            ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888

    broker1:
        image: wurstmeister/kafka
        restart: always
        container_name: broker1
        ports:
          - "9091:9092"
        depends_on:
          - zoo1
          - zoo2
          - zoo3
        environment:
          KAFKA_BROKER_ID: 1
          KAFKA_ADVERTISED_HOST_NAME: broker1
          KAFKA_ADVERTISED_PORT: 9092
          KAFKA_HOST_NAME: broker1
          KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
          KAFKA_LISTENERS: PLAINTEXT://broker1:9092
          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker1:9092
        volumes:
          - /var/run/docker.sock:/var/run/docker.sock
          - "/data/kafka/kafka1/:/kafka"

    broker2:
        image: wurstmeister/kafka
        restart: always
        container_name: broker2
        ports:
          - "9092:9092"
        depends_on:
          - zoo1
          - zoo2
          - zoo3
        environment:
          KAFKA_BROKER_ID: 2
          KAFKA_ADVERTISED_HOST_NAME: broker2
          KAFKA_ADVERTISED_PORT: 9092
          KAFKA_HOST_NAME: broker2
          KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
          KAFKA_LISTENERS: PLAINTEXT://broker2:9092
          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker2:9092
        volumes:
          - /var/run/docker.sock:/var/run/docker.sock
          - "/data/kafka/kafka2/:/kafka"

    broker3:
        image: wurstmeister/kafka
        restart: always
        container_name: broker3
        ports:
          - "9093:9092"
        depends_on:
          - zoo1
          - zoo2
          - zoo3
        environment:
          KAFKA_BROKER_ID: 3
          KAFKA_ADVERTISED_HOST_NAME: broker3
          KAFKA_ADVERTISED_PORT: 9092
          KAFKA_HOST_NAME: broker3
          KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
          KAFKA_LISTENERS: PLAINTEXT://broker3:9092
          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker3:9092
        volumes:
          - /var/run/docker.sock:/var/run/docker.sock
          - "/data/kafka/kafka3/:/kafka"

运行compose文件

#COMPOSE_PROJECT_NAME=kafkatest docker-compose -f kafka.yml up -d

查看docker启动进程

[root@localhost ~]# docker ps

zk 和kafka已启动

granfana监控docker容器模板 docker监控工具_docker


kafka-manager管理工具安装


#vim kafka_manager.yml

version: '3.1'
services:
  kafka-manager:
    container_name: kafka-manager
    image: sheepkiller/kafka-manager
    restart: always
    ports:
      - 9000:9000
    environment:
      KM_VERSION: 1.3.3.18
      ZK_HOSTS: zoo1:2181,zoo2:2182,zoo3:2183

启动docker容器

docker-compsoe -f kafka-manager.yml up -d

docker启动后，登陆到docker容器里

  #vim conf/application.conf


    kafka-manager.zkhosts="zoo1:2181,zoo2:2182,zoo3:2183"

    #kafka-manager.zkhosts=${?ZK_HOSTS}

将配置文件注掉一条，然后将zk的地址写入，不然界面是出不来的

以上是出现如下错误的解决方法

Yikes! Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [5000 ms]

kafka-manager不支持硬盘挂载，我也被坑了好久，


kafkaoffsetmonitor监控工具安装配置

#vim kafkaoffsetmonitor.yml


vim kafkaoffsetmonitor.yml
version: '2'
services:
  kafka-offset-monitor:
    image: 564239555/kafkaoffsetmonitor
    volumes:
      - /data/kafkaoffsetmonitor/conf:/kafkaoffsetmonitor
    ports:
      - "8089:8089"
    environment:
     ZK_HOSTS: zoo1:2181,zoo2:2182,zoo3:2183
     KAFKA_BROKERS: 172.16.70.51:9091,172.16.70.51:9092,172.16.70.51:90093
     REFRESH_SECENDS: 10
     RETAIN_DAYS: 2

启动镜像

docker-compose -f kafkaoffsetmonitor.yml up -d 

kafkaoffsetmonitor需要一个文件来启动，KafkaOffsetMonitor-assembly-0.2.1.jar，我已将些文件放到镜像中

或者自行下载这个软件，将KafkaOffsetMonitor-assembly-0.2.1.jar 拷贝到docker 容器里，

下载地址


wget https://github.com/quantifind/KafkaOffsetMonitor/releases/download/v0.2.1/KafkaOffsetMonitor-assembly-0.2.1.jar


docker cp /data/kafkaoffsetmonitor/conf/KafkaOffsetMonitor-assembly-0.2.1.jar

写个启动脚本

#vim /kafkaoffsetmonitor/start.sh

#!/bin/bash
nohup java -cp KafkaOffsetMonitor-assembly-0.2.1.jar com.quantifind.kafka.offsetapp.OffsetGetterWeb --zk zoo1:2181,zoo2:2182,zoo3:2183
如果启动报错，将--zk 改成宿主机IP

给执行权限然后启动脚本 

#chmod +x /kafkaoffsetmonitor/start.sh
#sh /kafkaoffsetmonitor/start.sh

看到8089的端口启动，就可以通过浏览器http:IP:8089去访问

网上很多镜像没有这个文件，启动就报错

或者直接下载我的镜像文件

https://hub.docker.com/r/564239555/kafkaoffsetmonitor/

docker pull 564239555/kafkaoffsetmonitor

docker启动后，登陆到docker容器里，执行脚本，看到8089的端口启动就可以通过浏览器去访问了http://宿主机IP:8089

#docker exec -it $dockerCONTAINER ID bash

修改docker启动所需要的脚本文件

#cd /kafkaoffsetmonitor/conf/
需要将zk更改成你自已集群的IP 地址
#./kafkaoffsetmonitor.sh
#ss -tunl





转载于:https://blog.51cto.com/536410/2322056