一、简介

1.1、zk,kafka,kafka-manager简介

Zookeeper是一个开源的,分布式的,为分布式应用提供协调服务的Apache项
Kafka最初是由LinkedIn公司开发,是一个分布式、分区的、多副本的、多订阅者,基于zookeeper协调的分布式日志系统。Apache Kafka也是一个开源消息系统,由Scala写成;Kafka是一个分布式消息队列。Kafka对消息保存是根据Topic进行归类,发送消息者称为Producer,消息接受者称为Consumer,此外kafka集群有多个kafka实例组成,每个实例(server)称为broker。无论是kafka集群,还是consumer都依赖于zookeeper集群保存一些meta(生产者,消费者,broker,topic等等)信息,来保证系统可用性。kafka主要用做存储系统与消息系统
Kafka-manager是目前最主流的kafka管理工具。该工具可以方便查看集群 主题分布情况,同时支持对 多个集群的管理、分区平衡以及创建主题等操作。

1.2、部署简介

1.2.1、版本

zookeeper:kubernetes-zookeeper:1.0-3.4.10
kafka:kubernetes-kafka:1.0-10.2.1
Kafka-manager:1.3.3

1.2.2、部署

zookeeper:使用hostPath类型的存储卷,使用statefulset控制器部署zk集群,pod数量为3。
kafka:使用hostPath类型的存储卷,使用statefulset控制器部署kafka集群,pod数量为3,kafka需配置连接zk集群方能启动。
Kafka manager:用Deployment方式部署,定义类型为NodePort的service以供集群外管理,pod数量为1。

1.3、statefulset控制器简介

statefulset控制器是有状态应用副本集;在k8s集群statefulset用于管理以下副本集:

  • 稳定且唯一的网络标识符
  • 稳定且持久的存储
  • 有序,平滑的部署和扩展
  • 有序,平滑的删除和终止
  • 有序的滚动更新
    statefulset包含三个组件:headless service、StatefulSet,volumeClaimTemplate
  • headless service:确保解析名称直达后端pod
  • volumeClaimTemplate:卷申请模板,每创建一个pod时,自动申请一个pvc,从而请求绑定pv
  • StatefulSet:控制器

1.4、Deployment控制器简介

通过控制ReplicaSet来控制pod,是目前管理无状态应用最好的控制器;ReplicaSet是无状态pod控制器,当由标签选择器发现现有pod数量不到用户期望的数量时,利用资源清单模板新建pod。

二、zk部署

2.1、创建pv

kubernetes上的存储卷类型
emptyDir:空目录,按需创建;删除pod后存储卷中的数据也被删除,一般用于临时目录或缓存
hostPath:主机路径,在node上找一个目录与pod中的容器建立联系
gitRepo:基于emptyDir实现的git仓库 clone到emptyDir存储卷
NFS:将现有NFS(网络文件系统)共享安装到Pod中

2.1.1、准备

master节点:

kubectl create ns bigdata

node节点:

mkdir -pv /data/pv/zk{1..3}

2.1.2、定义pv资源文件

实验环境为虚拟机环境,因此使用的存储卷类型为hostPath。

apiVersion: v1
kind: PersistentVolume
metadata: 
  name: k8s-pv-zk1
  namespace: bigdata
  labels: 
    type: local
spec: 
  capacity: 
    storage: 2Gi
  accessModes: 
    - ReadWriteOnce
  hostPath: 
    path: /data/pv/zk1
  persistentVolumeReclaimPolicy: Recycle
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: k8s-pv-zk2
  namespace: bigdata
  labels:
    type: local
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /data/pv/zk2
  persistentVolumeReclaimPolicy: Recycle
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: k8s-pv-zk3
  namespace: bigdata
  labels:
    type: local
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /data/pv/zk3
  persistentVolumeReclaimPolicy: Recycle

2.1.3、创建pv

kubectl  apply  -f  zookeeper-pv.yaml

2.1.4、查看pv

[root@master ~]# kubectl get pv 
NAME         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
k8s-pv-zk1   2Gi        RWO            Recycle          Available                                   6m42s
k8s-pv-zk2   2Gi        RWO            Recycle          Available                                   6m42s
k8s-pv-zk3   2Gi        RWO            Recycle          Available                                   6m42s

2.2、zk集群部署

2.2.1、官方资源清单修改

官方资源清单:https://kubernetes.io/zh/docs/tutorials/stateful-application/zookeeper/

apiVersion: v1
kind: Service
metadata:
  name: zk-hs
  namespace: bigdata
  labels:
    app: zk
spec:
  ports:
  - port: 2888
    name: server
  - port: 3888
    name: leader-election
  clusterIP: None
  selector:
    app: zk
---
apiVersion: v1
kind: Service
metadata:
  name: zk-cs
  namespace: bigdata
  labels:
    app: zk
spec:
  ports:
  - port: 2181
    name: client
  selector:
    app: zk
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: zk-pdb
  namespace: bigdata
spec:
  selector:
    matchLabels:
      app: zk
  maxUnavailable: 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zk
  namespace: bigdata
spec:
  selector:
    matchLabels:
      app: zk
  serviceName: zk-hs
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: OrderedReady
  template:
    metadata:
      labels:
        app: zk
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - zk
              topologyKey: "kubernetes.io/hostname"
      containers:
      - name: kubernetes-zookeeper
        imagePullPolicy: IfNotPresent
        image: "mirrorgcrio/kubernetes-zookeeper:1.0-3.4.10"
        resources:
          requests:
            memory: "500Mi"
            cpu: "0.5"
        ports:
        - containerPort: 2181
          name: client
        - containerPort: 2888
          name: server
        - containerPort: 3888
          name: leader-election
        command:
        - sh
        - -c
        - "start-zookeeper \
          --servers=3 \
          --data_dir=/var/lib/zookeeper/data \
          --data_log_dir=/var/lib/zookeeper/data/log \
          --conf_dir=/opt/zookeeper/conf \
          --client_port=2181 \
          --election_port=3888 \
          --server_port=2888 \
          --tick_time=2000 \
          --init_limit=10 \
          --sync_limit=5 \
          --heap=512M \
          --max_client_cnxns=60 \
          --snap_retain_count=3 \
          --purge_interval=12 \
          --max_session_timeout=40000 \
          --min_session_timeout=4000 \
          --log_level=INFO"
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - "zookeeper-ready 2181"
          initialDelaySeconds: 10
          timeoutSeconds: 5
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - "zookeeper-ready 2181"
          initialDelaySeconds: 10
          timeoutSeconds: 5
        volumeMounts:
        - name: datadir
          mountPath: /var/lib/zookeeper
  volumeClaimTemplates:
  - metadata:
      name: datadir
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 2Gi

2.2.2、创建zk集群

kubectl  apply  -f  zookeeper.yaml

2.2.3、查看pod

[root@master ~]# kubectl get pods -n bigdata -o wide
NAME   READY   STATUS    RESTARTS   AGE    IP               NODE    NOMINATED NODE   READINESS GATES
zk-0   1/1     Running   0          109s   10.122.104.4     node2   <none>           <none>
zk-1   1/1     Running   0          95s    10.122.166.134   node1   <none>           <none>
zk-2   1/1     Running   0          78s    10.122.135.1     node3   <none>           <none>

2.2.4、查看pvc

[root@master ~]# kubectl get pvc -n bigdata -o wide
NAME           STATUS   VOLUME       CAPACITY   ACCESS MODES   STORAGECLASS   AGE    VOLUMEMODE
datadir-zk-0   Bound    k8s-pv-zk1   2Gi        RWO                           19m    Filesystem
datadir-zk-1   Bound    k8s-pv-zk3   2Gi        RWO                           115s   Filesystem
datadir-zk-2   Bound    k8s-pv-zk2   2Gi        RWO                           98s    Filesystem

2.2.5、查看zk配置

[root@master ~]# kubectl exec -n bigdata zk-0 -- cat /opt/zookeeper/conf/zoo.cfg
#This file was autogenerated DO NOT EDIT
clientPort=2181
dataDir=/var/lib/zookeeper/data
dataLogDir=/var/lib/zookeeper/data/log
tickTime=2000
initLimit=10
syncLimit=5
maxClientCnxns=60
minSessionTimeout=4000
maxSessionTimeout=40000
autopurge.snapRetainCount=3
autopurge.purgeInteval=12
server.1=zk-0.zk-hs.bigdata.svc.cluster.local:2888:3888
server.2=zk-1.zk-hs.bigdata.svc.cluster.local:2888:3888
server.3=zk-2.zk-hs.bigdata.svc.cluster.local:2888:3888

2.2.6、查看集群状态

[root@master ~]# kubectl exec -n bigdata zk-0 zkServer.sh status
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [C
OMMAND] instead.ZooKeeper JMX enabled by default
Using config: /usr/bin/../etc/zookeeper/zoo.cfg
Mode: follower
[root@master ~]# kubectl exec -n bigdata zk-1 zkServer.sh status
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [C
OMMAND] instead.ZooKeeper JMX enabled by default
Using config: /usr/bin/../etc/zookeeper/zoo.cfg
Mode: leader
[root@master ~]# kubectl exec -n bigdata zk-2 zkServer.sh status
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [C
OMMAND] instead.ZooKeeper JMX enabled by default
Using config: /usr/bin/../etc/zookeeper/zoo.cfg
Mode: follower

三、kafka部署

3.1、创建pv

3.1.1、定义pv资源文件

实验环境为虚拟机环境,因此使用的存储卷类型为hostPath。

apiVersion: v1
kind: PersistentVolume
metadata: 
  name: k8s-pv-kafka1
  namespace: bigdata
  labels: 
    type: local
spec: 
  capacity: 
    storage: 2Gi
  accessModes: 
    - ReadWriteOnce
  hostPath: 
    path: /data/pv/kafka1
  persistentVolumeReclaimPolicy: Recycle
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: k8s-pv-kafka2
  namespace: bigdata
  labels:
    type: local
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /data/pv/kafka2
  persistentVolumeReclaimPolicy: Recycle
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: k8s-pv-kafka3
  namespace: bigdata
  labels:
    type: local
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /data/pv/kafka3
  persistentVolumeReclaimPolicy: Recycle

3.1.2、创建pv

kubectl apply -f kafka-pv.yaml 

3.2、kafka集群部署

3.2.1、定义资源清单文件

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: kafka-pdb
  namespace: bigdata
spec: 
  selector: 
    matchLabels: 
      app: kafka
  maxUnavailable: 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata: 
  name: kafka
  namespace: bigdata
spec: 
  selector:
    matchLabels:
      app: kafka
  serviceName: kafka-hs
  replicas: 3
  podManagementPolicy: Parallel
  updateStrategy: 
    type: RollingUpdate
  template: 
    metadata: 
      labels: 
        app: kafka
    spec: 
      affinity: 
        podAntiAffinity: 
          requiredDuringSchedulingIgnoredDuringExecution: 
            - labelSelector: 
                matchExpressions: 
                  - key: "app"
                    operator: In
                    values: 
                    - kafka
              topologyKey: "kubernetes.io/hostname"
        podAffinity: 
          preferredDuringSchedulingIgnoredDuringExecution: 
             - weight: 1
               podAffinityTerm: 
                 labelSelector: 
                    matchExpressions: 
                      - key: "app"
                        operator: In
                        values: 
                        - zk
                 topologyKey: "kubernetes.io/hostname"
      terminationGracePeriodSeconds: 300
      containers: 
      - name: kubernetes-kafka
        imagePullPolicy: IfNotPresent
        image: mirrorgcrio/kubernetes-kafka:1.0-10.2.1
        resources: 
          requests: 
            memory: "500Mi"
            cpu: "0.5"
        ports:
        - containerPort: 9093
          name: server
        command:
        - sh
        - -c
        - "exec kafka-server-start.sh /opt/kafka/config/server.properties --override broker.id=${HOSTNAME##*-} \
          --override listeners=PLAINTEXT://:9093 \
          --override zookeeper.connect=zk-cs.bigdata.svc.cluster.local:2181 \
          --override log.dir=/var/lib/kafka \
          --override auto.create.topics.enable=true \
          --override auto.leader.rebalance.enable=true \
          --override background.threads=10 \
          --override compression.type=producer \
          --override delete.topic.enable=false \
          --override leader.imbalance.check.interval.seconds=300 \
          --override leader.imbalance.per.broker.percentage=10 \
          --override log.flush.interval.messages=9223372036854775807 \
          --override log.flush.offset.checkpoint.interval.ms=60000 \
          --override log.flush.scheduler.interval.ms=9223372036854775807 \
          --override log.retention.bytes=-1 \
          --override log.retention.hours=168 \
          --override log.roll.hours=168 \
          --override log.roll.jitter.hours=0 \
          --override log.segment.bytes=1073741824 \
          --override log.segment.delete.delay.ms=60000 \
          --override message.max.bytes=1000012 \
          --override min.insync.replicas=1 \
          --override num.io.threads=8 \
          --override num.network.threads=3 \
          --override num.recovery.threads.per.data.dir=1 \
          --override num.replica.fetchers=1 \
          --override offset.metadata.max.bytes=4096 \
          --override offsets.commit.required.acks=-1 \
          --override offsets.commit.timeout.ms=5000 \
          --override offsets.load.buffer.size=5242880 \
          --override offsets.retention.check.interval.ms=600000 \
          --override offsets.retention.minutes=1440 \
          --override offsets.topic.compression.codec=0 \
          --override offsets.topic.num.partitions=50 \
          --override offsets.topic.replication.factor=3 \
          --override offsets.topic.segment.bytes=104857600 \
          --override queued.max.requests=500 \
          --override quota.consumer.default=9223372036854775807 \
          --override quota.producer.default=9223372036854775807 \
          --override replica.fetch.min.bytes=1 \
          --override replica.fetch.wait.max.ms=500 \
          --override replica.high.watermark.checkpoint.interval.ms=5000 \
          --override replica.lag.time.max.ms=10000 \
          --override replica.socket.receive.buffer.bytes=65536 \
          --override replica.socket.timeout.ms=30000 \
          --override request.timeout.ms=30000 \
          --override socket.receive.buffer.bytes=102400 \
          --override socket.request.max.bytes=104857600 \
          --override socket.send.buffer.bytes=102400 \
          --override unclean.leader.election.enable=true \
          --override zookeeper.session.timeout.ms=6000 \
          --override zookeeper.set.acl=false \
          --override broker.id.generation.enable=true \
          --override connections.max.idle.ms=600000 \
          --override controlled.shutdown.enable=true \
          --override controlled.shutdown.max.retries=3 \
          --override controlled.shutdown.retry.backoff.ms=5000 \
          --override controller.socket.timeout.ms=30000 \
          --override default.replication.factor=1 \
          --override fetch.purgatory.purge.interval.requests=1000 \
          --override group.max.session.timeout.ms=300000 \
          --override group.min.session.timeout.ms=6000 \
          --override inter.broker.protocol.version=0.10.2-IV0 \
          --override log.cleaner.backoff.ms=15000 \
          --override log.cleaner.dedupe.buffer.size=134217728 \
          --override log.cleaner.delete.retention.ms=86400000 \
          --override log.cleaner.enable=true \
          --override log.cleaner.io.buffer.load.factor=0.9 \
          --override log.cleaner.io.buffer.size=524288 \
          --override log.cleaner.io.max.bytes.per.second=1.7976931348623157E308 \
          --override log.cleaner.min.cleanable.ratio=0.5 \
          --override log.cleaner.min.compaction.lag.ms=0 \
          --override log.cleaner.threads=1 \
          --override log.cleanup.policy=delete \
          --override log.index.interval.bytes=4096 \
          --override log.index.size.max.bytes=10485760 \
          --override log.message.timestamp.difference.max.ms=9223372036854775807 \
          --override log.message.timestamp.type=CreateTime \
          --override log.preallocate=false \
          --override log.retention.check.interval.ms=300000 \
          --override max.connections.per.ip=2147483647 \
          --override num.partitions=1 \
          --override producer.purgatory.purge.interval.requests=1000 \
          --override replica.fetch.backoff.ms=1000 \
          --override replica.fetch.max.bytes=1048576 \
          --override replica.fetch.response.max.bytes=10485760 \
          --override reserved.broker.max.id=1000 "
        env:
        - name: KAFKA_HEAP_OPTS
          value : "-Xmx512M -Xms512M"
        - name: KAFKA_OPTS
          value: "-Dlogging.level=INFO"
        volumeMounts:
        - name: datadir
          mountPath: /var/lib/kafka
        readinessProbe:
          exec:
           command:
            - sh
            - -c
            - "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server=localhost:9093"
  volumeClaimTemplates:
  - metadata:
      name: datadir
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 2Gi
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-hs
  namespace: bigdata
  labels:
    app: kafka
spec:
  ports:
  - port: 9093
    name: server
  selector:
    app: kafka

3.2.2、创建kafka集群

kubectl apply -f kafka.yaml

3.2.3、查看pod

[root@master ~]# kubectl get pod -n bigdata -o wide
NAME      READY   STATUS    RESTARTS   AGE     IP               NODE    NOMINATED NODE   READINESS GATES
kafka-0   1/1     Running   0          13m     10.122.104.20    node2   <none>           <none>
kafka-1   1/1     Running   0          13m     10.122.166.154   node1   <none>           <none>
kafka-2   1/1     Running   0          13m     10.122.135.26    node3   <none>           <none>
zk-0      1/1     Running   1          2d17h   10.122.135.16    node3   <none>           <none>
zk-1      1/1     Running   1          2d17h   10.122.166.146   node1   <none>           <none>
zk-2      1/1     Running   1          2d17h   10.122.104.15    node2   <none>           <none>

四、kafka-manager部署

4.1、定义资源清单文件

apiVersion: v1
kind: Service
metadata:
  name: kafka-manager
  namespace: bigdata
  labels:
    app: kafka-manager
spec:
  type: NodePort
  ports:
  - name: kafka
    port: 9998
    targetPort: 9998
    nodePort: 30900
  selector:
    app: kafka-manager
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-manager
  namespace: bigdata
  labels:
    app: kafka-manager
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-manager
  template:
    metadata:
      labels:
        app: kafka-manager
    spec:
      containers:
      - name: kafka-manager
        image: kafka-manager:1.3.3.18
        imagePullPolicy: IfNotPresent
        ports:
        - name: kafka-manager
          containerPort: 9998
          protocol: TCP
        env:
        - name: KMZK_HOSTS
          value: zk-cs.bigdata.svc.cluster.local:2181
        livenessProbe:
          httpGet:
            path: /api/health
            port: kafka-manager
        readinessProbe:
          httpGet:
            path: /api/health
            port: kafka-manager
        resources:
          limits:
            cpu: 500m
            memory: 512Mi
          requests:
            cpu: 250m
            memory: 256Mi

Note:image字段需根据实际情况修改

4.2、创建kafka-manager

kubectl apply -f kafka-manager.yaml

4.3、访问Node:nodePort访问界面

http://192.168.118.21:30900/

k8s部署zk集群

五、错误汇总

5.1、kafka pod启动失败 状态ERROR

查看状态
kubectl describe -n bigdata pod kafka-1
k8s部署zk集群

查看pod日志
kubectl logs -n bigdata kafka-1
k8s部署zk集群

kafka启动失败的原因是无法连接zk集群
kafka的资源清单中定义的是--override zookeeper.connect=zk-cs.default.svc.cluster.local:2181,实验环境的zk的service定义在了bigdata名称空间中,不在default中,所以修改为--override zookeeper.connect=zk-cs.bigdata.svc.cluster.local:2181。

参考文章:
官方zk实例:https://kubernetes.io/zh/docs/tutorials/stateful-application/zookeeper/
资源清单相关:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#statefulset-v1-apps