Kafka Topic分区副本检测修复

检测分区副本缺失，不满足ISR一致性机制的Topic

执行脚本repair-topic.sh

#!/usr/bin/env bash
# Author: Fano.Lee

set -e

TopicCommand="kafka-topics.sh"
PartitionMigrateCommand="kafka-reassign-partitions.sh"
Zookeeper="zk1:2181,zk2:2181,zk3:2181"
BrokerIDList="1001,1002,1003"
UnderReplicaTopic="under-replicated-topic.lst"
WaitRepairTopicJson="wait-repair-topic.json"
RelayoutPartitionJson="relayout-partitions.json"
TempFile="execut.temp"

# 检查缺失分区副的的topic
${TopicCommand} --zookeeper ${Zookeeper} --describe --under-replicated-partitions | awk '{print $2}' | uniq  >${UnderReplicaTopic}

# 生成topic待修复Json文件
/usr/bin/python generate_repair_topics_json.py ${UnderReplicaTopic} ${WaitRepairTopicJson}

# 执行修复
/bin/bash partition-migrate.sh ${WaitRepairTopicJson} ${PartitionMigrateCommand} ${Zookeeper} ${BrokerIDList} ${RelayoutPartitionJson} ${TempFile}

为缺失分区副本的Topic生成预分配文件，生成文件内容为JSON格式

可执行脚本generate_repair_topics_json.py

#!/usr/bin/python
# Author: Fano.Lee

import sys
import json

if __name__ == '__main__':

    '''Generate migrate json for appoint topics'''
    if len(sys.argv) == 3:
        under_replica_topic_file = sys.argv[1]
        wait_repair_topic_json_file = sys.argv[2]
        format_wait_repair_topics_json = dict()
        topics = []

        f1 = open(under_replica_topic_file, 'r')
        while True:
            topic = f1.readline()
            if not topic:
                break
            topics.append({'topic': topic.strip("\n")})
        f1.close()

        format_wait_repair_topics_json.update({"topics": topics, "version": 1})

        with open(wait_repair_topic_json_file, 'w+') as f2:
            f2.write(json.dumps(format_wait_repair_topics_json))
    else:
        print "Usage: generate_move_topics_json.py UnderReplicaTopic WaitRepairTopicJson"
        sys.exit(5)

以下程序执行副分重分配，借助重分配修复副本，以恢复ISR机制

可执行脚本partition-migrate.sh

#!/bin/bash
# Author: Fano.Lee
# Can use when occur following:
# 1. When migrate kafka cluster broker
# 2. When offline kafka broker
# 3. When lost partition replicas of kafka topics

set -e

if [[ $# != 6 ]]; then
    echo "Usage: partition-migrate.sh WaitRepairTopicJson PartitionMigrateCommand Zookeeper BrokerIDList KeyWord RelayoutPartitionJson"
    echo -e "\tWaitRepairTopicJson PartitionMigrateCommand Zookeeper BrokerIDList KeyWord RelayoutPartitionJson as variable"
    exit 2
fi

KeyWord="Proposed partition reassignment configuration"

cat ${1}
echo -e "\n"

while read -p "Verification output. Whether continue: yes/no: " instruction
do
    if [[ ${instruction} == "no" ]];then
        echo "Has been exit."
        exit 0
    elif [[ ${instruction} == "yes" ]];then
        echo "Continue execute ... Enter"
        break
    else
        continue
    fi
done

${2} --zookeeper ${3} --topics-to-move-json-file ${1} --broker-list ${4} --generate | /usr/bin/sed -ne '/'"${KeyWord}"'/,$p'|grep -v "${KeyWord}" >${5}

cat ${5}
echo -e "\n"

while read -p "Verification output. Whether continue: yes/no: " instruction
do
    if [[ ${instruction} == "no" ]];then
        echo "Has been exit."
        exit 0
    elif [[ ${instruction} == "yes" ]];then
        echo "Continue execute ... Enter"
        break
    else
        continue
    fi
done

START_TIME=$(date +'%Y-%m-%d %H:%M:%S')

${2} --zookeeper ${3} --reassignment-json-file ${5} --execute >${6}

cat ${6}
echo -e "\n"

while read -p "Verification output. Whether continue: yes/no: " instruction
do
    if [[ ${instruction} == "no" ]];then
        echo "Has been exit."
        exit 0
    elif [[ ${instruction} == "yes" ]];then
        echo "Continue execute ... Enter"
        break
    else
        continue
    fi
done

while true
do
    result=$(${2} --zookeeper ${3} --reassignment-json-file ${5} --verify|grep still|wc -l)
    if [[ ${result} == 0 ]];then
        break
    else
        echo ${result}
        sleep 3
        continue
    fi
done

END_TIME=$(date +'%Y-%m-%d %H:%M:%S')

start_seconds=$(date --date="${START_TIME}" +%s)
end_seconds=$(date --date="${END_TIME}" +%s)

echo "Running Time: $((end_seconds-start_seconds))s"
echo "Migrate complete."

exit 0