问:磁盘打满扩容后能正常重启吗?
答:不一定
要看文件格式是否损坏(log、index等)。
如果损坏会报错:
index file due to requirement failed: Corrupt index found, index file (/data/kafka_/xx-5/00000000000237215019.index) has non-zero size but the last offset is 237215019 which is no larger than the base offset 237215019.}. deleting and rebuilding index... (kafka.log.Log)
index文件损坏好说:
删除所有index、timeindex,kafka会自动重建
- 删除单个Topic的index、timeindex
- 删除所有topic的index、timeindex
find /data/kafka_log/ -name “.index” |xargs rm -f
find /data/kafka_log/ -name “.timeindex” |xargs rm -f
log文件损坏的话就有点难受了
发现日志里有
ERROR There was an error in one of the threads during logs loading: org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'version': java.nio.BufferUnderflowException (kafka.log.LogManager)
[2020-05-03 10:28:55,092] FATAL [Kafka Server 2], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
需要排查到底哪个topic在索引重建的过程中抛异常了。
找到topic后,需要删除log损坏的文件,可以删最新的log,不影响历史的数据有效性;
另外为了保证日志数据的完整性,线上Topic尽量多副本,这样即使节点整个不可用,删除之后依然有备份;