kafka集成flume

原创

wx61cd54ea3a202 2022-10-16 00:27:30 ©著作权

文章标签 kafka flume 分布式 hadoop bootstrap 文章分类 OpenStack 云计算

©著作权归作者所有：来自51CTO博客作者wx61cd54ea3a202的原创作品，请联系作者获取转载授权，否则将追究法律责任

集成Flume生产者

添加flume job文件，flume入门 vim flume-to-kafka.conf

# 1 组件定义 
a1.sources = r1
a1.sinks = k1
a1.channels = c1
 
# 2 配置 source 
a1.sources.r1.type = TAILDIR
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /opt/module/applog/app.*
a1.sources.r1.positionFile = /opt/module/flume/taildir_position.json
 
# 3 配置 channel 
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
 
# 4 配置 sink 
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.bootstrap.servers = hadoop100:9092,hadoop101:9092,hadoop102:9092
a1.sinks.k1.kafka.topic = chen
a1.sinks.k1.kafka.flumeBatchSize = 20
a1.sinks.k1.kafka.producer.acks = 1
a1.sinks.k1.kafka.producer.linger.ms = 1
 
# 5 拼接组件 
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

启动zk集群
/home/zk.sh start

启动kafka集群
/home/kafka.sh start

启动 kafka 消费者
bin/kafka-console-consumer.sh -bootstrap-server hadoop100:9092 --topic chen --from-beginning

打开新窗口，启动 Flume
bin/flume-ng agent -c conf/ -n a1 -f job/flume-to-kafka.conf &

打开新窗口，在/opt/module/新建applog目录
mkdir applog

向/opt/module/applog/app.log里追加数据
echo hello >> /opt/module/applog/app.log

观察 kafka 消费者，能够看到消费的 hello 数据

集成Flume 消费者

vim kafka-to-flume.conf

# 1 组件定义 
a1.sources = r1 
a1.sinks = k1 
a1.channels = c1 
 
# 2 配置 source 
a1.sources.r1.type = org.apache.flume.source.kafka.KafkaSource 
a1.sources.r1.batchSize = 50 
a1.sources.r1.batchDurationMillis = 200 
a1.sources.r1.kafka.bootstrap.servers = hadoop100:9092 
a1.sources.r1.kafka.topics = first 
a1.sources.r1.kafka.consumer.group.id = custom.g.id 

# 3 配置 channel 
a1.channels.c1.type = memory 
a1.channels.c1.capacity = 1000 
a1.channels.c1.transactionCapacity = 100 
 
# 4 配置 sink 
a1.sinks.k1.type = logger

# 5 拼接组件 
a1.sources.r1.channels = c1 
a1.sinks.k1.channel = c1