flume 收集日志到 kafka 整合_apache

就是服务器B的Sink要换成kafka 的sink即可

服务器A还是不变:

# Define a memory channel called ch1 on agent1
agent1.channels.ch1.type = memory
agent1.channels.ch1.capacity = 1000
agent1.channels.ch1.transactionCapacity = 100


agent1.sources.exec-source1.channels = ch1
agent1.sources.exec-source1.type = exec
agent1.sources.exec-source1.command = tail -F /Users/walle/Documents/D2/testflume.log


# Define a logger sink that simply logs all events it receives
# and connect it to the other end of the same channel.
agent1.sinks.avro-sink1.channel = ch1
agent1.sinks.avro-sink1.type = avro
agent1.sinks.avro-sink1.hostname = localhost
agent1.sinks.avro-sink1.port = 44445

# Finally, now that we've defined all of our components, tell
# agent1 which ones we want to activate.
agent1.channels = ch1
agent1.sources = exec-source1
agent1.sinks = avro-sink1

 

服务器B

 

# Define a memory channel called ch1 on agent1
agent2.channels.ch2.type = memory
agent2.channels.ch2.capacity = 1000
agent2.channels.ch2.transactionCapacity = 100

# Define an Avro source called avro-source1 on agent1 and tell it
# to bind to 0.0.0.0:41414. Connect it to channel ch1.
agent2.sources.avro-source1.channels = ch2
agent2.sources.avro-source1.type = avro
agent2.sources.avro-source1.bind = localhost
agent2.sources.avro-source1.port = 44445


# Define a logger sink that simply logs all events it receives
# and connect it to the other end of the same channel.
agent2.sinks.kafka-sink1.channel = ch2
agent2.sinks.kafka-sink1.type = org.apache.flume.sink.kafka.KafkaSink
agent2.sinks.kafka-sink1.kafka.bootstrap.servers = localhost:9092
agent2.sinks.kafka-sink1.topic = test
agent2.sinks.kafka-sink1.flumeBatchSize = 2
agent2.sinks.kafka-sink1.kafka.producer.acks = 1


# Finally, now that we've defined all of our components, tell
# agent1 which ones we want to activate.
agent2.channels = ch2
agent2.sources = avro-source1
agent2.sinks = kafka-sink1

参考文章:

​http://www.codeblogbt.com/archives/1746​