flume-ng 1.4 安装配置
记录一下flume-ng 的安装配置
三个基本概念
source：数据源
channel：数据传输通道
sink：数据目的地，如存储在hdfs
配置：
flume.conf
#define
agent1.sources = source1
agent1.channels = channel1
agent1.sinks = sink1
#
agent1.sources.source1.type = spooldir
agent1.sources.source1.spoolDir = /tmp/test/
agent1.sources.source1.deletePolicy = immediate
#Describe the sink
agent1.sinks.sink1.type = logger
# Use a channel which buffers events in memory
agent1.channels.channel1.type = memory
agent1.channels.channel1.capacity = 1000
agent1.channels.channel1.transactionCapacity = 100
#Bind the source and sink to the channel
agent1.sources.source1.channels = channel1
agent1.sinks.sink1.channel = channel1
##sink配置
sink配置为logger类型，存储在日志文件中，测试无问题后再改为存储在hdfs中。
##源配置
将源的类型设置为spooldir用于监视目录下的文件
指定spooDir为tmp下的test文件夹
删除策略设置为及时删除，默认是不删除。
启动
flume-ng agent --conf /usr/local/flume/conf/ --conf-file flume.conf --name agent1 -Dflume.root.logger=INFO,console
测试
cd /tmp/test/
echo 'good day' > 123
结果
2014-06-22 10:03:39,540 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.deleteCurrentFile(ReliableSpoolingFileEventReader.java:366)] Preparing to delete file /tmp/test/123
2014-06-22 10:03:40,934 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{file=/tmp/test/123} body: 67 6F 6F 64 20 64 61 79 good day }
