几个配置实例

source、channel、sink类型

# 关于source
Avro Source              
Thrift Source             
Exec Source            
JMS Source            
Spooling Directory Source  
Twitter 1% firehose Source   
Netcat Source            
Sequence Generator Source  
Syslog Sources            
HTTP Source          
Legacy Sources      

# 关于channel
Memory Channel        
JDBC Channel            
File Channel             
Spillable Memory Channel  
Pseudo Transaction Channel
Custom Channel    

# 关于sink
HDFS Sink                 
Logger Sink             
Avro Sink                
Thrift Sink                 
IRC Sink                  
File Roll Sink           
Null Sink                  
HBase Sink              
Morphline Solr Sink      
ElasticSearch Sink         
Kite Dataset Sink          
Custom Sink         

Netcat Source - Memory Channel - Logger Sink

vim ${FLUME_HOME}/conf/nml.properties
nml.sources = r1
nml.sinks = k1
nml.channels = c1
nml.sources.r1.type = netcat
nml.sources.r1.bind = 0.0.0.0
nml.sources.r1.port = 44444
nml.sinks.k1.type = logger
nml.channels.c1.type = memory
nml.channels.c1.capacity = 1000
nml.channels.c1.transactionCapacity = 100
nml.sources.r1.channels = c1
nml.sinks.k1.channel = c1

flume-ng agent \
--name nml \
--conf ${FLUME_HOME}/conf \
--conf-file nml.properties \
-Dflume.root.logger=INFO,console

telnet 192.168.219.51 44444

Agent Sink - Agent Source

# ssh root@192.168.219.51
vim ${FLUME_HOME}/conf/a2a_node1.properties
a2a.sources = r1
a2a.sinks = k1
a2a.channels = c1
a2a.sources.r1.type = netcat
a2a.sources.r1.bind = 0.0.0.0
a2a.sources.r1.port = 44444
a2a.sinks.k1.type = avro
a2a.sinks.k1.hostname = 192.168.219.52
a2a.sinks.k1.port = 60000
a2a.channels.c1.type = memory
a2a.channels.c1.capacity = 1000
a2a.channels.c1.transactionCapacity = 100
a2a.sources.r1.channels = c1
a2a.sinks.k1.channel = c1

# ssh root@192.168.219.52
vim ${FLUME_HOME}/conf/a2a_node2.properties
a2a.sources = r1
a2a.sinks = k1
a2a.channels = c1
a2a.sources.r1.type = avro
a2a.sources.r1.bind = 0.0.0.0
a2a.sources.r1.port = 60000
a2a.sinks.k1.type = logger
a2a.channels.c1.type = memory
a2a.channels.c1.capacity = 1000
a2a.channels.c1.transactionCapacity = 100
a2a.sources.r1.channels = c1
a2a.sinks.k1.channel = c1

# ssh root@192.168.219.52
flume-ng agent \
--name a2a \
--conf ${FLUME_HOME}/conf \
--conf-file a2a_node2.properties \
-Dflume.root.logger=INFO,console

# ssh root@192.168.219.51
flume-ng agent \
--name a2a \
--conf ${FLUME_HOME}/conf \
--conf-file a2a_node1.properties \
-Dflume.root.logger=INFO,console

Exec Source - Memory Channel - Logger Sink

vim ${FLUME_HOME}/conf/eml.properties
eml.sources = r1
eml.sinks = k1
eml.channels = c1
eml.sources.r1.type = exec
eml.sources.r1.command = tail -F /log/flume/flume.exec.log
eml.sinks.k1.type = logger
eml.channels.c1.type = memory
eml.channels.c1.capacity = 1000
eml.channels.c1.transactionCapacity = 100
eml.sources.r1.channels = c1
eml.sinks.k1.channel = c1

flume-ng agent \
--name eml \
--conf ${FLUME_HOME}/conf \
--conf-file eml.properties \
-Dflume.root.logger=INFO,console

## Spooling Directory Source
vim ${FLUME_HOME}/conf/sml.properties
sml.sources = r1
sml.sinks = k1
sml.channels = c1
sml.sources.r1.type = spooldir
sml.sources.r1.spoolDir = /log
sml.sources.r1.fileHeader = true
sml.sinks.k1.type = logger
sml.channels.c1.type = memory
sml.channels.c1.capacity = 1000
sml.channels.c1.transactionCapacity = 100
sml.sources.r1.channels = c1
sml.sinks.k1.channel = c1

flume-ng agent \
--name sml \
--conf ${FLUME_HOME}/conf \
--conf-file sml.properties \
-Dflume.root.logger=INFO,console

其他命令

用户指导文档：FlumeUserGuide

avro-client向agent发送数据

flume-ng avro-client
-H 192.168.219.52
-p 60000
-F /usr/logs/log.10