Flume学习笔记

官方文档

核心组件

1. Source 收集

2. Channel 聚集

3. Sink 输出

Flume 安装前置条件

• Java Runtime Environment - Java 1.8 or later
• Memory - Sufficient memory for configurations used by sources, channels or sinks
• Disk Space - Sufficient disk space for configurations used by channels or sinks
• Directory Permissions - Read/Write permissions for directories used by agent

安装

1. 安装jdk
2. 下载并解压到用户目录
3. 配置环境变量

   export FLUME_HOME="/Users/gaowenfeng/Documents/bigdata/flume"
   export PATH=$FLUME_HOME/bin:$PATH
   

4. source 下让其生效
5. flume-env.sh 的配置

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home

6. 检测 $FLUME_HOME/bin/flume-ng version

需求1

使用flume的关键就是写配置文件

1. 配置Source
2. 配置Channel
3. 配置Sink
4. 把以上三个组件穿起来

# example.conf: A single-node Flume configuration

# a1 agent 的名称
# r1 surce 的名称
# k1 sink 的名称
# c1 channel 的名称

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory


# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

启动Agent

flume-ng agent \
--name a1 \
--conf conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/example.conf \
-Dflume.root.logger=INFO,console

使用telnet进行测试

telnet host ip

Event: { headers:{} body: 68 65 6C 6C 6F 0D                               hello. }
Event 是Flume 数据传输的基本单元
Event = 可选的header+byte array

需求2

Agent 选型：exec source +memory channel+logger sink

exex-memory-logger.conf

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /Users/gaowenfeng/data/data.log
a1.sources.r1.shell = /bin/sh -c

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory


# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

启动agent

flume-ng agent \
--name a1 \
--conf conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/exex-memory-logger.conf \
-Dflume.root.logger=INFO,console

需求3

技术选型:

    exec source + memory channel + avro sink
    avro source + memory channel + logger sink

exec-memory-avro.conf

# Name the components on this agent
exec-memory-avro.sources = exec-source
exec-memory-avro.sinks = avro-sink
exec-memory-avro.channels = memory-channel

# Describe/configure the source
exec-memory-avro.sources.exec-source.type = exec
exec-memory-avro.sources.exec-source.command = tail -F /Users/gaowenfeng/data/data.log
exec-memory-avro.sources.exec-source.shell = /bin/sh -c

# Describe the sink
exec-memory-avro.sinks.avro-sink.type = avro
exec-memory-avro.sinks.avro-sink.hostname = localhost
exec-memory-avro.sinks.avro-sink.port = 44444

# Use a channel which buffers events in memory
exec-memory-avro.channels.memory-channel.type = memory


# Bind the source and sink to the channel
exec-memory-avro.sources.exec-source.channels = memory-channel
exec-memory-avro.sinks.avro-sink.channel = memory-channel

avro-memory-logger.conf

# Name the components on this agent
avro-memory-logger.sources = avro-source
avro-memory-logger.sinks = logger-sink
avro-memory-logger.channels = memory-channel

# Describe/configure the source
avro-memory-logger.sources.avro-source.type = avro
avro-memory-logger.sources.avro-source.bind = localhost
avro-memory-logger.sources.avro-source.port = 44444

# Describe the sink
avro-memory-logger.sinks.logger-sink.type = logger

# Use a channel which buffers events in memory
avro-memory-logger.channels.memory-channel.type = memory


# Bind the source and sink to the channel
avro-memory-logger.sources.avro-source.channels = memory-channel
avro-memory-logger.sinks.logger-sink.channel = memory-channel

先启动agent

flume-ng agent \
--name avro-memory-logger \
--conf conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/avro-memory-logger.conf \
-Dflume.root.logger=INFO,console

再启动

flume-ng agent \
--name exec-memory-avro \
--conf conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/exec-memory-avro.conf \
-Dflume.root.logger=INFO,console

日志收集过程

1. 机器A上哪个监控一个文件，当我们访问主站的时候会有用户行为日志记录到access.log中
2. avro sin把新产生的日志输出到对应的avro source指定的hostname port 中
3. 通过avro source对应的agent将日志输出到对应的控制台[kafaka]