07-Flume01

1）array、map、struct

2）meta

3）join

4）compression

Flume

RDBMS ==> Sqoop ==> Hadoop

日志：分散在各个服务器上  ??? ===> Hadoop

Flume is a distributed, reliable, and available service

for efficiently collecting, aggregating, and moving large amounts of log data.

It has a simple and flexible architecture based on streaming data flows.

collecting   采集  source

aggregating    聚合  channel （找个地方把采集过来的数据暂存下）

moving        移动  sink

Flume： 编写配置文件，组合source、channel、sink三者之间的关系

Agent：就是由source、channel、sink组成

编写flume的配置文件其实就是配置agent的构成

Flume就是一个框架：针对日志数据进行采集汇总，把日志从A地方搬运到B地方去

Flume部署

1) 下载

2) 解压到~/app

3) 添加到系统环境变量 ~/.bash_profile

export FLUME_HOME=/home/hadoop/app/apache-flume-1.6.0-cdh5.7.0-bin

export PATH=$FLUME_HOME/bin:$PATH

4) $FLUME_HOME/conf/flume-env.sh

JAVA_HOME

flume-og

flume-ng

./flume-ng agent \

--name a1 \

--conf $FLUME_HOME/conf \

--conf-file /home/hadoop/script/flume/simple-flume.conf \

-Dflume.root.logger=INFO,console \

-Dflume.monitoring.type=http \

-Dflume.monitoring.port=34343

agent_name: 配置的agent的名称a1：就是agent的名称a1、r1、k1、c1# Name the components on this agent.sources =.sinks =.channels =.sources..type = xx.sinks..type = yyy.channels..type = zzz.sources..channels =.sinks..channel =从指定的网络端口上采集日志到控制台输出

a1.sources = r1

a1.sinks = k1

a1.channels = c1

# Describe/configure the source

a1.sources.r1.type = netcat

a1.sources.r1.bind = 0.0.0.0

a1.sources.r1.port = 44444

# Describe the sink

a1.sinks.k1.type = logger

# Use a channel which buffers events in memory

a1.channels.c1.type = memory

# Bind the source and sink to the channel

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

Event: 一条数据

Event: { headers:{} body: 72 75 6F 7A 65 64 61 74 61 0D                  ruozedata. }

Event：headers + body(字节数组)

Flume支持的source、channel、sink有哪些呢？

source

avro

exec  ： tail -F  xx.log

Spooling Directory:

Taildir

netcat

sink

HDFS

logger

avro : 配合avro source使用

kafka

channel

memory

file

Agent：各种组合source、channel、sink之间的关系

把一个文件中新增的内容收集到HDFS上去

exec - memory - hdfs

一个文件夹

spooling - memory - hdfs

文件数据写入kafka

exec - memory - kafka

exce - memory - hdfs ==> Spark/Hive/MR ETL ==> HDFS <== 分析

需求：采集指定文件的内容到HDFS

技术选型：exec - memory - hdfs

./flume-ng agent \

--name a1 \

--conf $FLUME_HOME/conf \

--conf-file /home/hadoop/script/flume/exec-memory-hdfs.conf \

-Dflume.root.logger=INFO,console \

-Dflume.monitoring.type=http \

-Dflume.monitoring.port=34343

需求：采集指定文件夹的内容到控制台

选型：spooling - memory - logger

./flume-ng agent \

--name a1 \

--conf $FLUME_HOME/conf \

--conf-file /home/hadoop/script/flume/spooling-memory-logger.conf \

-Dflume.root.logger=INFO,console \

-Dflume.monitoring.type=http \

-Dflume.monitoring.port=34343

taildir

./flume-ng agent \

--name a1 \

--conf $FLUME_HOME/conf \

--conf-file /home/hadoop/script/flume/taildir-memory-logger.conf \

-Dflume.root.logger=INFO,console \

-Dflume.monitoring.type=http \

-Dflume.monitoring.port=34343