3．执行监控配置
[atguigu@hadoop102 flume] sbin/start-dfs.sh
[atguigu@hadoop103 hadoop-2.7.2]$ sbin/start-yarn.sh
[atguigu@hadoop102 hive] 3.3 实时读取目录文件到HDFS案例 1）案例需求：使用Flume监听整个目录的文件 2）需求分析：  3）实现步骤： 1．创建配置文件flume-dir-hdfs.conf 创建一个文件 [atguigu@hadoop102 job]](https://math.jianshu.com/math?formula=bin%2Fhive%20hive%20(default)%3E%205%EF%BC%8E%E5%9C%A8HDFS%E4%B8%8A%E6%9F%A5%E7%9C%8B%E6%96%87%E4%BB%B6%E3%80%82%20!%5Bimage.png%5D(https%3A%2F%2Fupload-images.jianshu.io%2Fupload_images%2F10390889-4512ed3cafaf823e.png%3FimageMogr2%2Fauto-orient%2Fstrip%257CimageView2%2F2%2Fw%2F1240)%203.3%20%E5%AE%9E%E6%97%B6%E8%AF%BB%E5%8F%96%E7%9B%AE%E5%BD%95%E6%96%87%E4%BB%B6%E5%88%B0HDFS%E6%A1%88%E4%BE%8B%201%EF%BC%89%E6%A1%88%E4%BE%8B%E9%9C%80%E6%B1%82%EF%BC%9A%E4%BD%BF%E7%94%A8Flume%E7%9B%91%E5%90%AC%E6%95%B4%E4%B8%AA%E7%9B%AE%E5%BD%95%E7%9A%84%E6%96%87%E4%BB%B6%202%EF%BC%89%E9%9C%80%E6%B1%82%E5%88%86%E6%9E%90%EF%BC%9A%20!%5Bimage.png%5D(https%3A%2F%2Fupload-images.jianshu.io%2Fupload_images%2F10390889-d7e0ab54b655b2e0.png%3FimageMogr2%2Fauto-orient%2Fstrip%257CimageView2%2F2%2Fw%2F1240)%203%EF%BC%89%E5%AE%9E%E7%8E%B0%E6%AD%A5%E9%AA%A4%EF%BC%9A%201%EF%BC%8E%E5%88%9B%E5%BB%BA%E9%85%8D%E7%BD%AE%E6%96%87%E4%BB%B6flume-dir-hdfs.conf%20%E5%88%9B%E5%BB%BA%E4%B8%80%E4%B8%AA%E6%96%87%E4%BB%B6%20%5Batguigu%40hadoop102%20job%5D)
touch flume-dir-hdfs.conf
打开文件
[atguigu@hadoop102 job]$ vim flume-dir-hdfs.conf
添加如下内容
a3.sources = r3
a3.sinks = k3
a3.channels = c3
Describe/configure the source
a3.sources.r3.type = spooldir
a3.sources.r3.spoolDir = /opt/module/flume/upload
a3.sources.r3.fileSuffix = .COMPLETED
a3.sources.r3.fileHeader = true
忽略所有以.tmp结尾的文件，不上传
a3.sources.r3.ignorePattern = ([^ ]*.tmp)
Describe the sink
a3.sinks.k3.type = hdfs
a3.sinks.k3.hdfs.path = hdfs://hadoop102:9000/flume/upload/%Y%m%d/%H
上传文件的前缀
a3.sinks.k3.hdfs.filePrefix = upload-
是否按照时间滚动文件夹
a3.sinks.k3.hdfs.round = true
多少时间单位创建一个新的文件夹
a3.sinks.k3.hdfs.roundValue = 1
重新定义时间单位
a3.sinks.k3.hdfs.roundUnit = hour
是否使用本地时间戳
a3.sinks.k3.hdfs.useLocalTimeStamp = true
积攒多少个Event才flush到HDFS一次
a3.sinks.k3.hdfs.batchSize = 100
设置文件类型，可支持压缩
a3.sinks.k3.hdfs.fileType = DataStream
多久生成一个新的文件
a3.sinks.k3.hdfs.rollInterval = 600
设置每个文件的滚动大小大概是128M
a3.sinks.k3.hdfs.rollSize = 134217700
文件的滚动与Event数量无关
a3.sinks.k3.hdfs.rollCount = 0
最小冗余数
a3.sinks.k3.hdfs.minBlockReplicas = 1
Use a channel which buffers events in memory
a3.channels.c3.type = memory
a3.channels.c3.capacity = 1000
a3.channels.c3.transactionCapacity = 100
Bind the source and sink to the channel
a3.sources.r3.channels = c3
a3.sinks.k3.channel = c3
image.png
本教程由尚硅谷教育大数据研究院出品，如需转载请注明来源，欢迎大家关注尚硅谷公众号（atguigu）了解更多。
网友评论