Flume的安装与综合使用
https://www.jianshu.com/p/90e17b80f366
-
实时日志采集框架图
1.在$FLUME_HOME/conf
下新增一个配置文件avro-memory-kafka.conf
注意:Flume版本:flume-ng-1.6.0-cdh5.7.0
,其他版本的配置会有所不同。
具体配置参照:http://archive.cloudera.com/cdh5/cdh/5/
avro-memory-kafka.sources = avro-source
avro-memory-kafka.sinks = kafka-sink
avro-memory-kafka.channels = memory-channel
avro-memory-kafka.sources.avro-source.type = avro
avro-memory-kafka.sources.avro-source.bind = 192.168.21.110
avro-memory-kafka.sources.avro-source.port = 44444
avro-memory-kafka.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink
avro-memory-kafka.sinks.kafka-sink.brokerList = 192.168.21.110:9092
avro-memory-kafka.sinks.kafka-sink.topic = hello_topic
avro-memory-kafka.sinks.kafka-sink.batchSize = 5
avro-memory-kafka.sinks.kafka-sink.requiredAcks =1
avro-memory-kafka.channels.memory-channel.type = memory
avro-memory-kafka.sources.avro-source.channels = memory-channel
avro-memory-kafka.sinks.kafka-sink.channel = memory-channel
2.重点是配置kafka sink
的内容,参考flume-ng-1.6.0-cdh5.7.0
官方的文档:
3.先启动avro-memory-kafka
监听
flume-ng agent \
--name avro-memory-kafka \
--conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/avro-memory-kafka.conf \
-Dflume.root.logger=INFO,console
4.然后启动exec-memory-avro.conf
flume-ng agent \
--name exec-memory-avro \
--conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/exec-memory-avro.conf \
-Dflume.root.logger=INFO,console
5.启动kafka-console-consumer
监听kafka的消息变化
kafka-console-consumer.sh --zookeeper 192.168.21.110:2181 --topic hello_topic
网友评论