flume1.70 安装部署

环境

[root@node06 conf]# rpm -q centos-release
centos-release-7-6.1810.2.el7.centos.x86_64

[root@node06  conf]# java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

安装部署

1. 将apache-flume-1.7.0-bin.tar.gz上传到linux的/opt/jars目录下
2. 解压apache-flume-1.7.0-bin.tar.gz到/opt/module/目录下

[root@node06 jars]#  tar -zxvf apache-flume-1.7.0-bin.tar.gz -C /opt/module/

3. 修改apache-flume-1.7.0-bin的名称为flume-1.7.0

[root@node06 module]# mv apache-flume-1.7.0-bin flume-1.7.0

4. 将flume/conf下的flume-env.sh.template文件修改为flume-env.sh，并配置flume-env.sh文件

[root@node06 conf]# cd conf
[root@node06 conf]# mv flume-env.sh.template flume-env.sh
[root@node06 conf]# vi flume-env.sh
export JAVA_HOME=/opt/module/jdk1.8.0_144

Flume监控之Ganglia

1. 安装httpd服务与php

[root@node06 flume]# yum -y install httpd php

2. 安装其他依赖

[root@node06 flume]# yum -y install rrdtool perl-rrdtool rrdtool-devel
[root@node06 flume]# yum -y install apr-devel
[root@node06 conf]# yum install libconfuse
[root@node06  conf]# yum install libmemcached
## ganglia-web 需要依赖
[root@node06  conf]# yum install -y php-ZendFramework
[root@node06  conf]# yum install -y php-gd

3. 安装ganglia

[root@node06  conf]# sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/g/ganglia-gmetad-3.7.2-2.el7.x86_64.rpm
[root@node06  conf]# sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/g/ganglia-gmond-3.7.2-2.el7.x86_64.rpm
[root@node06  conf]# sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/g/ganglia-web-3.7.1-2.el7.x86_64.rpm

[root@node06  flume]# sudo yum -y install ganglia-gmetad 
[root@node06 flume]# sudo yum -y install ganglia-web
[root@node06 flume]# sudo yum install -y ganglia-gmond

4. 修改配置文件/etc/httpd/conf.d/ganglia.conf

[root@node06 flume]# sudo vim /etc/httpd/conf.d/ganglia.conf

###修改为
#
# Ganglia monitoring system php web frontend
#
Alias /ganglia /usr/share/ganglia
<Location /ganglia>
  Require all granted
  # Require ip 10.1.2.3
  # Require host example.org
</Location>

5. 修改配置文件/etc/ganglia/gmetad.conf

[root@node06 flume]# sudo vim /etc/ganglia/gmetad.conf

###修改为
data_source "node06" 192.168.2.165

6. 修改配置文件/etc/ganglia/gmond.conf

[root@node06 flume]# sudo vim /etc/ganglia/gmond.conf 

###修改为
cluster {
  name = "node06"
  owner = "unspecified"
  latlong = "unspecified"
  url = "unspecified"
}
udp_send_channel {
  #bind_hostname = yes # Highly recommended, soon to be default.
                       # This option tells gmond to use a source address
                       # that resolves to the machine's hostname.  Without
                       # this, the metrics may appear to come from any
                       # interface and the DNS names associated with
                       # those IPs will be used to create the RRDs.
 host = 192.168.2.165
  port = 8649
  ttl = 1
}
udp_recv_channel {
  port = 8649
  bind = 192.168.2.165
  retry_bind = true
  # Size of the UDP buffer. If you are handling lots of metrics you really
  # should bump it up to e.g. 10MB or even higher.
  # buffer = 10485760
}

7. 修改配置文件/etc/selinux/config

[root@node06 flume]# sudo vim /etc/selinux/config

###修改为

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted


提示：selinux本次生效关闭必须重启，如果此时不想重启，可以临时生效之：
[root@node06 flume]# sudo setenforce 0

5)启动ganglia

[root@node06 conf]# systemctl start httpd.service
[root@node06 conf]# systemctl start gmetad
[root@node06 conf]# systemctl start gmond

6. 打开网页浏览ganglia页面
   http://node06/ganglia

操作Flume测试监控

1. 修改/opt/module/flume/conf目录下的flume-env.sh配置：

JAVA_OPTS="-Dflume.monitoring.type=ganglia
-Dflume.monitoring.hosts=192.168.2.165:8649
-Xms100m
-Xmx200m"

2. 启动Flume任务

[root@node06 flume]# bin/flume-ng agent \
--conf conf/ \
--name a1 \
--conf-file job/flume-telnet-logger.conf \
-Dflume.root.logger==INFO,console \
-Dflume.monitoring.type=ganglia \
-Dflume.monitoring.hosts=192.168.2.165:8649

3. 发送数据观察ganglia监测图

[root@node06 flume]# telnet localhost 44444

图例说明：

字段（图表名称）	字段含义
EventPutAttemptCount	source尝试写入channel的事件总数量
EventPutSuccessCount	成功写入channel且提交的事件总数量
EventTakeAttemptCount	sink尝试从channel拉取事件的总数量。这不意味着每次事件都被返回，因为sink拉取的时候channel可能没有任何数据。
EventTakeSuccessCount	sink成功读取的事件的总数量
StartTime	channel启动的时间（毫秒）
StopTime	channel停止的时间（毫秒）
ChannelSize	目前channel中事件的总数量
ChannelFillPercentage	channel占用百分比
ChannelCapacity	channel的容量