【Nginx+Flume实现日志打点分析】

安装nginx（略）

修改nginx.conf然后 nginx -s reload

#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    gzip  on;

    server {
        listen       8080;
        server_name  localhost;

        # 开启 access_log，后面解析这个日志文件，log_format 注释也打开
        access_log  logs/host.access.log  main;

        # 打点地址，单向流程不需要返回什么，empty_gif 只有200多B，很小
        location /dig {
            empty_gif;
            # nginx不允许用POST方式请求静态资源，下面可以支持POST请求
            error_page 405 =200 $request_uri;  
            access_log /opt/data/access.log  log_format;
        }
    }
}

Flume入门

Flume官网地址：http://flume.apache.org/
文档查看地址：http://flume.apache.org/FlumeUserGuide.html
下载地址：http://archive.apache.org/dist/flume/

安装部署

（1）将apache-flume-1.9.0-bin.tar.gz上传到/opt/software目录下
（2）解压apache-flume-1.9.0-bin.tar.gz到/opt/module/目录下
（3）修改apache-flume-1.9.0-bin的名称为flume
（4）将lib文件夹下的guava-11.0.2.jar删除以兼容Hadoop 3.2.x
（5）修改conf目录下的log4j.properties配置文件，配置日志文件路径

tar -zxf /opt/software/apache-flume-1.9.0-bin.tar.gz -C /opt/module/
mv /opt/module/apache-flume-1.9.0-bin /opt/module/flume
rm /opt/module/flume/lib/guava-11.0.2.jar

vim log4j.properties
flume.log.dir=/opt/module/flume/logs

堆内存调整

修改/opt/module/flume/conf/flume-env.sh文件，配置如下参数
export JAVA_OPTS="-Xms4096m -Xmx4096m -Dcom.sun.management.jmxremote"