美文网首页linux
elk 搭建nginx 日志监控

elk 搭建nginx 日志监控

作者: 085a5be2950c | 来源:发表于2017-03-27 17:42 被阅读160次

    一 配置nginx日志字段收集源

    • 在nginx_home/nginx.conf的http模块里面加上以下配置
        log_format  main  '$http_host '
                          '$remote_addr - $remote_user [$time_local] '
                          '"$request" $status $body_bytes_sent "$request_body" '
                          '"$http_referer" "$http_user_agent" "$proxy_add_x_forwarded_for" '
                          '$request_time '
                          '$upstream_response_time';
    
    

    1.1、参数介绍

    二 配置logstash数据采集

    logstash.nginx.conf

    index 的名字 必须是 logstash开头

    ## nginx log format config
    #    log_format  main  '$http_host '
    #                      '$remote_addr - $remote_user [$time_local] '
    #                      '"$request" $status $body_bytes_sent "$request_body" '
    #                      '"$http_referer" "$http_user_agent" "$proxy_add_x_forwarded_for" '
    #                      '$request_time '
    #                      '$upstream_response_time';
    
    filter {
        grok {
            match => { "message" => "%{NGINXACCESS}" }
        }
    
        if [http_user_agent] =~ "inf-ssl-duty-scan" {
            drop { }
        }
    
        date {
            match => [ "time_local" , "dd/MMM/yyyy:HH:mm:ss Z" ]
        }
    
        geoip {
            source => "http_x_forwarded_for"
        }
        kv {
            source => "request"
            field_split => "&?"
            value_split => "="
            include_keys => [ "network", "country", "language", "deviceId" ]
        }
    
        urldecode {
            all_fields => true
        }
    
         mutate
         {
             replace => {"host" => "10.26.127.163"}
         }
    
    
    }
    
    output {
        elasticsearch {
    
            host => "10.169.97.191"
            port => 9200
            protocol => "http"
            index => "logstash-nginx-prd-%{+YYYY.MM.dd}"
      }
    #  stdout { codec => rubydebug }
    }
    
    
    pattens/nginx
    • pattens 的配置请看2.2的配置介绍
    NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:remote_addr} - %{USERNAME:remote_user} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{INT:status} %{INT:body_bytes_sent} %{QS:request_body} %{QS:http_referer} %{QS:http_user_agent} "%{IPORHOST:http_x_forwarded_for}" %{NUMBER:request_time:float} %{NUMBER:upstream_response_time:float}
    
    
    启动脚本
    • 脚本可以启动多个logstash实例,分别加载不同的配置文件即可
    • agent-nginx.sh
    #!/bin/sh
    status()
    {
            info=`ps -elf | grep logstash|grep -v "grep"`
            if [ -n "$info" ];then
                    echo "logstash is running."
            else
                    echo "logstash stopped."
            fi
    }
    
    stop()
    {
            pid=`ps -elf | grep logstash | grep -v "grep" | awk '{print $4}'`
            kill -9 $pid
    }
    case $1 in
            start)
            nohup ./logstash -f ./conf/logstash.nginx.conf -w 3 > nohup.out 2>&1 &
            status
            ;;
            stop)
            stop
            status
            ;;
            status)
            status
            ;;
            *)
            echo "Usage:{start|stop|status}"
            ;;
    esac
    
    

    2.2、配置介绍

    • 利用 ELK系统分析Nginx日志并对数据进行可视化展示
      • 重点:
        • 第2点的 patterns的配置方法 下面这个配置对应 ==一== 的 收集
          NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:remote_addr} - %{USERNAME:remote_user} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{INT:status} %{INT:body_bytes_sent} %{QS:request_body} %{QS:http_referer} %{QS:http_user_agent} "%{IPORHOST:http_x_forwarded_for}" %{NUMBER:request_time:float} %{NUMBER:upstream_response_time:float}
          
          

    相关文章

      网友评论

        本文标题:elk 搭建nginx 日志监控

        本文链接:https://www.haomeiwen.com/subject/mboubttx.html