使用Logstash简单搜集Nginx Access日志并储存到

1. 首先安装Java环境：apt-get install openjdk-8-jdk
2. 在官网下载Logstash：https://www.elastic.co/cn/downloads/logstash
3. 在conf目录新建一个配置文件：

input {
    #这里可以同时监控多个文件
    file {
        path => ["/usr/local/nginx/logs/error.log"]
        start_position => "beginning"
        type => "error"
    }
    file {
        path => ["/usr/local/nginx/logs/www.xxx.com.access.log"]
        start_position => "beginning"
        type => "access"
    }
}
filter {
    #每种文件需要配置自己的grok插件语法来搜集需要的数据
    if [type] == "access"{
        grok {
            match => {
                #这里的须发需要自定义配置
                "message" => "^%{IPV4:remote_addr} \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{INT:status} %{INT:body_bytes_sent} \"%{NOTSPACE:http_referer}\" %{NUMBER:request_time} \"%{IPV4:upstream_addr}:%{POSINT:upstream_port}\" %{NUMBER:upstream_response_time} \"%{DATA:http_user_agent}\" \"%{NOTSPACE:http_x_forwarded_for}\""
            }
        }
        #配置GeoIP的数据库解析ip
        geoip {
            source => "remote_addr"
        }   
    }
}
output {
    # 不满足筛选条件的就不写入
    if "_grokparsefailure" not in [tags] {
        #数据输出到elasticsearch
        elasticsearch {
            hosts => ["127.0.0.1:9200"]
            index => "logstash-nginx-%{type}-%{+YYYY-MM}"
        }
    }
    #调试
    stdout{codec => rubydebug}
}

NginxAccess日志格式：

log_format  main  '$remote_addr [$time_local] "$request" $status $body_bytes_sent "$http_referer" $request_time "$upstream_addr" $upstream_response_time "$http_user_agent" "$http_x_forwarded_for"';

日志示例：

69.126.145.85 [25/Jun/2018:07:31:27 +0000] "POST /api/userInfoRongCloud HTTP/1.1" 200 197 "-" 0.191 "18.191.5.101:9000" 0.191 "dating/1.0.5 (iPhone; iOS 12.0; Scale/3.00)" "-"

这一步的grok语法需要自己调试，在线调试地址：

https://grokdebug.herokuapp.com/

这里可以使用很多已经写好的模式变量，参见：

https://github.com/elastic/logstash/blob/1.4/patterns/grok-patterns

4. 使用-f参数指定配置文件测试解析结果是否正确
5. 添加到守护进程中执行(正式环境需要把配置文件里面的调试输出去掉，不然会报错)：

[program:logstash-worker]
process_name=%(program_name)s_%(process_num)02d
command=/usr/local/logstash-6.3.0/bin/logstash -f /usr/local/logstash-6.3.0/config/nginx-access.conf
autostart=true
autorestart=true
user=root
numprocs=1
redirect_stderr=true
stdout_logfile=/var/log/logstash-worker.log

supervisorctl reread
supervisorctl update