elk监控nginx日志

一：logstash配置
监控Nginx的难点在于如何通过Logstash获取想要的值
1、安装nginx并打开access_log
nginx环境 centos 6.5
rpm -ivh http://nginx.org/packages/centos/6/noarch/RPMS/nginx-release-centos-6-0.el6.ngx.noarch.rpm
yum install nginx
/etc/init.d/nginx start
nginx.conf打开如下配置
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

access_log  /var/log/nginx/access.log  main;

2：logstash创建日志格式的正则表达式
路径： /opt/logstash-5.3.0/patterns
cat nginx
NGUSERNAME [a-zA-Z.@-+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} - %{NOTSPACE:remote_user} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NOTSPACE:http_x_forwarded_for}

3: 配置logstash的启动配置文件
cat nginx_access.conf
input{
file{
path => "/var/log/nginx/access.log"
start_position => "beginning"
}
}
filter{
mutate { replace => {"type" => "nginx_access"}}
grok {
match => {"message" => "%{NGINXACCESS}"}}
date {
match => ["timestamp","dd/MMM/YYYY:HH:mm:ss Z"]
}
}
output{
stdout{
codec => rubydebug
}
}
注意，output我这里仅输出验证，正常情况下是输出到redis
../bin/logstash -f nginx_access.conf 启动logstash后的结果打印：
{
"request" => "/",
"agent" => ""Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) C
"verb" => "GET",
"message" => "192.168.85.1 - - [15/May/2017:22:18:38 +0800] "GET / HTTP/1.1" 304 0 "-" "L, like Gecko) Chrome/57.0.2987.133 Safari/537.36" "-"",
"type" => "nginx_access",
"remote_user" => "-",
"path" => "/var/log/nginx/access.log",
"referrer" => ""-"",
"@timestamp" => 2017-05-15T14:18:38.000Z,
"response" => "304",
"bytes" => "0",
"clientip" => "192.168.85.1",
"@version" => "1",
"host" => "0.0.0.0",
"http_x_forwarded_for" => ""-"",
"httpversion" => "1.1",
"timestamp" => "15/May/2017:22:18:38 +0800"
}

elasticsearch 和kibana的配置稍后补充