ELK 分析nginx日志

nginx日志格式
log_format escdn '$http_x_forwarded_for - $remote_user [$time_local] '
'"$request" $status $bytes_sent '
'"$http_referer" "$http_user_agent" "$host" '
'"$remote_addr" "$gzip_ratio" "$upstream_addr"
"$upstream_status" "$request_time"
"$upstream_response_time" "$upstream_cache_status"';

正则表达式
root># cat patterns/nginx
NGINXACCESS %{IPORHOST:remote_addr}" - - [%{HTTPDATE:time_local}] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}

或通配

%{IPORHOST:remote_addr}* - - [%{HTTPDATE:time_local}] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}

gork正则表达式调试（调试OK）
测试日志数据
10.71.64.34-[15/12/2016:10:18:27] "115.183.248.73" - - [15/Dec/2016:10:18:27 +0800] "GET /lottery/kcflash!getDataForCqsscFlash.jhtml HTTP/1.1" 200 541 "http://www.aicai.com/lottery/kcflash!getDataForCqsscFlash.jhtml" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022)" "124.193.230.158" "www.aicai.com" "10.71.65.75:10001" "200" "0.018" "0.018"

正则表达式
IP这一行，可以有""了
%{IPORHOST:remote_addr}" - - [%{HTTPDATE:time_local}] "%{WORD:method} %{URIPATH:path}(?:%{URIPARAM:param})? HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}

E8CB83F7-755A-473F-88B7-CE9F78851276.png

test.com<2016-12-14 21:49:59> /data1/logstash-2.4.1/conf
root># ../bin/logstash -f nginx_log.conf

4EFD973E-6062-4806-A2EF-18DCF83B17A9.png

curl -XPUT http://192.168.91.221:9200/_template/template_nginx_log -d '
{
"order": 0,
"template": "nginx_log-",
"settings": {
"index": {
"refresh_interval": "5s"
}
},
"mappings": {
"default": {
"dynamic_templates": [
{
"string_fields": {
"mapping": {
"index": "analyzed",
"omit_norms": true,
"type": "string",
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string",
"doc_values": true
}
}
},
"match_mapping_type": "string",
"match": ""
}
}
],
"_all": {
"enabled": true
},
"properties": {
"@timestamp": {
"format": "dateOptionalTime",
"index": "not_analyzed",
"type": "date",
"doc_values": true
},
"geoip": {
"dynamic": true,
"type": "object",
"properties": {
"location": {
"type": "geo_point"
}
}
},
"@version": {
"index": "not_analyzed",
"type": "string"
}
}
}
},
"aliases": {}
}'

更加详细的gork匹配nginx_access.log日志
NGINXACCESS %{IPORHOST:remote_addr} - - [%{HTTPDATE:time_local}] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent} "%{IP:remote_addr1}" "%{WORD:domain}" "%{URIHOST:client}" "%{INT:upstream_status}" "%{BASE16FLOAT:request_time}" "%{BASE16FLOAT:upstream_response_time}
注意：gork中调试中%{HOST:domain}中正常，但在logstash中需要换成

日志源文件
10.22.13.4-[26/12/2016:12:28:47] 36.4.145.161 - - [26/Dec/2016:12:28:47 +0800] "GET /qd/order?checkType=1 HTTP/1.1" 200 290 "http://rob.dididapiao.com/qd/index" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0" "rob.dididapiao.com" "58.216.106.14" "http" "10.22.12.6:12110" "200" "0.023" "0.023"

输出结果：

AB6FD4AE-D89C-4638-853D-563A1B3C82B1.png