一 配置nginx日志字段收集源
- 在nginx_home/nginx.conf的http模块里面加上以下配置
log_format main '$http_host '
'$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent "$request_body" '
'"$http_referer" "$http_user_agent" "$proxy_add_x_forwarded_for" '
'$request_time '
'$upstream_response_time';
1.1、参数介绍
二 配置logstash数据采集
logstash.nginx.conf
index 的名字 必须是 logstash开头
## nginx log format config
# log_format main '$http_host '
# '$remote_addr - $remote_user [$time_local] '
# '"$request" $status $body_bytes_sent "$request_body" '
# '"$http_referer" "$http_user_agent" "$proxy_add_x_forwarded_for" '
# '$request_time '
# '$upstream_response_time';
filter {
grok {
match => { "message" => "%{NGINXACCESS}" }
}
if [http_user_agent] =~ "inf-ssl-duty-scan" {
drop { }
}
date {
match => [ "time_local" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "http_x_forwarded_for"
}
kv {
source => "request"
field_split => "&?"
value_split => "="
include_keys => [ "network", "country", "language", "deviceId" ]
}
urldecode {
all_fields => true
}
mutate
{
replace => {"host" => "10.26.127.163"}
}
}
output {
elasticsearch {
host => "10.169.97.191"
port => 9200
protocol => "http"
index => "logstash-nginx-prd-%{+YYYY.MM.dd}"
}
# stdout { codec => rubydebug }
}
pattens/nginx
- pattens 的配置请看2.2的配置介绍
NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:remote_addr} - %{USERNAME:remote_user} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{INT:status} %{INT:body_bytes_sent} %{QS:request_body} %{QS:http_referer} %{QS:http_user_agent} "%{IPORHOST:http_x_forwarded_for}" %{NUMBER:request_time:float} %{NUMBER:upstream_response_time:float}
启动脚本
- 脚本可以启动多个logstash实例,分别加载不同的配置文件即可
- agent-nginx.sh
#!/bin/sh
status()
{
info=`ps -elf | grep logstash|grep -v "grep"`
if [ -n "$info" ];then
echo "logstash is running."
else
echo "logstash stopped."
fi
}
stop()
{
pid=`ps -elf | grep logstash | grep -v "grep" | awk '{print $4}'`
kill -9 $pid
}
case $1 in
start)
nohup ./logstash -f ./conf/logstash.nginx.conf -w 3 > nohup.out 2>&1 &
status
;;
stop)
stop
status
;;
status)
status
;;
*)
echo "Usage:{start|stop|status}"
;;
esac
2.2、配置介绍
-
利用 ELK系统分析Nginx日志并对数据进行可视化展示
- 重点:
- 第2点的 patterns的配置方法 下面这个配置对应 ==一== 的 收集
NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:remote_addr} - %{USERNAME:remote_user} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{INT:status} %{INT:body_bytes_sent} %{QS:request_body} %{QS:http_referer} %{QS:http_user_agent} "%{IPORHOST:http_x_forwarded_for}" %{NUMBER:request_time:float} %{NUMBER:upstream_response_time:float}
- 第2点的 patterns的配置方法 下面这个配置对应 ==一== 的 收集
- 重点:
网友评论