filebeat模块收集nginx日志

1.修改filebeat配置文件

[root@db01 ~]# cat /etc/filebeat/filebeat.yml
##启动module
filebeat.config.modules:
##设置module配置文件路径
  path: ${path.config}/modules.d/*.yml
##设置启用
  reload.period: 10s

setup.kibana:
  host: "172.16.210.53:5601"

output.elasticsearch:
  hosts: ["172.16.210.53:9200"]
  indices:
  - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
    when.contains:
      fileset.name: "access"

  - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
    when.contains:
      fileset.name: "error"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
[root@db01 ~]# systemctl restart filebeat ##重启filebeat
[root@db01 ~]# filebeat modules list ##
Enabled:

Disabled:
apache2
auditd
elasticsearch
--
---

2.启用nginx模块

[root@db01 ~]# filebeat modules enable nginx
Enabled nginx

3.修改nginx配置文件为普通格式

[root@db01 ~]# egrep -v "^$|#"  /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
    worker_connections 1024;
}
http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
 access_log  /var/log/nginx/access.log  main;
    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;
    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;
    include /etc/nginx/conf.d/*.conf;
    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        root         /usr/share/nginx/html;
        include /etc/nginx/default.d/*.conf;
        location / {
        }
        error_page 404 /404.html;
            location = /40x.html {
        }
        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }
}
[root@db01 ~]# systemctl restart nginx ##重启nginx

4.安装elasticearch插件

ingest-geoip一般是下载不下来的,可以点击ftp://www.lcvcqg.top/software/ingest-geoip-6.6.0.zip下载

[root@db01 ~]# /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent
[root@db01 ~]# /usr/share/elasticsearch/bin/elasticsearch-plugin install file:///root/ingest-geoip-6.6.0.zip 
-> Downloading file:///root/ingest-geoip-6.6.0.zip
[=================================================] 100%   
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.reflect.ReflectPermission suppressAccessChecks
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed ingest-geoip
[root@db01 ~]# systemctl restart elasticsearch ##重启es

5.修改nginx模块配置文件

[root@db01 ~]# vim/etc/filebeat/modules.d/nginx.yml 
- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    ##设置nginx访问日志路径
    var.paths: ["/var/log/nginx/access.log"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    ##设置nginx错误日志路径
    var.paths: ["/var/log/nginx/error.log"]
[root@db01 ~]# systemctl restart nginx ##重启nginx

6.回到kibana界面添加索引

image.png

选择read_timestamp

image.png image.png

选择read_timestamp

image.png

7.查看数据

点击discover 选择要查看的索引项

查看错误日志 image.png 查看记录日志 image.png