收集tomcat access访问日志

采用filebeat收集，logstash过滤，es索引存储的架构。
配置含义参考本文集其他文章。

一、Filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/tomcat/apache-tomcat-8.5.39/logs/localhost_access_log.*.txt
  tags: ["tomcat_access"]

output.logstash:
  hosts: ["xx.xx.xx.xx:5044"]  #logstash的端口，默认5044，可以修改

二、logstash

1 input.conf

input {
  #tomcat access log
  beats {
    port => "5044"
  }
}

2 tomcat_out.conf

#过滤器
filter {
    if "tomcat_access" in [tags] {
        grok {
            match => ["message", "%{IPORHOST:client_ip} (%{USER:ident}|-) (%{USER:auth}|-) \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:http_version})?|-)\" %{NUMBER:response} (%{NUMBER:bytes}|-)" ]
            remove_field => "message"
        }
        date {  #时间格式转换插件，将过滤字段@timestamp的时间重置为日志时间
            match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
            target => "@timestamp"
            remove_field => "timestamp"
        }
    }
}

output {
        #此配置为输出到控制台，调试时可以加，正式环境就可以去掉
    stdout {codec => rubydebug } 

    elasticsearch {
        hosts  => ["192.168.18.2:9200"]
        index => "web-tomcat-%{+YYYY.MM.dd}"
    }
}

如果是和system模块日志一起收，可以这么写

filter {
    if "tomcat_access" in [tags] {
        grok {
            match => ["message", "%{IPORHOST:client_ip} (%{USER:ident}|-) (%{USER:auth}|-) \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:http_version})?|-)\" %{NUMBER:response} (%{NUMBER:bytes}|-)" ]
            remove_field => "message"
        }
        date {
            match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
            target => "@timestamp"
            remove_field => "timestamp"
        }
    }
}


output {

    if "tomcat_access" in [tags] {
        elasticsearch {
            hosts => ["192.168.18.2:9200"]
            index => "web-tomcat-%{+YYYY.MM.dd}"
        }
    }

    if [event][module] == "system" {
        elasticsearch {
            hosts => ["192.168.18.2:9200"]
            index => "os-linux-%{+YYYY.MM.dd}"
        }
    }

}

三、tomcat access源日志

16.128.231.178 - - [11/Aug/2020:10:56:36 +0800] "GET /testcloud/getHouseholdRegister?idcard=512913560570 HTTP/1.1" 200 64
13.144.212.111 - - [11/Aug/2020:10:56:36 +0800] "GET /testcloud/gateway/Register?idcard=45135311007X HTTP/1.1" 200 342