一.Filebeat介绍
Filebeat附带预构建的模块，这些模块包含收集、解析、充实和可视化各种日志文件格式数据所需的配置，每个Filebeat模块由一个或多个文件集组成，这些文件集包含摄取节点管道、Elasticsearch模板、Filebeat勘探者配置和Kibana仪表盘。
Filebeat模块很好的入门，它是轻量级单用途的日志收集工具，用于在没有安装java的服务器上专门收集日志，可以将日志转发到logstash、elasticsearch或redis等场景中进行下一步处理
为什么使用filebeat?
filebeat比logstash占用更少的系统资源，特别是内存。
二.使用filebeat收集nginx日志
2.1使用filebeat收集普通的nginx日志
1.安装Nginx
cat >/etc/yum.repos.d/nginx.repo <<EOF
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/\$releasever/\$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/\$releasever/\$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
EOF
yum install nginx -y
systemctl start nginx
curl 127.0.0.1
2.配置Nginx并创建测试页面
rm -rf /etc/nginx/conf.d/default.conf
cat >/etc/nginx/conf.d/www.conf<<EOF
server {
listen 80;
server_name localhost;
location / {
root /code/www;
index index.html index.htm;
}
}
EOF
mkdir /code/www/ -p
echo "db01-www" > /code/www/index.html
nginx -t
systemctl restart nginx
curl 127.0.0.1
tail -f /var/log/nginx/access.log
3.安装filebet
rpm -ivh filebeat-6.6.0-x86_64.rpm
4.配置filebeat
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
EOF
5.启动测试
systemctl start filebeat
6.检查结果
tail -f /var/log/filebeat/filebeat
curl -s 127.0.0.1:9200/_cat/indices|awk '{print $3}'
7.es-head查看
8.kabana查看
说明：这样收集的日志，信息全是在messge这个字段，还是无法分离我们想要查看的内容。
2.2.filebeat收集Nginx的json格式日志
1.修改nginx配置文件使日志转换成json
log_format json '{ "time_local": "$time_local", '
'"remote_addr": "$remote_addr", '
'"referer": "$http_referer", '
'"request": "$request", '
'"status": $status, '
'"bytes": $body_bytes_sent, '
'"agent": "$http_user_agent", '
'"x_forwarded": "$http_x_forwarded_for", '
'"up_addr": "$upstream_addr",'
'"up_host": "$upstream_http_host",'
'"upstream_time": "$upstream_response_time",'
'"request_time": "$request_time"'
' }';
access_log /var/log/nginx/access.log json;
2.清除旧日志
> /var/log/nginx/access.log
3.检查并重启nginx
nginx -t
systemctl restart nginx
4.修改filebeat配置文件支持json解析
说明：由于filebeat是go语言开发的，所以默认的是不支持json解析的，需要额外配置。配置如下：
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
EOF
5.删除ES里以前的索引
es-head >> filebeat-6.6.0-2019.11.15 >> 动作 >>删除
6.重启filebeat
systemctl restart filebeat
7.es-head查看
8.kabana查看
没有配置filebeat的结果为：
配置的结果为：
2.3.filebeat自定义ES索引名称
1.理想的索引的名称要与收集的对象的日志相关，以便区分。
例如 nginx-6.6.0-2020.02
2.filebeat配置
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
3.测试访问后es-head查看
4.kabana添加新的模板查看
2.4.filebeat按照服务类型拆分索引
1.两种配置方法
1.第一种写法
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
source: "/var/log/nginx/access.log"
- index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
source: "/var/log/nginx/error.log"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
2.第二种写法：
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["error"]
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "access"
- index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "error"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
2.重启filebeat
systemctl restart filebeat
3.es-head查看
4.kabana查看结果
2.5.多服务器收集相同的日志合并
说明：默认是会自动合并
新开一台服务器，配置与之前的一样。
1.es-head查看结果
2.kabana查看结果
一起查询
分离查询所需查看的结果
2.6.使用filebeat模块收集nginx日志
官方配置说说明请参考：https://www.elastic.co/guide/en/beats/filebeat/6.6/filebeat-module-nginx.html
说明：之前的收集需要修改nginx日志格式为json以及配置filebeat支持解析json格式，这样的使用适合新的环境进行使用，如果后期才上线日志收集这一块，使用filebeat模块收集日志可以不用修改原有的普通日志的格式和修改filebeat的配置文件。
0.配置es支持nginx模块的插件
cd /usr/share/elasticsearch/
./bin/elasticsearch-plugin install file:///root/ingest-geoip-6.6.0.zip
./bin/elasticsearch-plugin install file:///root/ingest-user-agent-6.6.0.zip
systemctl restart elasticsearch
1.配置filebeat配置文件，配置支持模块功能
#默认配置是faslse
============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
#精简并修改配置文件为ture
[root@nginx ~]# vim /etc/filebeat/filebeat.yml
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 10s
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
event.dataset: "nginx.access"
- index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
event.dataset: "nginx.error"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
2.查看支持的模块
[root@nginx ~]# filebeat modules list
Enabled: #表示已经开启的模块
Disabled: #表示所有支持的模块或未开启的模块
apache2
auditd
elasticsearch
haproxy
icinga
iis
kafka
kibana
logstash
mongodb
mysql
nginx
osquery
postgresql
redis
suricata
system
traefik
3.开启nginx模块
[root@nginx ~]# filebeat modules enable nginx
Enabled nginx
[root@nginx ~]# filebeat modules list
Enabled:
nginx
Disabled:
apache2
auditd
elasticsearch
haproxy
icinga
iis
kafka
kibana
logstash
mongodb
mysql
osquery
postgresql
redis
suricata
system
traefik
4.配置nginx模块
#默认格式
[root@nginx /etc/filebeat]# vim modules.d/nginx.yml
- module: nginx
# Access logs
access:
denabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Error logs
error:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
#修改并简化配置文件
[root@nginx /etc/filebeat/modules.d]# vim nginx.yml
- module: nginx
access:
enabled: true
var.paths: ["/var/log/nginx/access.log"]
error:
enabled: true
var.paths: ["/var/log/nginx/error.log"]
5.启动filebeat
systemctl start filebeat.service
6.es-head查看结果
7.kabana界面创建索引模板并查看
说明：模块中字段拆分的更加详细
三.使用filebeat收集tomccat日志
1.修改tomact的日志文件问json
#默认格式
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log." suffix=".txt" pattern="%h %l %u %t "%r" %s %b" />
#修改为json格式
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log." suffix=".txt" pattern="{"clientip":"%h",&quo
t;ClientUser":"%l","authenticated":"%u","AccessTime&quo
t;:"%t","method":"%r","status":"%s","Sen
dBytes":"%b","Query?string":"%q","partner":"%
{Referer}i","AgentVersion":"%{User-Agent}i"}"/>
2.filebeat配置文件设置
[root@tomcat ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /usr/local/tomcat/logs/localhost_access_log.*.txt
json.keys_under_root: true
json.overwrite_keys: true
tags: ["tomcat"]
output.elasticsearch:
hosts: ["10.0.0.72:9200"]
index: "tomcat_access-%{[beat.version]}-%{+yyyy.MM}"
setup.template.name: "tomcat"
setup.template.pattern: "tomcat_*"
setup.template.enabled: false
setup.template.overwrite: true
3.重启filebeat
[root@tomcat ~]# systemctl restart filebeat
4.测试访问查看es-head
5.kabana
四. filebeat收集java多行匹配模式
官方配置请参考链接：https://www.elastic.co/guide/en/beats/filebeat/6.6/multiline-examples.html
1.filebeat配置文件
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/elasticsearch/elasticsearch.log
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
index: "es-%{[beat.version]}-%{+yyyy.MM}"
setup.template.name: "es"
setup.template.pattern: "es-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
2.重启filebeat
systemctl restart filebeat
五.使用filebeat模块收集mysql慢日志和错误日志
1.配置mysql错误日志和慢日志路径
编辑my.cnf
vim /etc/my.cnf
[mysqld]
slow_query_log=ON
slow_query_log_file=/data/mysql/data/slow.log
long_query_time=1
2.重启mysql并制造慢日志
systemctl restart mysql
慢日志制造语句
select sleep(2) user,host from mysql.user ;
3.确认慢日志和错误日志确实有生成
mysql -uroot -poldboy123 -e "show variables like '%slow_query_log%'"
4.激活filebeat的mysql模块
filebeat module enable mysql
5.配置mysql的模块
[root@db05 ~]# vim /etc/filebeat/modules.d/mysql.yml
- module: mysql
# Error logs
error:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/application/mysql/data/error.log"]
# Slow logs
slowlog:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/application/mysql/data/slow.log"]
6.配置filebeat根据日志类型做判断
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 10s
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "mysql-slow-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
source: "/application/mysql/data/slow.log"
- index: "mysql-err-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
source: "/application/mysql/data/error.log"
setup.template.name: "mysql"
setup.template.pattern: "mysql-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
7.重启filebeat
systemctl restart filebeat
8.es-head查看
六.使用filebeat收集docker日志
filebeat收集docker日志终极杀人王火云邪神版
1.需求分析
json格式并且按照下列索引生成
docker-nginx-access-6.6.0-2020.02
docker-db-access-6.6.0-2020.02
docker-db-error-6.6.0-2020.02
docker-nginx-error-6.6.0-2020.02
3.创建新容器并挂载本地的目录到容器的日志文件目录下
docker run -d -p 80:80 -v /opt/nginx:/var/log/nginx nginx
docker run -d -p 8080:80 -v /opt/mysql:/var/log/nginx nginx
4.准备json格式的nginx配置文件并拷贝到容器里并重启
docker cp nginx.conf 5d62b35651e6:/etc/nginx/
docker cp nginx.conf 310e85addbcd:/etc/nginx/
docker stop $(docker ps -qa)
docker start Nginx容器的ID
docker start mysql容器的ID
5.配置filebeat配置文件
cat >/etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /opt/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["nginx_access"]
- type: log
enabled: true
paths:
- /opt/nginx/error.log
tags: ["nginx_err"]
- type: log
enabled: true
paths:
- /opt/mysql/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["db_access"]
- type: log
enabled: true
paths:
- /opt/mysql/error.log
tags: ["db_err"]
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "docker-nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "nginx_access"
- index: "docker-nginx-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "nginx_err"
- index: "docker-db-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "db_access"
- index: "docker-db-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "db_err"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
6.启动filebeat
systemctl restart filebeat
7.访问并测试
curl 127.0.0.1
curl 127.0.0.1:8080/
cat /opt/nginx/access.log
cat /opt/mysql/access.log
8.es-head查看
网友评论