第十一章: filebeat使用module收集普通格式的nginx日志
1.删除以前的es索引和kibana索引
2.确认Nginx日式知否为普通格式
systemctl stop nginx
rm -rf /var/log/nginx/*
自己修改日志格式为main的普通格式
systemctl start nginx
3.安装nginx模块所需的插件
cd /usr/share/elasticsearch/
./bin/elasticsearch-plugin install file:///root/ingest-geoip-6.6.0.zip
./bin/elasticsearch-plugin install file:///root/ingest-user-agent-6.6.0.zip
systemctl restart elasticsearch
4.检查filebeat配置文件里是否包含模块相关参数
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 10s
5.激活filebeat模块并查看
filebeat modules list
filebeat modules enable nginx
6.配置filebeat的nginx模块
[root@web01 ~]# cat /etc/filebeat/modules.d/nginx.yml
- module: nginx
access:
enabled: true
var.paths: ["/var/log/nginx/access.log"]
error:
enabled: true
var.paths: ["/var/log/nginx/error.log"]
7.filebeat配置
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 10s
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
source: "/var/log/nginx/access.log"
- index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
source: "/var/log/nginx/error.log"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
8.配置nginx日志为正常日志
vim /etc/nginx/nginx.conf
access_log /var/log/nginx/access.log main; #打开注释 改成普通日志格式
nginx -t
systemctl restart nginx
> /var/log/nginx/access.log
tail -1 /var/log/nginx/access.log
9.重启filebeat
systemctl restart filebeat
10.kibana添加索引
注意:
error添加的时候选择 read_timestamp
11.查看日志是否被解析成了json格式
第十二章: filebeat使用模块收集mysql慢日志
1.配置mysql错误日志和慢日志路径
编辑my.cnf
[mysqld]
slow_query_log=ON
slow_query_log_file=/var/log/mariadb/slow.log
long_query_time=1
2.重启mysql并制造慢日志
systemctl restart mysql
慢日志制造语句
select sleep(2) user,host from mysql.user ;
3.确认慢日志和错误日志确实有生成
mysql -uroot -poldboy123 -e "show variables like '%slow_query_log%'"
4.激活filebeat的mysql模块
filebeat module enable mysql
5.配置mysql的模块
module: mysql
error:
enabled: true
var.paths: ["/var/log/mariadb/mariadb.log"]
slowlog:
enabled: true
var.paths: ["/var/log/mariadb/slow.log"]
6.配置filebeat根据日志类型做判断
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 10s
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "mysql_slowlog-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
fileset.module: "mysql"
fileset.name: "slowlog"
- index: "mysql_error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
fileset.module: "mysql"
fileset.name: "error"
setup.template.name: "mysql"
setup.template.pattern: "mysql_*"
setup.template.enabled: false
setup.template.overwrite: true
7.重启filebeat
systemctl restart filebeat
第十三章: filebeat收集docker类型日志
1.安装dockder
yum install -y yum-utils device-mapper-persistent-data lvm2
wget -O /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo
sed -i 's+download.docker.com+mirrors.tuna.tsinghua.edu.cn/docker-ce+' /etc/yum.repos.d/docker-ce.repo
yum makecache fast
yum install docker-ce -y
mkdir -p /etc/docker
tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://ig2l319y.mirror.aliyuncs.com"]
}
EOF
systemctl daemon-reload
systemctl restart docker
2.启动2个Nginx容器并访问测试
docker run -d -p 80:80 nginx
docker run -d -p 8080:80 nginx
http://10.0.0.51:8080
http://10.0.0.51
3.配置filebeat
vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: docker
containers.ids:
- '*'
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
index: "docker-%{[beat.version]}-%{+yyyy.MM}"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
4.重启filebeat
systemctl restart filebeat
5.生成测试访问数据
curl 10.0.0.51/1111111
curl 10.0.0.51/2222222
第十四章: filebeat收集docker日志可以早下班版
1.假设的场景
nginx容器 80端口
mysql容器 8080端口
2.理想中的索引名称
docker-nginx-6.6.0-2020.XX
docker-mysql-6.6.0-2020.XX
3.理想的日志记录格式
nginx容器日志：
{
"log": "xxxxxx",
"stream": "stdout",
"time": "xxxx"，
"service": "nginx"
}
mysql容器日志：
{
"log": "xxxxxx",
"stream": "stdout",
"time": "xxxx"，
"service": "mysql"
}
4.docker-compose配置
yum install docker-compose -y
cat >docker-compose.yml<<EOF
version: '3'
services:
nginx:
image: nginx:latest
labels:
service: nginx
logging:
options:
labels: "service"
ports:
- "80:80"
db:
image: nginx:latest
labels:
service: db
logging:
options:
labels: "service"
ports:
- "8080:80"
EOF
5.删除旧的容器
docker stop $(docker ps -q)
docker rm $(docker ps -qa)
6.启动容器
docker-compose up -d
7.配置filebeat
cat >/etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/lib/docker/containers/*/*-json.log
json.keys_under_root: true
json.overwrite_keys: true
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "docker-nginx-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
attrs.service: "nginx"
- index: "docker-db-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
attrs.service: "db"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
8.重启filebeat
systemctl restart filebeat
9.生成访问日志
curl 127.0.0.1/nginxxxxxxxxxxx
curl 127.0.0.1:8080/dbbbbbbbbb
网友评论