filebeat安装配置
elasticsearch logstash kibana filebeat 最好版本一致 我使用的是7.6.2版本的
下载rpm包并安装
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.6.2-x86_64.rpm
sudo rpm -vi filebeat-7.6.2-x86_64.rpm
修改配置文件
filebeat.inputs:
- type: log
enabled: true
paths:
- /data/docker/containers/*/*.log
fields:
log_source: bymex.dev
fields_under_root: true
multiline.pattern: ^\d{4}-\d{1,2}-\d{1,2}
multiline.negate: true
multiline.match: after
scan_frequency: 5s
close_inactive: 1h
ignore_older: 24h
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
output.logstash:
hosts: ["a192.168.1.2:5044"]
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata:
host: "unix:///var/run/docker.sock"
match_source: true
match_source_index: 3
- add_kubernetes_metadata: ~
logging.level: info
修改logstash配置
在/etc/logstash/conf.d下添加文件docker.conf
input {
beats {
port => 5044
}
}
output {
if [log_source] == "tmex.dev" {
elasticsearch {
hosts => ["es01.loc:9200","es02.loc:9200","es03.loc:9200"]
index => "tmex.dev-%{+YYYY.MM.dd}"
}
}
if [log_source] == "tmex.pro" {
elasticsearch {
hosts => ["es01.loc:9200","es02.loc:9200","es03.loc:9200"]
index => "tmex.pro-%{+YYYY.MM.dd}"
}
}
if [log_source] == "bymex.dev" {
elasticsearch {
hosts => ["es01.loc:9200","es02.loc:9200","es03.loc:9200"]
index => "bymex.dev-%{+YYYY.MM.dd}"
}
}
}
说明
在filebeat收集docker日志时，通过fields的log_source标记日志来源，我这里是bymex.dev，说明对日志来自bymex.dev环境。配置日志输出到logstash，填写logstash的IP和端口。在logstash的docker.conf配置文件中，监听来源于5044端口的beat日志，输出到elasticsearch存储的时候，通过判断log_source来设置不同的索引存储到es库，这样就通过不同的索引查找区分不同来源的日志。
网友评论