Filebeat简介

Filebeat是一款基于libbeat的轻量级日志采集器，用于监控日志文件、目录，收集事件并将其转发至Elasticsearch或Logstash。

工作基本原理

Filebeat支持多路输入。每个文件启动一个对应的harvester用于监控文件变化并将数据发送至libbeat。libbeat将日志数据汇聚，并发送至输出。

Filebeat原理

快速安装入门

linux版本下载地址

Filebeat

1. 安装

直接放到所需位置解压即可

tar xzvf filebeat-7.10.0-linux-x86_64.tar.gz

2. 配置连接到Elastic Static

filebeat.yml文件

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to. ES集群的host和端口
  hosts: ["localhost:9200"]

  # Protocol - either `http` (default) or `https`. 如果是https协议需要打开切换协议头
  #protocol: "https"

  # Authentication credentials - either API key or username/password. 
  # 如果需要输入API key或者用户名密码则开启配置
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

如果要使用kibana的默认dashboards

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
#如果kibana部署在本机上就不用配置
  host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

3. 启动和配置数据采集模块

1. 确认需要启动的模块，linux下执行：

./filebeat modules list

列出原生的采集模块，如果没有所需的模块，还可以手动配置input section

2. 启动需要的模块，以elasticsearch为例

./filebeat modules enable elasticsearch

3. 在modules.d目录下，修改已启动模块的配置，elasticsearch.yml

# Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.10/filebeat-module-elasticsearch.html

- module: elasticsearch
  # Server log
  server:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: 
      - /u01/elastic/logs/log-application.log

  gc:
    enabled: false
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  audit:
    enabled: false
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  slowlog:
    enabled: false
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  deprecation:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: 
        - /u01/elastic/logs/log-application_deprecation.log

4. 上传ES索引模板，配置kibana dashboards

./filebeat setup -e

5. 启动filebeat

sudo ./filebeat -e

查看kibana界面

ES的日志大概就是这个样子

kibana界面