ELK 搭建及实战

作者: 全村滴希望 | 来源:发表于2020-07-13 10:01 被阅读0次

ELK 搭建及实战

JDK1.8环境搭建和Kibana实战部署

ELK介绍和JDK1.8环境搭建

实战环境

Centos7
关闭Iptables / firewalld
关闭Selinux

ELK功能

Elasticsearch用来存储数据
Logstash用来收集数据
Kibana用来展现数据

实战环境

192.168.220.135 部署Kibana、ES
192.168.220.136 部署Logstash
JDK1.8环境搭建
- 安装JDK
- 配置环境变量

Yum安装jdk1.8，不建议

链接：https://www.elastic.co/downloads/logstash
yum install java-1.8.0-openjdk -y
Elasticsearch、Logstash依赖于java环境

JDK的二进制安装

Jdk1.8二进制包下载路径http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
下载账号 2696671285@qq.com=Oracle123
解压到对应安装目录/usr/local/或者/opt/
验证安装/usr/local/jdk1.8.0_201/bin/java -version

# 安装命令
cd /usr/local/src
tar -zxf jdk-8u201-linux-x64.tar.gz
mv jdk1.8.0_201 /usr/local/

# 配置Java环境变量/etc/profile
export JAVA_HOME=/usr/local/jdk1.8.0_201/
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH\

# 验证环境变量
java -version

Kibana 二进制安装和启动

ELK功能

Kibana用来展现数据
Elasticsearch用来存储数据
Logstash用来收集数据

ELK下载地址

ELK6版本
https://artifacts.elastic.co
ELK下载比较慢，提供百度网盘的下载链接

资源下载地址和提取码

链接：https://pan.baidu.com/s/1r4KsX9nKxuRbXWmdacAChg

提取码：cxcx

实战环境

192.168.237.135部署Kibana、ES
192.168.237.136部署Logstash

安装Kibana

下载Kibana二进制包
解压到/usr/local完成安装

Kibana安装脚本

cd /usr/local/src/
tar -zxf kibana-7.8.0-linux-x86_64.tar.gz
mv kibana-7.8.0-linux-x86_64 /usr/local/kibana-7.8.0

修改Kibana配置/usr/local/kibana-7.8.0/config/kibana.yml

server.port: 5601
server.host: "0.0.0.0"
#elasticsearch.url: "http://localhost:9200"
#elasticsearch.username: "user"
#elasticsearch.password: "pass"

Kibana的启动和访问

前台启动Kibana

/usr/local/kibana-7.8.0/bin/kibana --allow-root

后台启动Kibana

nohup /usr/local/kibana-7.8.0/bin/kibana  --allow-root >/tmp/kibana.log 2>/tmp/kibana.log &

访问Kibana，需要开放5601端口

Kibana的安全说明

默认无密码，也是谁都能够访问
如果使用云厂商，可以在安全组控制某个IP的访问
建议借用Nginx实现用户名密码登录

Kibana借用Nginx实现认证

默认的Kibana

任何人都能无密码访问Kibana
借用Nginx实现登录认证
Nginx控制源IP访问、Nginx可以使用用户名密码的方式

Kibana借用Nginx来实现简单认证

Kibana监听在127.0.0.1
部署Nginx，使用Nginx来转发

Nginx编译安装

yum install -y lrzsz wget gcc gcc-c++ make pcre pcre-devel zlib zlib-devel
cd /usr/local/src
wget 'http://nginx.org/download/nginx-1.18.0.tar.gz'
tar -zxvf nginx-1.18.0.tar.gz
cd nginx-1.18.0
./configure --prefix=/usr/local/nginx && make && make install

Nginx环境变量设置

export PATH=$PATH:/usr/local/nginx/sbin/
验证环境变量

Nginx两种限制

限制源IP访问，比较安全，访问的IP得不变
使用用户名密码的方式，通用

Nginx限制源IP访问

vi /usr/local/nginx/conf/nginx.conf

    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
           '$status $body_bytes_sent "$http_referer" '
           '"$http_user_agent" "$http_x_forwarded_for"';

  access_log logs/access.log main;

  server {
    listen    5602;
    location / {
      allow 127.0.0.1;
      allow 192.168.220.1;
      deny all;
      proxy_pass http://127.0.0.1:5601;
    }
  }
  
nginx -s reload

观察访问日志

/usr/local/nginx/logs/access.log
如果被拒绝了可以在日志里找到源IP

Nginx配置使用用户名密码的方式

# 创建用户名密码文件
# 这里用户名为 admin， 密码为 admin（需要使用openssl进行加密）
printf "admin:$(openssl passwd -1 admin)\n" >/usr/local/nginx/conf/htpasswd

vi /usr/local/nginx/conf/nginx.conf
location / {
      # allow 127.0.0.1;
      # allow 192.168.220.1;
      # deny all;
      auth_basic "elk auth";
      auth_basic_user_file /usr/local/nginx/conf/htpasswd;
      proxy_pass http://127.0.0.1:5601;
    }
  
nginx -s reload

# 访问测试

Elasticsearch实战部署和使用入门

Elasticsearch二进制安装和启动

ELK功能

Kibana用来展现数据
Elasticsearch用来存储数据
Logstash用来收集数据

Elasticsearch

使用Java开发，安装方便
Elasticsearch提供Http接口
Elasticsearch提供集群模式

Kibana网页访问问题

Kibana网页在Elasticsearch还没安装前无法访问
安装完Elasticsearch就好了

Elasticsearch的安装

下载二进制包
解压到对应目录完成安装/usr/local/
目录属主更新为elk，Elasticsearch无法用root启动

ES的安装脚本

cd /usr/local/src
tar -zxf elasticsearch-7.8.0.tar.gz
mv elasticsearch-7.8.0 /usr/local/

Elasticsearch配置

vi /usr/local/elasticsearch-7.8.0/config/elasticsearch.yml
path.data: /usr/local/elasticsearch-7.8.0/data
path.logs: /usr/local/elasticsearch-7.8.0/logs
network.host: 127.0.0.1
http.port: 9200

JVM的内存限制更改jvm.options

vi /usr/local/elasticsearch-7.8.0/config/jvm.options/
-Xms128M
-Xmx128M

Elasticsearch的启动，得用普通用户启动

useradd -s /sbin/nologin elk
chown -R elk:elk /usr/local/elasticsearch-7.8.0/
su - elk -s /bin/bash
/usr/local/elasticsearch-7.8.0/bin/elasticsearch -d

验证启动是否成功

观察日志

tail -f /usr/local/elasticsearch-7.8.0/logs/elasticsearch.log

观察Kibana网页

Elasticsearch启动注意事项

Elasticsearch启动注意

Elasticsearch如果启动在127.0.0.1的话，可以启动成功
Elasticsearch如果要跨机器通讯，需要监听在真实网卡上
监听在真实网卡需要调整系统参数才能正常启动

Elasticsearch监听在非127.0.0.1

监听在0.0.0.0或者内网地址
以上两种监听都需要调整系统参数

vi /usr/local/elasticsearch-7.8.0/config/elasticsearch.yml
network.host: 0.0.0.0

ES启动四个报错的处理


[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

[2]: max number of threads [3829] for user [elk] is too low, increase to at least [4096]

[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

[4]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

[elk@ES ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 3818
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 3818
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[elk@ES ~]$ cat /etc/security/limits.d/20-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     4096
root       soft    nproc     unlimited

[elk@ES ~]# sysctl -a | grep max_map_count
sysctl: reading key "net.ipv6.conf.all.stable_secret"
sysctl: reading key "net.ipv6.conf.default.stable_secret"
sysctl: reading key "net.ipv6.conf.ens33.stable_secret"
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
vm.max_map_count = 65530

错误1：最大文件打开数调整

vi /etc/security/limits.conf
* - nofile 65536

错误2：最大打开进程数调整

vi /etc/security/limits.d/20-nproc.conf
* - nproc 4096

错误3：内核参数调整

vi /etc/sysctl.conf
vm.max_map_count = 262144
[root@ES ~]# sysctl -p
vm.max_map_count = 262144

错误4：ip替换host1等，多节点请添加多个ip地址，单节点可写按默认来

vi /usr/local/elasticsearch-7.8.0/config/elasticsearch.yml
node.name: node-1 # 开放该行注释
cluster.initial_master_nodes: ["node-1","node-2"] #这里的node-1为node-name配置的值

Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/local/elasticsearch-7.8.0/data]] with lock id [0]; maybe these locations are not writable or mult
iple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

寻找主要信息：failed to obtain node locks

ps aux | grep ‘elasticsearch

重启 elasticsearch 可以看到监听在 0.0.0.0:9200和9300端口

su - elk -s /bin/bash
/usr/local/elasticsearch-7.8.0/bin/elasticsearch -d
netstat -lnp
tcp6       0      0 :::9200                 :::*                    LISTEN      8167/java
tcp6       0      0 :::9300                 :::*                    LISTEN      8167/java

访问 192.168.237.135:9200 可以看到返回的json

Elasticsearch监听网卡建议

如果学习，建议监听在127.0.0.1
如果是云服务器的话，一定把9200和9300公网入口在安全组限制一下
自建机房的话，建议监听在内网网卡，监听在公网会被入侵

用网页访问ES

Elasticsearch的基本操作

Elasticsearch的概念

索引 ->类似于 Mysql 中的数据库
类型 ->类似于 Mysql 中的数据表
文档 ->存储数据

Elasticsearch的数据操作

手动 curl 操作 Elasticsearch 会比较难
借用 Kibana 来操作 Elasticsearch

测试Web接口（确保kibana 和 elasticsearch 都成功启动）

浏览器访问 kibana
Kibana操作：在 kibana 首页菜单栏的 Management 中找到 Dev Tools，输入 GET / 运行，会看到es的返回

索引操作

创建索引
PUT /shijiange

删除索引
DELETE /shijiange

获取所有索引
GET /_cat/indices?v

Elasticsearch增删改查

ES插入数据
PUT /shijiange/users/1
{
 "name":"shijiange", 
 "age": 30
}

PUT /shijiange/users/2
{
 "name":"justdoit", 
 "age": 20
}

ES查询数据
GET /shijiange/users/1

GET /shijiange/_search?q=*

修改数据、覆盖, 此时会覆盖 /user/1的所有内容
PUT /shijiange/users/1
{
 "name": "justdoit",
 "age": 45
}

ES删除数据
DELETE /shijiange/users/1

修改某个字段、不覆盖
POST /shijiange/users/2/_update
{
 "doc": {
  "age": 29
 }
}

修改所有的数据
POST /shijiange/_update_by_query
{
 "script": {
  "source": "ctx._source['age']=30" 
 },
 "query": {
  "match_all": {}
 }
}

 
增加一个字段
POST /shijiange/_update_by_query
{
 "script":{
  "source": "ctx._source['city']='hangzhou'"
 },
 "query":{
  "match_all": {}
 }
}

Logstash实战部署和简单使用

Logstash二进制安装和启动

ELK功能

Kibana用来展现数据
Elasticsearch用来存储数据
Logstash用来收集数据

Logstash的安装

依赖于Java环境
下载二进制安装文件
解压到对应目录完成安装/usr/local/

# Logstash的安装脚本
cd /usr/local/src
tar -zxf logstash-7.8.0.tar.gz
mv logstash-7.8.0 /usr/local/

Logstash的JVM配置文件更新

vi /usr/local/logstash-7.8.0/config/jvm.options

-Xms200M
-Xmx200M

Logstash支持

Logstash分为输入、输出
输入：标准输入、日志等
输出：标准输出、ES等

Logstash最简单配置

vi /usr/local/logstash-7.8.0/config/logstash.conf

input{
  stdin{}
}

output{
  stdout{
    codec=>rubydebug
  }
}

Logstash 启动和测试

# 前台启动logStash
/usr/local/logstash-7.8.0/bin/logstash -f /usr/local/logstash-7.8.0/config/logstash.conf
# 后台启动logStash
nohup /usr/local/logstash-7.8.0/bin/logstash -f /usr/local/logstash-7.8.0/config/logstash.conf >/tmp/logstash.log 2>/tmp/logstash.log &
[2020-07-13T02:16:32,349][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
shijiange
{
          "host" => "Logstash",
       "message" => "shijiange",
      "@version" => "1",
    "@timestamp" => 2020-07-13T06:19:46.926Z
}

logstash启动较慢，因此使用 haveged 来调优

yum install -y epel-release
yum install -y haveged
systemctl enable haveged
systemctl start haveged

Logstash读取日志

vi /usr/local/logstash-7.8.0/config/logstash.conf

input {
  file {
    path => "/var/log/secure"
  }
}

output{
  stdout{
    codec=>rubydebug
  }
}

/var/log/secure 是登录日志内容，Logstash不会收集旧的日志，只会收集新的，当新的shell登录进服务器后，Logstash 会显示登录日志

[2020-07-13T02:31:35,125][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
/usr/local/logstash-7.8.0/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
    "@timestamp" => 2020-07-13T06:32:12.536Z,
          "path" => "/var/log/secure",
          "host" => "Logstash",
       "message" => "Jul 13 02:32:12 localhost sshd[1086]: Accepted password for root from 192.168.220.1 port 51120 ssh2",
      "@version" => "1"
}
{
    "@timestamp" => 2020-07-13T06:32:12.574Z,
          "path" => "/var/log/secure",
          "host" => "Logstash",
       "message" => "Jul 13 02:32:12 localhost sshd[1086]: pam_unix(sshd:session): session opened for user root by (uid=0)",
      "@version" => "1"
}

Logstash读取日志发送到ES

实战环境

192.168.220.135: es
192.168.220.136: logstash

Logstash和ES结合说明

Logstash支持读取日志发送到ES
但Logstash用来收集日志比较消耗内存，后面将对这个进行优化

Logstash配置发送日志到ES数据库 ( Logstash 要先配置 Nginx，然后启动)

vi /usr/local/logstash-7.8.0/config/logstash.conf

input {
  file {
    path => "/usr/local/nginx/logs/access.log"
  }
}

output {
  elasticsearch {
    hosts => ["http://192.168.220.135:9200"]
  }
}

重载配置

# 重启 logstash
ps -aux | grep logstash
root       1025  8.1 47.6 2383176 476408 pts/0  Sl+  02:30  26:53 /usr/local/jdk1.8.0_251//bin/java
kill -1 1025

# 浏览器访问 logstash 服务器地址，会请求到nginx，access日志 logstash 会发送到 ES
# 此时在 kibana 中 GET /_cat/indices?v 能够查到 es 多了一条 名为 logstash + 日期的索引

Logstash收集日志必要点

日志文件需要有新日志产生
Logstash跟Elasticsearch要能通讯

Kibana上查询数据

GET /logstash-2019.02.20/_search?q=*
Kibana上创建索引直接查看日志

kibana 菜单栏选择 Discover ，在Step 1 of 2: Define index pattern 的 Index pattern 中填入 logstash-* 查看所有日期的 logstash 日志，点击 next step。 Time Filter field name Refresh 选择 @timestamp 点击Create index pattern。创建成功后，再点击Discover 就可以看到直观的索引数据

Kibana简单查询

根据字段查询：message: "_msearch"
根据字段查询：选中查询

ELK流程

Logstash读取日志 -> ES存储数据 -> Kibana展现

ELK实战分析Nginx日志

正则表达式基础简介

发送整行日志存在的问题

整行message一般我们并不关心
需要对message进行段拆分，需要用到正则表达式

正则表达式

使用给定好的符号去表示某个含义
例如.代表任意字符
正则符号当普通符号使用需要加反斜杠

正则的发展

普通正则表达式
扩展正则表达式

普通正则表达式

. 任意一个字符

* 前面一个字符出现0次或者多次

[abc] 中括号内任意一个字符

[^abc] 非中括号内的字符

[0-9] 表示一个数字

[a-z] 小写字母

[A-Z] 大写字母

[a-zA-Z] 所有字母

[a-zA-Z0-9] 所有字母+数字

[^0-9] 非数字

^xx 以xx开头

xx$ 以xx结尾

\d 任何一个数字

\s 任何一个空白字符

扩展正则表达式，在普通正则符号再进行了扩展

? 前面字符出现0或者1次

+ 前面字符出现1或者多次

{n} 前面字符匹配n次

{a,b} 前面字符匹配a到b次

{,b} 前面字符匹配0次到b次

{a,} 前面字符匹配a或a+次

(string1|string2) string1或string2

简单提取IP

1.1.1.1 114.114.114.114 255.277.277.277

1-3个数字.1-3个数字.1-3个数字.1-3个数字

[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}

Logstash正则分析Nginx日志

Nginx日志说明

192.168.220.1 - - [13/Jul/2020:22:57:22 -0400] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36"

访问IP地址 - - [访问时间] "请求方式(GET/POST) /请求URL" 状态码响应body大小 "-" "Referer User Agent"

Logstash正则提取日志

需要懂得正则，Logstash支持普通正则和扩展正则
需要了解Grok，利用Kibana的Grok学习Logstash正则提取日志

Grok提取Nginx日志

Grok使用 (?<xxx>提取内容) 来提取xxx字段
提取客户端IP：(?<clientip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})
提取时间：\[(?<requesttime>[^ ]+ \+[0-9]+)\]

Grok提取Nginx日志

(?<clientip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) - - \[(?<requesttime>[^ ]+ \-?\+?[0-9]+)\] "(?<requesttype>[A-Z]+) (?<requesturl>[^ ]+) HTTP/\d.\d" (?<status>[0-9]+) (?<bodysize>[0-9]+) "[^"]+" "(?<ua>[^"]+)"

提取Tomcat等日志使用类似的方法

Logstash正则提取Nginx日志

vi /usr/local/logstash-7.8.0/config/logstash.conf

input {
  file {
    path => "/usr/local/nginx/logs/access.log"
  }
}

filter {
  grok {
    match => {
      "message" => '(?<clientip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) - - \[(?<requesttime>[^ ]+ \-?\+?[0-9]+)\] "(?<requesttype>[A-Z]+) (?<requesturl>[^ ]+) HTTP/\d.\d" (?<status>[0-9]+) (?<bodysize>[0-9]+) "[^"]+" "(?<ua>[^"]+)"'
    } 
  }
}

output {
  elasticsearch {
    hosts => ["http://192.168.220.135:9200"]
  }
}

重启 logstash ，进入 kibana 的 Discover 可以看到左侧多出现了 Available fields

注意正则提取失败的情况

echo "shijiange" >> /usr/local/nginx/logs/access.log、

会出现一个 tags 显示 _grokparsefailure

Logstash正则提取出错就不输出到ES

vi /usr/local/logstash-7.8.0/config/logstash.conf

output{
  if "_grokparsefailure" not in [tags] and "_dateparsefailure" not in [tags] {
    elasticsearch {
      hosts => ["http://192.168.220.135:9200"]
    }
  }
}

Logstash去除不需要的字段

去除字段注意

只能去除_source里的
非_source里的去除不了
remove_field => ["message","@version","path"]

Logstash配置去除不需要的字段

vi /usr/local/logstash-7.8.0/config/logstash.conf

filter {
  grok {
    match => {
      "message" => '(?<clientip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) - - \[(?<requesttime>[^ ]+ \-?\+?[0-9]+)\] "(?<requesttype>[A-Z]+) (?<requesturl>[^ ]+) HTTP/\d.\d" (?<status>[0-9]+) (?<bodysize>[0-9]+) "[^"]+" "(?<ua>[^"]+)"'
    }
    remove_field => ["message","@version","path"]
  }
}

去除字段

减小ES数据库的大小
提升搜索效率

ELK覆盖时间轴和全量分析Nginx

默认ELK时间轴

以发送日志的时间为准
而Nginx上本身记录着用户的访问时间
分析Nginx上的日志以用户的访问时间为准，而不以发送日志的时间

Logstash分析所有Nginx日志(包括之前的日志)

vi /usr/local/logstash-7.8.0/config/logstash.conf

input {
  file {
    path => "/usr/local/nginx/logs/access.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

Logstash的filter里面加入配置24/Feb/2019:21:08:34 +0800

vi /usr/local/logstash-7.8.0/config/logstash.conf

input {
  file {
    path => "/usr/local/nginx/logs/access.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  grok {
    match => {
      "message" => '(?<clientip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) - - \[(?<requesttime>[^ ]+ \-?\+?[0-9]+)\] "(?<requesttype>[A-Z]+) (?<requesturl>[^ ]+) HTTP/\d.\d" (?<status>[0-9]+) (?<bodysize>[0-9]+) "[^"]+" "(?<ua>[^"]+)"'
    }
    remove_field => ["message","@version","path"]
  }

  date {
    match => ["requesttime", "dd/MMM/yyyy:HH:mm:ss Z"]
    target => "@timestamp"
  }
}

output {
  elasticsearch {
    hosts => ["http://192.168.220.135:9200"]
  }
}

统计Nginx的请求和网页显示进行对比

cat /usr/local/nginx/logs/access.log |awk '{print $4}' |cut -b 1-19 |sort |uniq -c

测试

while true; do curl -d "aaaa=bbbb" http://192.168.220.136/get/users; sleep 1; done

不同的时间格式，覆盖的时候格式要对应

20/Feb/2019:14:50:06 -> dd/MMM/yyyy:HH:mm:ss
2016-08-24 18:05:39,830 -> yyyy-MM-dd HH:mm:ss,SSS
如果时间解析失败，会报 _dateparsefailure 错误，此时 @timestamp 还原为日志发送时间

ELK架构引入Filebeat

Filebeat二进制安装与启动

Logstash收集日志

依赖于Java环境，用来收集日志比较重，占用内存和CPU
Filebeat相对轻量，占用服务器资源小
一般选用Filebeat来进行日志收集

Filebeat的安装

下载二进制文件
解压移到对应的目录完成安装/usr/local/

Filebeat的二进制安装

cd /usr/local/src/
tar -zxvf filebeat-7.8.0-linux-x86_64.tar.gz
mv filebeat-7.8.0-linux-x86_64 /usr/local/filebeat-7.8.0

部署服务介绍

192.168.220.135 部署Kibana、ES
192.168.220.136 部署Filebeat

Filebeat发送日志到ES配置/usr/local/filebeat-7.8.0/filebeat.yml

vi /usr/local/filebeat-7.8.0/filebeat.yml

filebeat.inputs:
- type: log
  tail_files: true
  backoff: "1s"
  paths:
      - /usr/local/nginx/logs/access.log

output:
  elasticsearch:
    hosts: ["192.168.220.135:9200"]

启动Filebeat

前台启动

/usr/local/filebeat-7.8.0/filebeat -e -c /usr/local/filebeat-7.8.0/filebeat.yml

后台启动

nohup /usr/local/filebeat-7.8.0/filebeat -e -c /usr/local/filebeat-7.8.0/filebeat.yml >/tmp/filebeat.log 2>&1 &

Kibana上查看日志数据

GET /filebeat-7.8.0-year.month.day-000001/_search?q=*
创建索引观察

kibana 菜单栏的 Stack Management 中，点击 Kibana 的Index Patterns 创建索引

Filebeat -> ES -> Kibana

适合查看日志
不适合具体日志的分析

Filebeat+Logstash新架构

Filebeat和Logstash说明

Filebeat：轻量级，但不支持正则、不能移除字段等
Logstash：比较重，但支持正则、支持移除字段等

搭建架构演示

Logstash -> Elasticsearch -> Kibana
Filebeat -> Elasticsearch -> Kibana
Filebeat -> Logstash -> Elasticsearch -> Kibana

部署服务介绍

192.168.220.135 部署Kibana、ES
192.168.220.136 部署Logstash、Filebeat

Filebeat配置发往Logstash

vi /usr/local/filebeat-7.8.0/filebeat.yml

filebeat.inputs:
- type: log
  tail_files: true
  backoff: "1s"
  paths:
      - /usr/local/nginx/logs/access.log

output:
  logstash:
    hosts: ["192.168.220.136:5044"]

Logstash配置监听在5044端口，接收Filebeat发送过来的日志

vi /usr/local/logstash-7.8.0/config/logstash.conf

input {
  file {
    path => "/usr/local/nginx/logs/access.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

# 改为

input {
  beats {
    host => '0.0.0.0'
    port => 5044
  }
}

Kibana上查看数据

GET /logstash/_search?q=*
创建索引观察

kibana 菜单栏的 Stack Management 中，点击 Kibana 的Index Patterns 创建索引

Logstash上移除不必要的字段

Filebeat发过来的无用字段比较多
remove_field => ["message","@version","path","beat","input","log","offset","prospector","source","tags"]

Filebeat批量部署比Logstash要方便得多

Logstash监听在内网
Filebeat发送给内网的Logstash

新架构

Filebeat(多台) -> Logstash(正则) -> Elasticsearch(入库) -> Kibana展现

ELK采集Json格式日志

Json的好处

原生日志需要做正则匹配，比较麻烦
Json格式的日志不需要正则能直接分段采集

Nginx使用Json格式日志

vi /usr/local/nginx/conf/nginx.conf

#access_log  logs/access.log  main;
log_format json '{"@timestamp":"$time_iso8601",'
         '"clientip":"$remote_addr",'
         '"status":$status,'
         '"bodysize":$body_bytes_sent,'
         '"referer":"$http_referer",'
         '"ua":"$http_user_agent",'
         '"handletime":$request_time,'
         '"url":"$uri"}';
access_log logs/access.log;
access_log logs/access.json.log json;

部署服务介绍

192.168.220.135 部署Kibana、ES
192.168.220.136 部署Logstash、Filebeat

Filebeat采集Json格式的日志

vi /usr/local/filebeat-7.8.0/filebeat.yml

filebeat.inputs:
- type: log
  tail_files: true
  backoff: "1s"
  paths:
    - /usr/local/nginx/logs/access.json.log

output:
 logstash:
  hosts: ["192.168.220.136:5044"]

Logstash解析Json日志

vi /usr/local/logstash-7.8.0/config/logstash.conf

input {
  beats {
    host => '0.0.0.0'
    port => 5044
  }
}

filter {
  json {       
    source => "message"
    remove_field => ["message","@version","path","beat","input","log","offset","prospector","source","tags"]
  }
}

output {
  elasticsearch {
    hosts => ["http://192.168.220.135:9200"]
  }
}

Filebeat采集多个日志

采集多个日志

收集单个Nginx日志
如果有采集多个日志的需求

Filebeat采集多个日志配置

vi /usr/local/filebeat-7.8.0/filebeat.yml

filebeat.inputs:
- type: log
  tail_files: true
  backoff: "1s"
  paths:
      - /usr/local/nginx/logs/access.json.log
  fields:
    type: access
  fields_under_root: true
- type: log
  tail_files: true
  backoff: "1s"
  paths:
      - /var/log/secure
  fields:
    type: secure
  fields_under_root: true
 
output:
  logstash:
    hosts: ["192.168.220.136:5044"]

Logstash如何判断两个日志

Filebeat加入一字段用来区别
Logstash使用区别字段来区分

Logstash通过type字段进行判断

vi /usr/local/logstash-7.8.0/config/logstash.conf

input {
  beats {
    host => '0.0.0.0'
    port => 5044
  }
}

filter {
  if [type] == "access" {
    json {       
      source => "message"
      remove_field => ["message","@version","path","beat","input","log","offset","prospector","source","tags"]
    }
  }
}

output{
  if [type] == "access" {
    elasticsearch {
      hosts => ["http://192.168.220.135:9200"]
      index => "access-%{+YYYY.MM.dd}"
    }
 }
 else if [type] == "secure" {
   elasticsearch {
     hosts => ["http://192.168.220.135:9200"]
     index => "secure-%{+YYYY.MM.dd}"
   }
 }
}

网页上建立索引

access索引
secure索引

kibana 菜单栏的 Stack Management 中，点击 Kibana 的Index Patterns 创建索引

ELK架构引入缓存Redis或Kafka

Redis服务器的编译安装

之前架构

Filebeat(多台) -> Logstash(正则) -> Elasticsearch(入库) -> Kibana展现

架构存在的问题

Logstash性能不足的时候
扩容Logstash，Filebeat的配置可能会不一致。（如果有多台Logstash，不同的Filebeat需要配置到不同的Logstash）

架构优化

Filebeat(多台) Logstash

Filebeat(多台) -> Redis、Kafka -> Logstash(正则) -> Elasticsearch(入库) -> Kibana展现

Filebeat(多台) Logstash

部署服务介绍

192.168.220.135: Kibana、ES
192.168.220.136: Logstash、Filebeat、Redis

Redis服务器搭建

yum install -y wget net-tools gcc gcc-c++ make tar openssl openssl-devel cmake
cd /usr/local/src
wget 'http://download.redis.io/releases/redis-4.0.9.tar.gz'
tar -zxf redis-4.0.9.tar.gz
cd redis-4.0.9
make
mkdir -pv /usr/local/redis/conf /usr/local/redis/bin
cp src/redis* /usr/local/redis/bin/
cp redis.conf /usr/local/redis/conf

验证Redis服务器

更改Redis配置（bind、daemon、dir、requirepass）

vi /usr/local/redis/conf/redis.conf

bind 0.0.0.0

daemonize yes

dir /tmp/

requirepass 1234qwer

密码设置为1234qwer-
验证set、get操作

Redis的启动命令

/usr/local/redis/bin/redis-server /usr/local/redis/conf/redis.conf

Redis的简单操作

/usr/local/redis/bin/redis-cli
auth '1234qwer'
set name shijiange

get name

[root@Logstash redis-4.0.9]# /usr/local/redis/bin/redis-cli
127.0.0.1:6379> info
NOAUTH Authentication required.
127.0.0.1:6379> auth '1234qwer'
OK
127.0.0.1:6379> info

Filebeat和Logstash间引入Redis

部署服务介绍

192.168.220.135: Kibana、ES
192.168.220.136: Logstash、Filebeat、Redis

Filebeat配置写入到Redis

vi /usr/local/filebeat-7.8.0/filebeat.yml

filebeat.inputs:
- type: log
  tail_files: true
  backoff: "1s"
  paths:
      - /usr/local/nginx/logs/access.json.log
  fields:
    type: access
  fields_under_root: true

output.redis:
  hosts: ["192.168.220.136"]
  port: 6379
  password: "1234qwer"
  key: "access"

2020-07-24T01:08:07.223-0400 INFO instance/beat.go:310 Setup Beat: filebeat; Version: 7.8.0
2020-07-24T01:08:07.224-0400 INFO instance/beat.go:436 filebeat stopped.
2020-07-24T01:08:07.224-0400 ERROR instance/beat.go:958 Exiting: error initializing publisher: 1 error: setting 'output.redis.port' has been removed
Exiting: error initializing publisher: 1 error: setting 'output.redis.port' has been removed

如果出现以上错误，可能是版本不匹配的问题，将redis版本升级，或者将filebeat版本降级，这里降级为 6.0.6

此时可以看到 filebeat 的输出为

2020-07-24T01:24:52.555-0400 INFO log/harvester.go:255 Harvester started for file: /usr/local/nginx/logs/access.json.log
2020-07-24T01:24:53.556-0400 INFO pipeline/output.go:95 Connecting to redis(tcp://192.168.220.136:6379)
2020-07-24T01:24:53.557-0400 INFO pipeline/output.go:105 Connection to redis(tcp://192.168.220.136:6379) established

查看 redis 中的记录

127.0.0.1:6379> keys *
...
db0:keys=2,expires=0,avg_ttl=0
127.0.0.1:6379> keys *
1) "name"
2) "access"
127.0.0.1:6379> LRANGE access 0-1
(error) ERR wrong number of arguments for 'lrange' command
127.0.0.1:6379> LRANGE access 0 -1
1) "{\"@timestamp\":\"2020-07-24T05:24:52.555Z\",\"@metadata\":...
2) "{\"@timestamp\":\"2020-07-24T05:24:53.556Z\",\"@metadata\":...

Logstash从Redis中读取数据

vi /usr/local/logstash-7.8.0/config/logstash.conf

input {
  redis {
    host => '192.168.220.136'
    port => 6379
    key => "access"
    data_type => "list"
    password => '1234qwer'
  }
}

架构优化

Filebeat(多台) Logstash

Filebeat(多台) -> Redis、Kafka -> Logstash(正则) -> Elasticsearch(入库) -> Kibana展现

ELK 搭建及实战

ELK 搭建及实战

JDK1.8环境搭建和Kibana实战部署

ELK介绍和JDK1.8环境搭建

Kibana 二进制安装和启动

Kibana借用Nginx实现认证

Elasticsearch实战部署和使用入门

Elasticsearch二进制安装和启动

Elasticsearch启动注意事项

Elasticsearch的基本操作

Logstash实战部署和简单使用

Logstash二进制安装和启动

Logstash读取日志发送到ES

ELK实战分析Nginx日志

正则表达式基础简介

Logstash正则分析Nginx日志

Logstash去除不需要的字段

ELK覆盖时间轴和全量分析Nginx

ELK架构引入Filebeat

Filebeat二进制安装与启动

Filebeat+Logstash新架构

ELK采集Json格式日志

Filebeat采集多个日志

ELK架构引入缓存Redis或Kafka

Redis服务器的编译安装

Filebeat和Logstash间引入Redis

Kafka服务器二进制安装

Filebeat和Logstash间引入Kafka

ELK7安全认证功能实战

Kibana7二进制安装

Elasticsearch7二进制认证功能

Logstash7二进制安装

Kibana图表使用实战

Kibana图表使用实战

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读