elk（二）

作者: Freestyle_0f85 | 来源:发表于2020-01-21 20:13 被阅读0次

image.png

第一章: filbeat模块module
作用:
可以将特定的服务的普通日志转成json格式

1.查看filbeat模块路径
rpm -qc filebeat 

2.配置模块
filebeat添加模块相关参数
=============================
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s
=============================

3.查看并激活模块
filebeat modules list
filebeat modules enable nginx
filebeat modules list

4.配置filebeat的nginx模块
[root@db-01 ~]# cat /etc/filebeat/modules.d/nginx.yml 
- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/bbs.log"]

  error:
    enabled: true
    var.paths: ["/var/log/nginx/error.log"]

5.配置filebeat根据日志类型做判断
[root@db-01 ~]# cat /etc/filebeat/filebeat.yml 
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "nginx_bbs_access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        fileset.name: "access"
    - index: "nginx_error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        fileset.name: "error"

setup.template.name: "nginx"
setup.template.pattern: "nginx_*"
setup.template.enabled: false
setup.template.overwrite: true

6.配置nginx日志为正常日志
nginx -t
systemctl restart nginx
> /var/log/nginx/bbs.log 
tail -1 /var/log/nginx/bbs.log  

7.安装es插件
cd /usr/share/elasticsearch/
./bin/elasticsearch-plugin install file:///data/soft/ingest-geoip-6.6.0.zip
./bin/elasticsearch-plugin install file:///data/soft/ingest-user-agent-6.6.0.zip

8.重启es
systemctl restart elasticsearch
删除以前的旧索引

9.重启filebeat
systemctl restart filebeat

10.kibana添加索引
注意:
error添加的时候选择 read_timestamp

11.查看日志是否被解析成了json格式


报错1:
[root@db-01 ~]# filebeat modules list
Error in modules manager: modules management requires 'filebeat.config.modules.path' setting

报错2:
2019-09-11T09:04:40.562+0800    ERROR   pipeline/output.go:100  Failed to connect to backoff(elasticsearch(http://10.0.0.51:9200)): Connection marked as failed because the onConnect callback failed: Error loading pipeline for fileset nginx/access: This module requires the following Elasticsearch plugins: ingest-user-agent, ingest-geoip. You can install them by running the following commands on all the Elasticsearch nodes:
    sudo bin/elasticsearch-plugin install ingest-user-agent
    sudo bin/elasticsearch-plugin install ingest-geoip

第二章: filebeat使用模块收集mysql日志

yum -y install mariadb mariadb-server
1.配置mysql错误日志和慢日志路径
编辑my.cnf
log-error=/var/lib/mysql/error.log
slow_query_log=ON
slow_query_log_file=/var/lib/mysql/slow.log
long_query_time=3

2.重启mysql并制造慢日志
systemctl restart mysql 
慢日志制造语句
select sleep(2) user,host from mysql.user ;

3.确认慢日志和错误日志确实有生成

4.激活filebeat的mysql模块
filebeat modules enable mysql

5.配置mysql的模块
- module: mysql
  error:
    enabled: true
    var.paths: ["/var/lib/mysql/error.log"]

  slowlog:
    enabled: true 
    var.paths: ["/var/lib/mysql/slow.log"]
    
6.配置filebeat根据日志类型做判断
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

output.elasticsearch:
  hosts: ["10.0.0.114:9200"]
  indices:
    - index: "nginx_access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        fileset.module: "nginx"
        fileset.name: "access"
    - index: "nginx_error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        fileset.module: "nginx"
        fileset.name: "error"
        
    - index: "mysql_slowlog-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        fileset.module: "mysql"
        fileset.name: "slowlog"
    - index: "mysql_error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        fileset.module: "mysql"
        fileset.name: "error"

setup.template.name: "nginx"
setup.template.pattern: "nginx_*"
setup.template.enabled: false
setup.template.overwrite: true

7.重启filebeat
systemctl restart filebeat

第三章: 使用input的docker类型收集docker日志

0.docker安装命令
rm -fr /etc/yum.repos.d/local.repo
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.ustc.edu.cn/docker-ce/linux/centos/docker-ce.repo
sed -i 's#download.docker.com#mirrors.tuna.tsinghua.edu.cn/docker-ce#g' /etc/yum.repos.d/docker-ce.repo
yum install docker-ce -y

1.启动2个nginx容器
docker run -d -p 80:80 nginx
docker run -d -p 8080:80 nginx

2.修改filebeat配置文件
filebeat.inputs:
- type: docker
  containers.ids: 
    - '*'

output.elasticsearch:
  hosts: ["10.0.0.114:9200"]
  indices:
    - index: "docker_access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        stream: "stdout"
    - index: "docker_error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        stream: "stderr"

setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true

3.重启filebeat 
systemctl restart filebeat

4.访问nginx制造日志
curl 127.0.0.1/11111111111111111111
curl 127.0.0.1:8080/22222222222222222222
  
5.es-head和kibana查看

第四章: 收集docker日志可以早下班版
场景:
容器1: nginx
容器2: mysql

理想中的情况:
docker_nginx-6.6.0-2019.09
docker_mysql-6.6.0-2019.09

前提条件:
存在可以唯一区分容器业务类型的key
docker-compose安装报错解决地址https://www.cnblogs.com/eddie1127/p/12003358.html
容器编排: docker-compose

1.安装docker-compose 
yum install -y python2-pip 
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple pip -U
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
pip install docker-compose

2.编写docker-compose文件
[root@db-02 ~]# cat docker-compose.yml 
version: '3'
services:
  nginx:
    image: nginx:latest
    labels:
      service: nginx
    logging:
      options:
        labels: "service"
    ports:
      - "80:80"
  db:
    image: nginx:latest
    labels:
      service: db 
    logging:
      options:
        labels: "service"
    ports:
      - "8080:80"

3.删除以前的容器!谨慎操作!
docker rm -f  $(docker ps -a -q)

4.使用docker-compose启动docker容器
docker-compose up -d 


5.配置filebeat配置文件
filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/lib/docker/containers/*/*-json.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["10.0.0.114:9200"]
  indices:
    - index: "docker_nginx_access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "nginx"  
        stream: "stdout"
    - index: "docker_nginx_error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "nginx"  
        stream: "stderr"
    - index: "docker_db_access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "db"  
        stream: "stdout"
    - index: "docker_db_error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        attrs.service: "db"  
        stream: "stderr"

setup.template.name: "docker"
setup.template.pattern: "docker_*"
setup.template.enabled: false
setup.template.overwrite: true

6.重启filebeat
systemctl restar filebeat

7.访问nginx制造日志
curl 127.0.0.1/nginxxxxxxxxx
curl 127.0.0.1:8080/dbbbbbbbbbbbbbbbbb

8.es-head和kibana查看

第五章: 收集docker日志涨薪版

0.创建容器日志目录
mkdir /opt/{nginx,mysql}

1.将容器的日志目录挂载到宿主机
docker ps 
docker cp /etc/nginx/nginx.conf 容器ID:/etc/nginx/nginx.conf
docker commit 容器ID nginx:v3
docker rm -f  $(docker ps -a -q)
docker run -d -p 80:80 -v /opt/nginx:/var/log/nginx nginx:v3
docker run -d -p 8080:80 -v /opt/mysql:/var/log/nginx nginx:v3

2.修改filebeat配置文件
filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /opt/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["nginx_access"]

- type: log
  enabled: true 
  paths:
    - /opt/nginx/error.log
  tags: ["nginx_error"]

- type: log
  enabled: true 
  paths:
    - /opt/mysql/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["mysql_access"]

- type: log
  enabled: true 
  paths:
    - /opt/mysql/error.log
  tags: ["mysql_error"]

output.elasticsearch:
  hosts: ["10.0.0.114:9200"]
  indices:
    - index: "docker_nginx_access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "nginx_access"
    - index: "docker_nginx_error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "nginx_error"
    - index: "docker_db_access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "mysql_access"
    - index: "docker_db_error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "mysql_error"

setup.template.name: "docker"
setup.template.pattern: "docker_*"
setup.template.enabled: false
setup.template.overwrite: true

3.重启filebeat
systemctl restart filebeat

4.访问nginx制造日志
curl 127.0.0.1/nginxxxxxxxxx
curl 127.0.0.1:8080/dbbbbbbbbbbbbbbbbb

第六章: 使用缓存服务来缓解ES压力

1.安装配置redis
yum install redis -y
systemctl start redis 
redis-cli set k1 v1 
redis-cli get k1 

2.配置filebeat
filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/log/nginx/bbs.access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true 
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

output.redis:
  hosts: ["127.0.0.1"]
  keys:
    - key: "nginx_access"
      when.contains:
        tags: "access"
    - key: "nginx_error"
      when.contains:
        tags: "error"

setup.template.name: "nginx"
setup.template.pattern: "nginx_*"
setup.template.enabled: false
setup.template.overwrite: true

3.确保nginx日志为json格式
>/var/log/nginx/bbs.access.log 
ab -c 10 -n 100 http://10.0.0.114/oooooooooo
tail -1 /var/log/nginx/bbs.access.log 

4.启动filebeat并测试是否能存到redis里
systectl restart filebeat

redis-cli
keys * 
TYPE nginx_access 
LLEN nginx_access 
LRANGE nginx_access 1 2 

5.安装配置logstash
rpm -ivh logstash-6.6.0.rpm

[root@db-01 /data/soft]# cat /etc/logstash/conf.d/redis.conf 
input {
  redis {
    host => "127.0.0.1"
    port => "6379"
    db => "0"
    key => "nginx_access"
    data_type => "list"
  }
  redis {
    host => "127.0.0.1"
    port => "6379"
    db => "0"
    key => "nginx_error"
    data_type => "list"
  }
}

filter {
  mutate {
    convert => ["upstream_time", "float"]
    convert => ["request_time", "float"]
  }
}

output {
   stdout {}
   if "access" in [tags] {
      elasticsearch {
        hosts => "http://localhost:9200"
        manage_template => false
        index => "nginx_access-%{+yyyy.MM}"
      }
    }
    if "error" in [tags] {
      elasticsearch {
        hosts => "http://localhost:9200"
        manage_template => false
        index => "nginx_error-%{+yyyy.MM}"
      }
    }
}

6.启动Logstash
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis.conf

7.检查redis里是否被取走了
redis-cli
LLEN nginx_access 

8.es-head和kibana查看

第七章: 存入redis优化方案

1.优化filebeat,将所有的日志存入一个key中
filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/log/nginx/bbs.access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true 
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

output.redis:
  hosts: ["127.0.0.1"]
  key: "all"

setup.template.name: "nginx"
setup.template.pattern: "nginx_*"
setup.template.enabled: false
setup.template.overwrite: true

2.logstash从一个key里读取,根据tag标签判断 
input {
  redis {
    host => "127.0.0.1"
    port => "6379"
    db => "0"
    key => "all"
    data_type => "list"
  }
}

filter {
  mutate {
    convert => ["upstream_time", "float"]
    convert => ["request_time", "float"]
  }
}

output {
   stdout {}
   if "access" in [tags] {
      elasticsearch {
        hosts => "http://localhost:9200"
        manage_template => false
        index => "nginx_access-%{+yyyy.MM}"
      }
    }
    if "error" in [tags] {
      elasticsearch {
        hosts => "http://localhost:9200"
        manage_template => false
        index => "nginx_error-%{+yyyy.MM}"
      }
    }
}

elk（二）

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读